Oct 30 00:38:38.155200 ip-10-0-133-32 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 00:38:38.155210 ip-10-0-133-32 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 00:38:38.155217 ip-10-0-133-32 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 00:38:38.155442 ip-10-0-133-32 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 00:38:48.207744 ip-10-0-133-32 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 00:38:48.207765 ip-10-0-133-32 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ff1937be18574d00bf2d3fcf44b7c139 --
Oct 30 00:40:49.718050 ip-10-0-133-32 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 00:40:50.083927 ip-10-0-133-32 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:40:50.083927 ip-10-0-133-32 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 00:40:50.083927 ip-10-0-133-32 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:40:50.083927 ip-10-0-133-32 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:40:50.089143 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089066    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:40:50.089143 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089143    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:40:50.089143 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089147    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089150    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089154    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089157    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089160    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089163    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089166    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089168    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089171    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089173    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089176    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089178    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089181    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089183    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089187    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089190    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089192    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089200    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089203    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089206    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:40:50.089237 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089208    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089212    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089214    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089217    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089220    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089222    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089225    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089228    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089230    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089233    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089235    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089238    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089241    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089244    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089247    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089250    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089253    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089255    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089258    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089260    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:40:50.089687 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089262    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089265    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089267    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089270    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089272    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089275    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089277    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089280    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089283    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089286    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089288    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089291    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089293    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089296    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089299    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089302    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089305    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089307    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089311    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:40:50.090162 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089314    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089316    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089319    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089322    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089324    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089326    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089329    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089332    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089334    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089337    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089339    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089342    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089344    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089347    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089349    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089353    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089356    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089359    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089363    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:40:50.090602 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089367    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089371    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089374    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089377    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089380    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089383    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089386    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089389    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089392    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089394    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089397    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089399    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089402    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089405    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089408    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089411    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089413    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089415    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089418    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089420    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:40:50.091064 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089423    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089425    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089428    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089430    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089433    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.089435    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090215    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090221    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090224    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090227    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090230    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090233    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090236    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090239    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090241    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090244    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090248    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090250    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090253    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:40:50.091518 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090256    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090259    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090262    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090266    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090269    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090272    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090275    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090277    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090280    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090282    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090285    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090288    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090290    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090293    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090296    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090298    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090301    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090303    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090306    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:40:50.092018 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090308    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090311    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090313    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090315    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090319    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090321    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090324    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090326    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090329    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090332    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090334    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090337    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090340    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090342    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090346    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090348    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090351    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090354    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090356    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090359    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:40:50.092460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090363    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090365    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090367    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090370    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090372    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090375    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090378    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090380    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090383    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090385    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090388    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090390    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090393    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090395    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090398    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090400    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090404    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090406    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090409    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090411    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:40:50.092951 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090414    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090416    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090419    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090421    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090424    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090427    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090430    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090432    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090435    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090437    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090440    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090443    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090445    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090449    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090453    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090456    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090459    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090462    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090464    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090467    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:40:50.093407 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090469    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090472    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090475    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090477    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090480    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090482    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090485    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090487    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090491    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090494    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090496    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090499    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090501    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090504    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090976    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090981    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090985    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090988    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090991    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:40:50.093879 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090993    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090996    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.090998    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091001    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091004    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091006    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091009    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091011    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091014    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091016    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091019    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091021    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091024    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091026    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091029    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091031    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091034    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091037    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091040    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091043    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:40:50.094392 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091045    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091048    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091051    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091053    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091056    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091058    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091061    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091063    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091066    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091069    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091072    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091075    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091077    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091080    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091083    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091085    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091088    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091090    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091093    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091095    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:40:50.094881 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091098    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091100    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091103    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091105    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091108    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091110    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091112    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091115    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091117    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091120    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091123    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091125    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091128    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091131    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091133    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091136    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091138    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091141    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091143    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091146    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:40:50.095341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091148    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091151    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091153    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091156    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091159    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091162    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091165    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091168    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091171    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091173    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091176    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091178    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091181    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091183    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091185    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091188    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091191    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091194    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091198    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:40:50.095816 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091201    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091204    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091207    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091210    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091213    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091215    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091218    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091223    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091227    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091231    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091234    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091236    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091239    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091242    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091245    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091247    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091250    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091253    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091255    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:40:50.096293 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091258    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091260    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.091263    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091370    2572 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091377    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091383    2572 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091388    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091392    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091395    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091400    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091404    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091407    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091410    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091414    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091421    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091424    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091427    2572 flags.go:64] FLAG: --cgroup-root=""
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091430    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091433    2572 flags.go:64] FLAG: --client-ca-file=""
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091436    2572 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091440    2572 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091443    2572 flags.go:64] FLAG: --cluster-domain=""
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091446    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 00:40:50.096736 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091449    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091452    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091456    2572 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091459    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091462    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091465    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091468    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091471    2572 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091475    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091478    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091481    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091484    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091487    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091490    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091493    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091496    2572 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091499    2572 flags.go:64] FLAG: --enable-server="true"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091502    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091506    2572 flags.go:64] FLAG: --event-burst="100"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091509    2572 flags.go:64] FLAG: --event-qps="50"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091513    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091515    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091518    2572 flags.go:64] FLAG: --eviction-hard=""
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091524    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 00:40:50.097317 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091527    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091530    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091533    2572 flags.go:64] FLAG: --eviction-soft=""
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091536    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091539    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091543    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091546    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091549    2572 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091552    2572 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091554    2572 flags.go:64] FLAG: --feature-gates=""
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091561    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091564    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091567    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091570    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091573    2572 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091576    2572 flags.go:64] FLAG: --help="false"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091579    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091582    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091585    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091588    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091592    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091595    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091598    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 00:40:50.097903 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091601    2572 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091604    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091607    2572 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091610    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091613    2572 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091615    2572 flags.go:64] FLAG: --kube-reserved=""
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091618    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091621    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091626    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091629    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091632    2572 flags.go:64] FLAG: --lock-file=""
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091634    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091637    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091640    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091647    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091650    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091652    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091655    2572 flags.go:64] FLAG: --logging-format="text"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091658    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091661    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091664    2572 flags.go:64] FLAG: --manifest-url=""
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091667    2572 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091671    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091674    2572 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091679    2572 flags.go:64] FLAG: --max-pods="110"
Oct 30 00:40:50.098446 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091682    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091685    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091688    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091691    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091694    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091697    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091700    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091714    2572 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091717    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091720    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091724    2572 flags.go:64] FLAG: --pod-cidr=""
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091726    2572 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091729    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091732    2572 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091735    2572 flags.go:64] FLAG: --port="10250"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091739    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091742    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-088f2a0bb20a306cc"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091745    2572 flags.go:64] FLAG: --qos-reserved=""
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091748    2572 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091751    2572 flags.go:64] FLAG: --register-node="true"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091754    2572 flags.go:64] FLAG: --register-with-taints=""
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091759    2572 flags.go:64] FLAG: --registry-burst="10"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091762    2572 flags.go:64] FLAG: --registry-qps="5"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091765    2572 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091767    2572 flags.go:64] FLAG: --reserved-memory=""
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091772    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 00:40:50.099279 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091775    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091777    2572 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091780    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091783    2572 flags.go:64] FLAG: --runonce="false"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091786    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091789    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091803    2572 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091806    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091809    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091814    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091817    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091820    2572 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091823    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091826    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091829    2572 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091831    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091835    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091838    2572 flags.go:64] FLAG: --system-cgroups=""
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091841    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091846    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091848    2572 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091853    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091856    2572 flags.go:64] FLAG: --tls-min-version=""
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091859    2572 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091861    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 00:40:50.100119 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091864    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091867    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091871    2572 flags.go:64] FLAG: --v="2"
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091876    2572 flags.go:64] FLAG: --version="false"
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091880    2572 flags.go:64] FLAG: --vmodule=""
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091884    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.091887    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092018    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092022    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092025    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092028    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092030    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092033    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092036    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092038    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092041    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092043    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092046    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092050    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092052    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092055    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:40:50.100694 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092059    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092062    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092065    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092068    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092070    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092073    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092076    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092079    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092082    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092085    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092087    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092090    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092093    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092096    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092099    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092102    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092104    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092107    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092109    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:40:50.101200 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092112    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092114    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092117    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092119    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092122    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092124    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092127    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092129    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092132    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092134    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092138    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092140    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092143    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092146    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092148    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092150    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092153    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092155    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092158    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092160    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:40:50.101645 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092164    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092167    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092169    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092172    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092175    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092177    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092181    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092183    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092186    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092188    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092191    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092194    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092196    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092199    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092201    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092204    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092206    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092209    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092211    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092214    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:40:50.102183 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092216    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092219    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092221    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092227    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092230    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092232    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092234    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092237    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092239    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092242    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092244    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092247    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092252    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092255    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092258    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092261    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092264    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092266    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092270    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:40:50.102639 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092272    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:40:50.103093 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092275    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:40:50.103093 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092278    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:40:50.103093 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092280    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:40:50.103093 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092283    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:40:50.103093 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092285    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:40:50.103093 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092288    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:40:50.103093 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092290    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:40:50.103093 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092293    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:40:50.103093 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092295    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:40:50.103093 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092298    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:40:50.103093 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092300    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:40:50.103093 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092303    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:40:50.103093 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092305    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:40:50.103093 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.092330    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:40:50.103093 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092364    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:40:50.103093 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092368    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092372    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092375    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092377    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092380    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092383    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092385    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092388    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092390    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092393    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092397    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092399    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092402    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092404    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092407    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092411    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092413    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092416    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092418    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:40:50.103460 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092421    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092423    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092426    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092428    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092431    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092433    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092436    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092438    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092441    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092443    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092446    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092449    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092451    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092453    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092456    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092460    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092463    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092465    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092468    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092471    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:40:50.103910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092473    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092476    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092479    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092483    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092486    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092488    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092491    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092494    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092497    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092499    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092502    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092505    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092508    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092511    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092513    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092516    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092518    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092521    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092523    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092526    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:40:50.104376 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092528    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092530    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092533    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092535    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092538    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092540    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092543    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092545    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092549    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092551    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092555    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092559    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092561    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092566    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092568    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092572    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092574    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092577    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092579    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:40:50.104858 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092582    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092585    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092587    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092590    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092592    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092595    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092597    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092600    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092602    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092605    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092607    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092610    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092612    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092615    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092617    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092620    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092622    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092625    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092627    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092630    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:40:50.105309 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092632    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092635    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092638    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092640    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092643    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092645    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.092650    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.100231    2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.100366    2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.101002    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.101009    2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101064    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101069    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101073    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101076    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101079    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101082    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101085    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101087    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:40:50.105910 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101090    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101093    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101095    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101098    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101100    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101103    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101106    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101108    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101110    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101113    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101121    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101124    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101127    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101129    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101132    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101135    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101138    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101140    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101143    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101146    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:40:50.106345 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101149    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101152    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101155    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101157    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101160    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101163    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101165    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101168    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101171    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101173    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101176    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101179    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101181    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101183    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101186    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101189    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101191    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101194    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101196    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101199    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:40:50.106811 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101202    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101205    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101208    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101210    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101213    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101216    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101218    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101221    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101223    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101226    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101228    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101231    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101233    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101236    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101239    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101242    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101245    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101247    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101250    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101253    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:40:50.107283 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101255    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101258    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101261    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101263    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101266    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101270    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101274    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101277    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101280    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101283    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101285    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101289    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101292    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101295    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101297    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101301    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101303    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101306    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:40:50.107745 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101309    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101311    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101314    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101317    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101319    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101322    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101325    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101328    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101330    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101333    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101336    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101339    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101341    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101343    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101346    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101349    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101351    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101354    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101357    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:40:50.108175 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101361    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:40:50.108667 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.101389    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:40:50.108667 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101522    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:40:50.108667 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101527    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:40:50.108667 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101530    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:40:50.108667 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101533    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:40:50.108667 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101536    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:40:50.108667 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101540    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:40:50.108667 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101542    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:40:50.108667 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101545    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:40:50.108667 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101548    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:40:50.108667 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101550    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:40:50.108667 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101553    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:40:50.108667 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101556    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:40:50.108667 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101558    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:40:50.108667 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101561    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:40:50.108667 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101564    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101567    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101569    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101572    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101574    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101578    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101581    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101583    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101586    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101589    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101591    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101594    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101596    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101600    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101603    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101606    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101609    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101611    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:40:50.109107 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101614    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101616    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101619    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101621    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101624    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101626    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101630    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101632    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101635    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101637    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101640    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101642    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101645    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101647    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101650    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101653    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101656    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101658    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101661    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101664    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:40:50.109583 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101667    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101669    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101672    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101675    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101677    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101680    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101682    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101685    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101688    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101690    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101693    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101697    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101700    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101703    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101706    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101710    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101714    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101717    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101721    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:40:50.110063 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101725    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101728    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101731    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101734    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101736    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101739    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101741    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101744    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101746    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101748    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101751    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101754    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101757    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101759    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101762    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101764    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101767    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101770    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101772    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101775    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:40:50.110499 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101777    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:40:50.110964 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101779    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:40:50.110964 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101782    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:40:50.110964 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101784    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:40:50.110964 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101787    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:40:50.110964 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101790    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:40:50.110964 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101810    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:40:50.110964 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101813    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:40:50.110964 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101815    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:40:50.110964 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101818    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:40:50.110964 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101820    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:40:50.110964 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101823    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:40:50.110964 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101825    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:40:50.110964 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101828    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:40:50.110964 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101834    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:40:50.110964 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.101860    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:40:50.110964 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101891    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101896    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101899    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101902    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101904    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101907    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101910    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101912    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101915    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101917    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101920    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101922    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101925    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101927    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101930    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101933    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101936    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101938    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101941    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101943    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:40:50.111341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101946    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101948    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101951    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101953    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101956    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101958    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101961    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101964    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101966    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101969    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101973    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101976    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101978    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101981    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101984    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101986    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101989    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101991    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101994    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101997    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:40:50.111805 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.101999    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102002    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102004    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102007    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102010    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102012    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102015    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102017    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102020    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102023    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102025    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102029    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102032    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102035    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102038    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102040    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102043    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102046    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:40:50.112341 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102048    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102051    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102053    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102056    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102060    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102063    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102066    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102069    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102071    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102073    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102076    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102078    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102081    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102083    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102086    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102088    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102091    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102093    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102096    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102098    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:40:50.112860 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102101    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102104    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102106    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102109    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102111    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102114    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102116    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102119    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102121    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102126    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102129    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102132    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102135    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102137    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102140    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102142    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102146    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102149    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102152    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:40:50.113417 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102155    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:40:50.113985 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102158    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:40:50.113985 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102160    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:40:50.113985 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102163    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:40:50.113985 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102165    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:40:50.113985 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102168    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:40:50.113985 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102170    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:40:50.113985 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102173    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:40:50.113985 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:50.102175    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:40:50.113985 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.102225    2572 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 00:40:50.113985 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.104024    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 00:40:50.113985 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.104930    2572 server.go:1014] "Starting client certificate rotation"
Oct 30 00:40:50.113985 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.105021    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:40:50.113985 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.105052    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:40:50.113985 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.106857    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:40:50.113985 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.108423    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:40:50.118287 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.118271    2572 log.go:25] "Validated CRI v1 runtime API"
Oct 30 00:40:50.121282 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.121266    2572 log.go:25] "Validated CRI v1 image API"
Oct 30 00:40:50.122035 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.122008    2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 00:40:50.125992 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.125973    2572 fs.go:135] Filesystem UUIDs: map[6f0a5da8-8e2e-4ee6-99e8-3e1d6cbd6a2f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b22f0fda-b965-41a8-948e-a1c224d47f38:/dev/nvme0n1p4]
Oct 30 00:40:50.126057 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.125990    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 00:40:50.130948 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.130849    2572 manager.go:217] Machine: {Timestamp:2026-10-30 00:40:50.129502919 +0000 UTC m=+0.302770145 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3202563 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26eea458e551a80bead520c7e6f516 SystemUUID:ec26eea4-58e5-51a8-0bea-d520c7e6f516 BootID:ff1937be-1857-4d00-bf2d-3fcf44b7c139 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e2:0f:ce:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e2:0f:ce:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:4e:e7:45:18:80 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 00:40:50.131648 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.131638    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 00:40:50.131755 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.131743    2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 00:40:50.135172 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.135150    2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 00:40:50.135361 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.135174    2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-32.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 00:40:50.135399 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.135364    2572 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 00:40:50.135399 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.135372    2572 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 00:40:50.135456 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.135408    2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:40:50.135456 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.135427    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:40:50.136135 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.136125    2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 00:40:50.139578 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.139566    2572 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 00:40:50.139879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.139869    2572 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 00:40:50.140608 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.140599    2572 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 00:40:50.140637 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.140615    2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 00:40:50.140637 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.140635    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 00:40:50.140728 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.140644    2572 kubelet.go:405] "Adding apiserver pod source"
Oct 30 00:40:50.140728 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.140714    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 00:40:50.141181 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.141161    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:40:50.145510 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.145470    2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 00:40:50.149518 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.149505    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:40:50.150397 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.150385    2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 00:40:50.150443 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.150414    2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 00:40:50.150566 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.150555    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 00:40:50.150599 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.150570    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 00:40:50.150599 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.150576    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 00:40:50.150599 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.150581    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 00:40:50.150599 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.150587    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 00:40:50.150599 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.150593    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 00:40:50.151085 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.151076    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 00:40:50.151085 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.151085    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 00:40:50.151137 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.151092    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 00:40:50.151137 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.151097    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 00:40:50.151137 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.151106    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 00:40:50.151137 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.151114    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 00:40:50.152097 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.152088    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 00:40:50.152097 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.152097    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 00:40:50.156498 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.156482    2572 server.go:1263] "Started kubelet"
Oct 30 00:40:50.156599 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.156566    2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 00:40:50.156684 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.156617    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 00:40:50.156684 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.156675    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 00:40:50.157832 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.157790    2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 00:40:50.158526 ip-10-0-133-32 systemd[1]: Started Kubernetes Kubelet.
Oct 30 00:40:50.160093 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.160079    2572 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 00:40:50.164349 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.164333    2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 00:40:50.164349 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.164340    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 00:40:50.164577 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.164555    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Oct 30 00:40:50.165196 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.165078    2572 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 00:40:50.165196 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.165094    2572 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 00:40:50.165196 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.165137    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 00:40:50.165370 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.165206    2572 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 00:40:50.165370 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.165214    2572 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 00:40:50.165370 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.165243    2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 00:40:50.165554 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.165541    2572 factory.go:55] Registering systemd factory
Oct 30 00:40:50.165586 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.165555    2572 factory.go:223] Registration of the systemd container factory successfully
Oct 30 00:40:50.165823 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.165809    2572 factory.go:153] Registering CRI-O factory
Oct 30 00:40:50.165881 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.165827    2572 factory.go:223] Registration of the crio container factory successfully
Oct 30 00:40:50.165914 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.165897    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 00:40:50.165914 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.165911    2572 factory.go:103] Registering Raw factory
Oct 30 00:40:50.165965 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.165922    2572 manager.go:1199] Started watching for new ooms in manager
Oct 30 00:40:50.166783 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.166770    2572 manager.go:319] Starting recovery of all containers
Oct 30 00:40:50.180559 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.180528    2572 manager.go:324] Recovery completed
Oct 30 00:40:50.185399 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.185386    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:40:50.188103 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.187158    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-32.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 00:40:50.188103 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.187292    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:40:50.188103 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.187325    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:40:50.188103 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.187340    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:40:50.188103 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.187375    2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-32.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 00:40:50.188103 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.188027    2572 cpu_manager.go:226] "Starting" policy="none"
Oct 30 00:40:50.188103 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.188036    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:40:50.188103 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.188040    2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 00:40:50.188103 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.188073    2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 00:40:50.188493 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.187328    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-32.ec2.internal.18e327cfd1e079b9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-32.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-32.ec2.internal,},FirstTimestamp:2026-10-30 00:40:50.156452281 +0000 UTC m=+0.329719508,LastTimestamp:2026-10-30 00:40:50.156452281 +0000 UTC m=+0.329719508,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-32.ec2.internal,}"
Oct 30 00:40:50.188837 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.188823    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:40:50.188906 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.188849    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:40:50.188906 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.188858    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:40:50.190234 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.190224    2572 policy_none.go:50] "Start"
Oct 30 00:40:50.190270 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.190237    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 00:40:50.190270 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.190247    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 00:40:50.192120 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.192110    2572 policy_none.go:44] "Start"
Oct 30 00:40:50.201827 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.201777    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 00:40:50.201827 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.201814    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-32.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 00:40:50.201943 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.201776    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-32.ec2.internal.18e327cfd3b751e9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-32.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-32.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-32.ec2.internal,},FirstTimestamp:2026-10-30 00:40:50.187309545 +0000 UTC m=+0.360576782,LastTimestamp:2026-10-30 00:40:50.187309545 +0000 UTC m=+0.360576782,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-32.ec2.internal,}"
Oct 30 00:40:50.201943 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.201843    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 00:40:50.213289 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.213271    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 00:40:50.213353 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.213309    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-79gc6"
Oct 30 00:40:50.216864 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.216774    2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 00:40:50.217835 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.217753    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-32.ec2.internal.18e327cfd3b7a5c9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-32.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-32.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-32.ec2.internal,},FirstTimestamp:2026-10-30 00:40:50.187331017 +0000 UTC m=+0.360598251,LastTimestamp:2026-10-30 00:40:50.187331017 +0000 UTC m=+0.360598251,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-32.ec2.internal,}"
Oct 30 00:40:50.218573 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.218556    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-79gc6"
Oct 30 00:40:50.231918 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.231905    2572 manager.go:341] "Starting Device Plugin manager"
Oct 30 00:40:50.231996 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.231936    2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 00:40:50.231996 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.231945    2572 server.go:87] "Starting device plugin registration server"
Oct 30 00:40:50.232158 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.232147    2572 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 00:40:50.232202 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.232158    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 00:40:50.232259 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.232245    2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 00:40:50.232324 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.232314    2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 00:40:50.232365 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.232324    2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 00:40:50.232871 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.232853    2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 00:40:50.232965 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.232899    2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-32.ec2.internal\" not found"
Oct 30 00:40:50.236240 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.236183    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-32.ec2.internal.18e327cfd3b824e3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-32.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-32.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-32.ec2.internal,},FirstTimestamp:2026-10-30 00:40:50.187363555 +0000 UTC m=+0.360630786,LastTimestamp:2026-10-30 00:40:50.187363555 +0000 UTC m=+0.360630786,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-32.ec2.internal,}"
Oct 30 00:40:50.275655 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.275629    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 00:40:50.276725 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.276703    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 00:40:50.276725 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.276726    2572 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 00:40:50.276892 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.276761    2572 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 00:40:50.276892 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.276812    2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 00:40:50.292876 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.292857    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:40:50.332670 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.332649    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:40:50.333615 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.333585    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:40:50.333724 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.333622    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:40:50.333724 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.333636    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:40:50.333724 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.333668    2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.354650 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.354574    2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.354650 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.354596    2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-32.ec2.internal\": node \"ip-10-0-133-32.ec2.internal\" not found"
Oct 30 00:40:50.378062 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.378040    2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"]
Oct 30 00:40:50.378132 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.378091    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:40:50.378910 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.378895    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:40:50.378953 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.378925    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:40:50.378953 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.378935    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:40:50.380684 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.380672    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:40:50.380853 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.380837    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.380917 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.380872    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:40:50.381401 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.381384    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:40:50.381455 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.381413    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:40:50.381455 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.381424    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:40:50.381512 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.381389    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:40:50.381512 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.381483    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:40:50.381512 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.381496    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:40:50.383292 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.383275    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.383357 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.383306    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:40:50.384432 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.384416    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:40:50.384497 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.384441    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:40:50.384497 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.384451    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:40:50.394524 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.394510    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Oct 30 00:40:50.405841 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.405821    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-32.ec2.internal\" not found" node="ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.411906 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.411891    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-32.ec2.internal\" not found" node="ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.495526 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.495500    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Oct 30 00:40:50.566830 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.566809    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f92445811ce6c3905e7d9666e6d2a0cd-config\") pod \"kube-apiserver-proxy-ip-10-0-133-32.ec2.internal\" (UID: \"f92445811ce6c3905e7d9666e6d2a0cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.566919 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.566843    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f710fb4e919766c9ee824cfee175df00-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"f710fb4e919766c9ee824cfee175df00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.566919 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.566885    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f710fb4e919766c9ee824cfee175df00-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"f710fb4e919766c9ee824cfee175df00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.595861 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.595841    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Oct 30 00:40:50.667667 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.667610    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f92445811ce6c3905e7d9666e6d2a0cd-config\") pod \"kube-apiserver-proxy-ip-10-0-133-32.ec2.internal\" (UID: \"f92445811ce6c3905e7d9666e6d2a0cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.667667 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.667638    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f710fb4e919766c9ee824cfee175df00-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"f710fb4e919766c9ee824cfee175df00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.667667 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.667653    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f710fb4e919766c9ee824cfee175df00-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"f710fb4e919766c9ee824cfee175df00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.667825 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.667707    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f710fb4e919766c9ee824cfee175df00-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"f710fb4e919766c9ee824cfee175df00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.667825 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.667756    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f92445811ce6c3905e7d9666e6d2a0cd-config\") pod \"kube-apiserver-proxy-ip-10-0-133-32.ec2.internal\" (UID: \"f92445811ce6c3905e7d9666e6d2a0cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.667825 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.667776    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f710fb4e919766c9ee824cfee175df00-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"f710fb4e919766c9ee824cfee175df00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.696225 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.696205    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Oct 30 00:40:50.708530 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.708512    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.714847 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:50.714834    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:40:50.797054 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.797012    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Oct 30 00:40:50.897454 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.897432    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Oct 30 00:40:50.997925 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:50.997866    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Oct 30 00:40:51.098527 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:51.098505    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Oct 30 00:40:51.104666 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:51.104650    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 00:40:51.104844 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:51.104822    2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 00:40:51.199219 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:51.199050    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Oct 30 00:40:51.223047 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:51.223011    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:35:51 +0000 UTC" deadline="2028-07-31 08:05:21.947320806 +0000 UTC"
Oct 30 00:40:51.223047 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:51.223043    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15367h24m30.724280505s"
Oct 30 00:40:51.224422 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:51.224391    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92445811ce6c3905e7d9666e6d2a0cd.slice/crio-bf9b95b5a46c6d0a903419448e97bfadf4b3f5abc5a2d89b079dac9001e7c953 WatchSource:0}: Error finding container bf9b95b5a46c6d0a903419448e97bfadf4b3f5abc5a2d89b079dac9001e7c953: Status 404 returned error can't find the container with id bf9b95b5a46c6d0a903419448e97bfadf4b3f5abc5a2d89b079dac9001e7c953
Oct 30 00:40:51.224824 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:51.224779    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf710fb4e919766c9ee824cfee175df00.slice/crio-db088824e75d4d815e976a2beaf1ff299248af53e5d9b89145f60e2232895bba WatchSource:0}: Error finding container db088824e75d4d815e976a2beaf1ff299248af53e5d9b89145f60e2232895bba: Status 404 returned error can't find the container with id db088824e75d4d815e976a2beaf1ff299248af53e5d9b89145f60e2232895bba
Oct 30 00:40:51.229296 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:51.229281    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:40:52.844723 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:52.844683    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal" event={"ID":"f92445811ce6c3905e7d9666e6d2a0cd","Type":"ContainerStarted","Data":"bf9b95b5a46c6d0a903419448e97bfadf4b3f5abc5a2d89b079dac9001e7c953"}
Oct 30 00:40:52.846361 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:52.846342    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal" event={"ID":"f710fb4e919766c9ee824cfee175df00","Type":"ContainerStarted","Data":"db088824e75d4d815e976a2beaf1ff299248af53e5d9b89145f60e2232895bba"}
Oct 30 00:40:52.863782 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:52.863763    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Oct 30 00:40:52.894494 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:52.894477    2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:40:52.929191 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:52.929168    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:40:52.930169 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:52.930144    2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:40:52.952813 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:52.952796    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:40:52.953795 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:52.953782    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Oct 30 00:40:52.974009 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:52.973992    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:40:53.066333 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.066318    2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:40:53.664655 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.664624    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:40:53.707223 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.707198    2572 apiserver.go:52] "Watching apiserver"
Oct 30 00:40:53.730458 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.730434    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 00:40:53.734369 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.734345    2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 00:40:53.734748 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.734728    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal","openshift-cluster-node-tuning-operator/tuned-nn559","openshift-dns/node-resolver-8k6rj","openshift-multus/multus-additional-cni-plugins-q7mpt","openshift-network-diagnostics/network-check-target-qs6rx","kube-system/konnectivity-agent-5lmgn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j","openshift-image-registry/node-ca-7zgwc","openshift-multus/network-metrics-daemon-nqtxx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal","openshift-multus/multus-xjww4","openshift-network-operator/iptables-alerter-w22p8","openshift-ovn-kubernetes/ovnkube-node-857b8","kube-system/global-pull-secret-syncer-fw99d"]
Oct 30 00:40:53.737447 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.737422    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:40:53.737536 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:53.737503    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nqtxx" podUID="6a4ab2cc-af6d-460b-b8dd-9f16078cc11d"
Oct 30 00:40:53.739659 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.739638    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.741866 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.741845    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:40:53.744101 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.744081    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.746455 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.746432    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.748900 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.748878    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:40:53.751243 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.751225    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.753478 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.753459    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.755860 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.755843    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:40:53.757349 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.756742    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 00:40:53.757349 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.756823    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:40:53.757349 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.756925    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 00:40:53.757349 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.757026    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5b9kk\""
Oct 30 00:40:53.757349 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.757190    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 00:40:53.757349 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.757224    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-92qkq\""
Oct 30 00:40:53.758786 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.758247    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5lmgn"
Oct 30 00:40:53.758786 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.758479    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 00:40:53.758786 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.758636    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7sn5q\""
Oct 30 00:40:53.758786 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.758732    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 00:40:53.759002 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.758817    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 00:40:53.759002 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.758869    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 00:40:53.759002 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.758879    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 00:40:53.759990 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.759973    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 00:40:53.760234 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.760208    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 00:40:53.760561 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.760545    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:40:53.760643 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:53.760619    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rx" podUID="93171d29-b786-4586-80aa-9643856f242f"
Oct 30 00:40:53.761351 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.761329    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7htjs\""
Oct 30 00:40:53.764013 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.762306    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 00:40:53.764013 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.763349    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 00:40:53.764414 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.764369    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 00:40:53.764414 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.764375    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 00:40:53.765532 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.765512    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:40:53.765634 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:53.765596    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw99d" podUID="372da31c-aac8-485a-bcc4-31a830f176a9"
Oct 30 00:40:53.767354 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.767107    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 00:40:53.767354 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.767349    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:40:53.767495 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.767354    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-szv55\""
Oct 30 00:40:53.768821 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.768803    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 00:40:53.768821 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.768819    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 00:40:53.768947 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.768876    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 00:40:53.768947 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.768924    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cg6zp\""
Oct 30 00:40:53.769072 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.769053    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 00:40:53.769247 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.769065    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 00:40:53.769247 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.769197    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jgsn4\""
Oct 30 00:40:53.769395 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.769369    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z5p7p\""
Oct 30 00:40:53.770094 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.770077    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 00:40:53.770205 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.770143    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 00:40:53.770542 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.770513    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 00:40:53.774021 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.774006    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 00:40:53.774098 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.774047    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 00:40:53.774165 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.774134    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f5gxs\""
Oct 30 00:40:53.786020 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.785999    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 00:40:53.786097 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.786064    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2h5rw"
Oct 30 00:40:53.798712 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.798685    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2h5rw"
Oct 30 00:40:53.831574 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.831555    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 00:40:53.840587 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.840563    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-systemd-units\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.840688 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.840596    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-multus-certs\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.840688 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.840618    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.840794 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.840685    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-cni-netd\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.840794 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.840741    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79677abd-0ee7-4081-b31e-9f07e913fd2d-iptables-alerter-script\") pod \"iptables-alerter-w22p8\" (UID: \"79677abd-0ee7-4081-b31e-9f07e913fd2d\") " pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:40:53.840794 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.840777    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-run\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.840937 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.840819    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-daemon-config\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.840937 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.840852    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.840937 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.840886    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-k8s-cni-cncf-io\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.841068 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.840939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-tmp\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.841068 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.840969    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovnkube-script-lib\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.841068 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841001    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-kubelet\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.841068 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841024    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l57jg\" (UniqueName: \"kubernetes.io/projected/ae89a416-9d4c-4477-bf72-a5371f55701a-kube-api-access-l57jg\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.841068 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841050    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-socket-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.841276 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841072    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-etc-selinux\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.841276 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841118    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-tmp-dir\") pod \"node-resolver-8k6rj\" (UID: \"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414\") " pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:40:53.841276 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841138    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn7ts\" (UniqueName: \"kubernetes.io/projected/098b604a-a614-4584-abc6-bfbe37ae40e9-kube-api-access-wn7ts\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.841276 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841183    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-socket-dir-parent\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.841276 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841205    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-modprobe-d\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.841276 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841234    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:40:53.841471 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841295    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-ovn\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.841471 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841333    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn77j\" (UniqueName: \"kubernetes.io/projected/79677abd-0ee7-4081-b31e-9f07e913fd2d-kube-api-access-zn77j\") pod \"iptables-alerter-w22p8\" (UID: \"79677abd-0ee7-4081-b31e-9f07e913fd2d\") " pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:40:53.841471 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841360    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-system-cni-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.841471 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841393    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-netns\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.841471 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841424    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-var-lib-kubelet\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.841471 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841448    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-tuned\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.841676 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841487    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-cni-bin\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.841676 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841517    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-sys-fs\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.841676 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841533    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7qg2\" (UniqueName: \"kubernetes.io/projected/57ac048f-898e-4478-87f5-251679ad7bcf-kube-api-access-l7qg2\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.841676 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841578    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-kubernetes\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.841676 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841598    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysctl-conf\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.841676 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841637    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t7j2\" (UniqueName: \"kubernetes.io/projected/d1bada76-8261-4ed7-adaf-8de93cf733c9-kube-api-access-6t7j2\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.841676 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841659    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79677abd-0ee7-4081-b31e-9f07e913fd2d-host-slash\") pod \"iptables-alerter-w22p8\" (UID: \"79677abd-0ee7-4081-b31e-9f07e913fd2d\") " pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:40:53.841676 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841677    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-hostroot\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.841942 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841700    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pq68\" (UniqueName: \"kubernetes.io/projected/f6870c47-e336-47b5-adfa-7143eed6d6ab-kube-api-access-8pq68\") pod \"node-ca-7zgwc\" (UID: \"f6870c47-e336-47b5-adfa-7143eed6d6ab\") " pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:40:53.841942 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841753    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs\") pod \"network-metrics-daemon-nqtxx\" (UID: \"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d\") " pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:40:53.841942 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841768    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/372da31c-aac8-485a-bcc4-31a830f176a9-kubelet-config\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:40:53.841942 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841782    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-systemd\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.841942 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841795    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-var-lib-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.841942 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841808    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae89a416-9d4c-4477-bf72-a5371f55701a-cni-binary-copy\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.841942 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841822    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhbvm\" (UniqueName: \"kubernetes.io/projected/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-kube-api-access-qhbvm\") pod \"network-metrics-daemon-nqtxx\" (UID: \"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d\") " pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:40:53.841942 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841839    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysconfig\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.841942 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841860    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f65424f1-dbb4-4e42-ab0a-a3334cef4dba-agent-certs\") pod \"konnectivity-agent-5lmgn\" (UID: \"f65424f1-dbb4-4e42-ab0a-a3334cef4dba\") " pod="kube-system/konnectivity-agent-5lmgn"
Oct 30 00:40:53.841942 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841911    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/372da31c-aac8-485a-bcc4-31a830f176a9-dbus\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:40:53.841942 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841926    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-run-netns\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.841942 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841941    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-etc-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.842352 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841961    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-device-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.842352 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841976    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-sys\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.842352 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.841990    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-cnibin\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.842352 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842004    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-slash\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.842352 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842036    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovnkube-config\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.842352 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842058    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-cnibin\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.842352 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842075    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-cni-multus\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.842352 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842100    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-conf-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.842352 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842136    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.842352 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842167    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f6870c47-e336-47b5-adfa-7143eed6d6ab-serviceca\") pod \"node-ca-7zgwc\" (UID: \"f6870c47-e336-47b5-adfa-7143eed6d6ab\") " pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:40:53.842352 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842183    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f65424f1-dbb4-4e42-ab0a-a3334cef4dba-konnectivity-ca\") pod \"konnectivity-agent-5lmgn\" (UID: \"f65424f1-dbb4-4e42-ab0a-a3334cef4dba\") " pod="kube-system/konnectivity-agent-5lmgn"
Oct 30 00:40:53.842352 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842201    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jrrr\" (UniqueName: \"kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr\") pod \"network-check-target-qs6rx\" (UID: \"93171d29-b786-4586-80aa-9643856f242f\") " pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:40:53.842352 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842220    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-system-cni-dir\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.842352 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842316    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.842352 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842341    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.842816 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842357    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-env-overrides\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.842816 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842378    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-systemd\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.842816 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842391    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-hosts-file\") pod \"node-resolver-8k6rj\" (UID: \"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414\") " pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:40:53.842816 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842405    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-node-log\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.842816 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842418    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-log-socket\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.842816 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842432    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovn-node-metrics-cert\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.842816 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842452    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-cni-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.842816 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842476    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysctl-d\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.842816 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842499    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75h4s\" (UniqueName: \"kubernetes.io/projected/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-kube-api-access-75h4s\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.842816 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842530    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.842816 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842546    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.842816 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842570    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-registration-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.842816 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842583    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6870c47-e336-47b5-adfa-7143eed6d6ab-host\") pod \"node-ca-7zgwc\" (UID: \"f6870c47-e336-47b5-adfa-7143eed6d6ab\") " pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:40:53.842816 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842597    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-lib-modules\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.842816 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842618    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76pzc\" (UniqueName: \"kubernetes.io/projected/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-kube-api-access-76pzc\") pod \"node-resolver-8k6rj\" (UID: \"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414\") " pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:40:53.842816 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842638    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-kubelet\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.843343 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842652    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-cni-bin\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.843343 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842667    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-etc-kubernetes\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.843343 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842681    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-host\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.843343 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.842703    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.943794 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.943719    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-node-log\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.943794 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.943751    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-log-socket\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.943794 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.943769    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovn-node-metrics-cert\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.943794 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.943784    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-cni-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.944087 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.943805    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysctl-d\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.944087 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.943820    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75h4s\" (UniqueName: \"kubernetes.io/projected/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-kube-api-access-75h4s\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.944087 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.943836    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.944087 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.943859    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.944087 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.943878    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-registration-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.944087 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.943891    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6870c47-e336-47b5-adfa-7143eed6d6ab-host\") pod \"node-ca-7zgwc\" (UID: \"f6870c47-e336-47b5-adfa-7143eed6d6ab\") " pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:40:53.944087 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.943904    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-lib-modules\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.944087 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.943925    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76pzc\" (UniqueName: \"kubernetes.io/projected/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-kube-api-access-76pzc\") pod \"node-resolver-8k6rj\" (UID: \"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414\") " pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:40:53.944087 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.943942    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-kubelet\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.944087 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.943957    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-cni-bin\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.944087 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.943978    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-etc-kubernetes\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.944087 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.943999    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-host\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.944087 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944017    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.944087 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944032    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-systemd-units\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.944087 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944049    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-multus-certs\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.944087 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944068    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.944662 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944110    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-cni-netd\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.944662 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944133    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79677abd-0ee7-4081-b31e-9f07e913fd2d-iptables-alerter-script\") pod \"iptables-alerter-w22p8\" (UID: \"79677abd-0ee7-4081-b31e-9f07e913fd2d\") " pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:40:53.944662 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944165    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-run\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.944662 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944188    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-daemon-config\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.944662 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944212    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.944662 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944231    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-k8s-cni-cncf-io\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.944662 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944246    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-tmp\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.944662 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944268    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovnkube-script-lib\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.944662 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944282    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-kubelet\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.944662 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944300    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l57jg\" (UniqueName: \"kubernetes.io/projected/ae89a416-9d4c-4477-bf72-a5371f55701a-kube-api-access-l57jg\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.944662 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944319    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-socket-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.944662 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944341    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-etc-selinux\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.944662 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944356    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-tmp-dir\") pod \"node-resolver-8k6rj\" (UID: \"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414\") " pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:40:53.944662 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944371    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn7ts\" (UniqueName: \"kubernetes.io/projected/098b604a-a614-4584-abc6-bfbe37ae40e9-kube-api-access-wn7ts\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.944662 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944390    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-socket-dir-parent\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.944662 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944409    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-modprobe-d\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.944662 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944426    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:40:53.945200 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944455    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-ovn\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.945200 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944478    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn77j\" (UniqueName: \"kubernetes.io/projected/79677abd-0ee7-4081-b31e-9f07e913fd2d-kube-api-access-zn77j\") pod \"iptables-alerter-w22p8\" (UID: \"79677abd-0ee7-4081-b31e-9f07e913fd2d\") " pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:40:53.945200 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944494    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-system-cni-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.945200 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944513    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-netns\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.945200 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944528    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-var-lib-kubelet\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.945200 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944547    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-tuned\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.945200 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944563    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-cni-bin\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.945200 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944581    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-sys-fs\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.945200 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944596    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7qg2\" (UniqueName: \"kubernetes.io/projected/57ac048f-898e-4478-87f5-251679ad7bcf-kube-api-access-l7qg2\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.945200 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944612    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-kubernetes\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.945200 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944631    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysctl-conf\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.945200 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944647    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t7j2\" (UniqueName: \"kubernetes.io/projected/d1bada76-8261-4ed7-adaf-8de93cf733c9-kube-api-access-6t7j2\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.945200 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944664    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79677abd-0ee7-4081-b31e-9f07e913fd2d-host-slash\") pod \"iptables-alerter-w22p8\" (UID: \"79677abd-0ee7-4081-b31e-9f07e913fd2d\") " pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:40:53.945200 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944679    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-hostroot\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.945200 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944701    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pq68\" (UniqueName: \"kubernetes.io/projected/f6870c47-e336-47b5-adfa-7143eed6d6ab-kube-api-access-8pq68\") pod \"node-ca-7zgwc\" (UID: \"f6870c47-e336-47b5-adfa-7143eed6d6ab\") " pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:40:53.945200 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944716    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs\") pod \"network-metrics-daemon-nqtxx\" (UID: \"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d\") " pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:40:53.945200 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944731    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/372da31c-aac8-485a-bcc4-31a830f176a9-kubelet-config\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:40:53.945879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944746    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-systemd\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.945879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944762    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-var-lib-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.945879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944781    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae89a416-9d4c-4477-bf72-a5371f55701a-cni-binary-copy\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.945879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944798    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhbvm\" (UniqueName: \"kubernetes.io/projected/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-kube-api-access-qhbvm\") pod \"network-metrics-daemon-nqtxx\" (UID: \"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d\") " pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:40:53.945879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944813    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysconfig\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.945879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944828    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f65424f1-dbb4-4e42-ab0a-a3334cef4dba-agent-certs\") pod \"konnectivity-agent-5lmgn\" (UID: \"f65424f1-dbb4-4e42-ab0a-a3334cef4dba\") " pod="kube-system/konnectivity-agent-5lmgn"
Oct 30 00:40:53.945879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944846    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/372da31c-aac8-485a-bcc4-31a830f176a9-dbus\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:40:53.945879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944865    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-run-netns\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.945879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944879    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-etc-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.945879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944894    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-device-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.945879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944908    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-sys\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.945879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944927    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-cnibin\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.945879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944943    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-slash\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.945879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944959    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovnkube-config\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.945879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.944983    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-cnibin\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.945879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.945005    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-cni-multus\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.945879 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.945020    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-conf-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.945038    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.945058    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f6870c47-e336-47b5-adfa-7143eed6d6ab-serviceca\") pod \"node-ca-7zgwc\" (UID: \"f6870c47-e336-47b5-adfa-7143eed6d6ab\") " pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.945074    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f65424f1-dbb4-4e42-ab0a-a3334cef4dba-konnectivity-ca\") pod \"konnectivity-agent-5lmgn\" (UID: \"f65424f1-dbb4-4e42-ab0a-a3334cef4dba\") " pod="kube-system/konnectivity-agent-5lmgn"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.945090    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrrr\" (UniqueName: \"kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr\") pod \"network-check-target-qs6rx\" (UID: \"93171d29-b786-4586-80aa-9643856f242f\") " pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.945106    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-system-cni-dir\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.945121    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.945137    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.945177    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-env-overrides\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.945205    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-systemd\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.945232    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-hosts-file\") pod \"node-resolver-8k6rj\" (UID: \"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414\") " pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.945365    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-hosts-file\") pod \"node-resolver-8k6rj\" (UID: \"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414\") " pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.945424    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-node-log\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.945465    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-log-socket\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.945856    2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.945905    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-var-lib-kubelet\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.946283    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/372da31c-aac8-485a-bcc4-31a830f176a9-dbus\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.946382    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-cni-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.946482 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.946464    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysctl-d\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.947237 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.946644    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-conf-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.947237 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.946703    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-run-netns\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.947237 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.946723    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-cni-bin\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.947237 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.946763    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-etc-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.947237 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.946782    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-k8s-cni-cncf-io\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.947237 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.946788    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-sys\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.947237 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.946801    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-device-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.947237 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.946836    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-cnibin\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.947237 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.946846    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-sys-fs\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.947237 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.946881    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-slash\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.947237 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.946882    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.947237 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.947071    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-daemon-config\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.947237 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.947108    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-kubernetes\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.947955 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.947896    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.948368    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.948387    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovnkube-script-lib\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.948441    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-registration-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.948491    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6870c47-e336-47b5-adfa-7143eed6d6ab-host\") pod \"node-ca-7zgwc\" (UID: \"f6870c47-e336-47b5-adfa-7143eed6d6ab\") " pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.948592    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-lib-modules\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.948711    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovnkube-config\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.948779    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysctl-conf\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.948823    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-kubelet\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.948876    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.948928    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-cni-bin\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.948961    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-etc-kubernetes\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.949015    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.949086    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-systemd-units\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.949107    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f6870c47-e336-47b5-adfa-7143eed6d6ab-serviceca\") pod \"node-ca-7zgwc\" (UID: \"f6870c47-e336-47b5-adfa-7143eed6d6ab\") " pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.949138    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-multus-certs\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.949532    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-host\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.949587    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.951515 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.949604    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-socket-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.952379 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.949638    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-cni-netd\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.952379 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.949683    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-run\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.952379 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.949711    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-etc-selinux\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.952379 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.949716    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-kubelet\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.952379 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.949765    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-cnibin\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.952379 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.949895    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.952379 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.950084    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-tmp-dir\") pod \"node-resolver-8k6rj\" (UID: \"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414\") " pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:40:53.952379 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.950303    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/372da31c-aac8-485a-bcc4-31a830f176a9-kubelet-config\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:40:53.952379 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.950477    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-systemd\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.952379 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.950714    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovn-node-metrics-cert\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.952379 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.950795    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-socket-dir-parent\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.952379 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.950936    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-modprobe-d\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.952379 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.950990    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysconfig\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.952379 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.951023    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79677abd-0ee7-4081-b31e-9f07e913fd2d-host-slash\") pod \"iptables-alerter-w22p8\" (UID: \"79677abd-0ee7-4081-b31e-9f07e913fd2d\") " pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:40:53.952379 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.951383    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-hostroot\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.952379 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:53.951466    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:53.952379 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:53.951542    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret podName:372da31c-aac8-485a-bcc4-31a830f176a9 nodeName:}" failed. No retries permitted until 2026-10-30 00:40:54.451522066 +0000 UTC m=+3.060429073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret") pod "global-pull-secret-syncer-fw99d" (UID: "372da31c-aac8-485a-bcc4-31a830f176a9") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:53.953170 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.952108    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-var-lib-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.953170 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:53.952146    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:53.953170 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:53.952266    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs podName:6a4ab2cc-af6d-460b-b8dd-9f16078cc11d nodeName:}" failed. No retries permitted until 2026-10-30 00:40:54.452229748 +0000 UTC m=+3.061136756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs") pod "network-metrics-daemon-nqtxx" (UID: "6a4ab2cc-af6d-460b-b8dd-9f16078cc11d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:53.953170 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.952332    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-cni-multus\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.953170 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.952408    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-system-cni-dir\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.953170 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.952455    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-system-cni-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.953170 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.952556    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-ovn\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.953170 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.952620    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-netns\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.953170 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.952655    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-systemd\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.956901 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.956877    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae89a416-9d4c-4477-bf72-a5371f55701a-cni-binary-copy\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.956987 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.956885    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f65424f1-dbb4-4e42-ab0a-a3334cef4dba-konnectivity-ca\") pod \"konnectivity-agent-5lmgn\" (UID: \"f65424f1-dbb4-4e42-ab0a-a3334cef4dba\") " pod="kube-system/konnectivity-agent-5lmgn"
Oct 30 00:40:53.956987 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.956932    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-tuned\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.957285 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.957261    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79677abd-0ee7-4081-b31e-9f07e913fd2d-iptables-alerter-script\") pod \"iptables-alerter-w22p8\" (UID: \"79677abd-0ee7-4081-b31e-9f07e913fd2d\") " pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:40:53.958188 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.958165    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.958470 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.958444    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-env-overrides\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:53.958573 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.958551    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-tmp\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.961243 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.961221    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f65424f1-dbb4-4e42-ab0a-a3334cef4dba-agent-certs\") pod \"konnectivity-agent-5lmgn\" (UID: \"f65424f1-dbb4-4e42-ab0a-a3334cef4dba\") " pod="kube-system/konnectivity-agent-5lmgn"
Oct 30 00:40:53.969194 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.969132    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7qg2\" (UniqueName: \"kubernetes.io/projected/57ac048f-898e-4478-87f5-251679ad7bcf-kube-api-access-l7qg2\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:53.969291 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.969263    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pq68\" (UniqueName: \"kubernetes.io/projected/f6870c47-e336-47b5-adfa-7143eed6d6ab-kube-api-access-8pq68\") pod \"node-ca-7zgwc\" (UID: \"f6870c47-e336-47b5-adfa-7143eed6d6ab\") " pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:40:53.971359 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.971339    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76pzc\" (UniqueName: \"kubernetes.io/projected/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-kube-api-access-76pzc\") pod \"node-resolver-8k6rj\" (UID: \"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414\") " pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:40:53.973001 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.972978    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn7ts\" (UniqueName: \"kubernetes.io/projected/098b604a-a614-4584-abc6-bfbe37ae40e9-kube-api-access-wn7ts\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:53.973417 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.973375    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75h4s\" (UniqueName: \"kubernetes.io/projected/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-kube-api-access-75h4s\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:53.979355 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:53.978904    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:40:53.979355 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:53.978928    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:40:53.979355 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:53.978943    2572 projected.go:196] Error preparing data for projected volume kube-api-access-9jrrr for pod openshift-network-diagnostics/network-check-target-qs6rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:53.979355 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:53.978996    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr podName:93171d29-b786-4586-80aa-9643856f242f nodeName:}" failed. No retries permitted until 2026-10-30 00:40:54.478979757 +0000 UTC m=+3.087886770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9jrrr" (UniqueName: "kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr") pod "network-check-target-qs6rx" (UID: "93171d29-b786-4586-80aa-9643856f242f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:53.980931 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.980699    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn77j\" (UniqueName: \"kubernetes.io/projected/79677abd-0ee7-4081-b31e-9f07e913fd2d-kube-api-access-zn77j\") pod \"iptables-alerter-w22p8\" (UID: \"79677abd-0ee7-4081-b31e-9f07e913fd2d\") " pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:40:53.980931 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.980840    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhbvm\" (UniqueName: \"kubernetes.io/projected/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-kube-api-access-qhbvm\") pod \"network-metrics-daemon-nqtxx\" (UID: \"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d\") " pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:40:53.981415 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.981378    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l57jg\" (UniqueName: \"kubernetes.io/projected/ae89a416-9d4c-4477-bf72-a5371f55701a-kube-api-access-l57jg\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:40:53.981925 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:53.981887    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t7j2\" (UniqueName: \"kubernetes.io/projected/d1bada76-8261-4ed7-adaf-8de93cf733c9-kube-api-access-6t7j2\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:54.055017 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.054985    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:40:54.066642 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:54.066622    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bdb0b4_fc37_4886_93eb_9dc6ce910e63.slice/crio-5e92eaca5a85cf3eef9872798ee6f21661180e698557c3aac3b667276677d3be WatchSource:0}: Error finding container 5e92eaca5a85cf3eef9872798ee6f21661180e698557c3aac3b667276677d3be: Status 404 returned error can't find the container with id 5e92eaca5a85cf3eef9872798ee6f21661180e698557c3aac3b667276677d3be
Oct 30 00:40:54.074993 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.074970    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:40:54.079030 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.079013    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:40:54.088536 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:54.088515    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b80e7e_75db_4e4e_9c1b_0d2f4b754414.slice/crio-01807897f26a162bbd2dbb2979a3b8bf22171bd050a356bb06d7d95550eb041a WatchSource:0}: Error finding container 01807897f26a162bbd2dbb2979a3b8bf22171bd050a356bb06d7d95550eb041a: Status 404 returned error can't find the container with id 01807897f26a162bbd2dbb2979a3b8bf22171bd050a356bb06d7d95550eb041a
Oct 30 00:40:54.089947 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:54.089929    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098b604a_a614_4584_abc6_bfbe37ae40e9.slice/crio-ed625f5cad539fbb5aaa93de13ca93d89d6c585b85dbd26f2a472f681f877255 WatchSource:0}: Error finding container ed625f5cad539fbb5aaa93de13ca93d89d6c585b85dbd26f2a472f681f877255: Status 404 returned error can't find the container with id ed625f5cad539fbb5aaa93de13ca93d89d6c585b85dbd26f2a472f681f877255
Oct 30 00:40:54.092912 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.092891    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:40:54.097313 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.097290    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:40:54.098331 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.098314    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xjww4"
Oct 30 00:40:54.101574 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:54.101553    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79677abd_0ee7_4081_b31e_9f07e913fd2d.slice/crio-5c0eab2a4a7a80ad6c3562b408163aeb13743bd43038107b928e7441be33ff0c WatchSource:0}: Error finding container 5c0eab2a4a7a80ad6c3562b408163aeb13743bd43038107b928e7441be33ff0c: Status 404 returned error can't find the container with id 5c0eab2a4a7a80ad6c3562b408163aeb13743bd43038107b928e7441be33ff0c
Oct 30 00:40:54.104763 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.104746    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:40:54.106808 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.106789    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:40:54.112035 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.112021    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5lmgn"
Oct 30 00:40:54.131717 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:54.131696    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1bada76_8261_4ed7_adaf_8de93cf733c9.slice/crio-46963cd6ffcef17e1b289b99fdc1296214f353768c00c222a500aa7ee2b6ab01 WatchSource:0}: Error finding container 46963cd6ffcef17e1b289b99fdc1296214f353768c00c222a500aa7ee2b6ab01: Status 404 returned error can't find the container with id 46963cd6ffcef17e1b289b99fdc1296214f353768c00c222a500aa7ee2b6ab01
Oct 30 00:40:54.133884 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:40:54.133851    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65424f1_dbb4_4e42_ab0a_a3334cef4dba.slice/crio-a30e8b1969efd766b01fadee5adb70c7cec6959b6b2fee32c3ff3fadcc6bdbb3 WatchSource:0}: Error finding container a30e8b1969efd766b01fadee5adb70c7cec6959b6b2fee32c3ff3fadcc6bdbb3: Status 404 returned error can't find the container with id a30e8b1969efd766b01fadee5adb70c7cec6959b6b2fee32c3ff3fadcc6bdbb3
Oct 30 00:40:54.551677 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.551598    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrrr\" (UniqueName: \"kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr\") pod \"network-check-target-qs6rx\" (UID: \"93171d29-b786-4586-80aa-9643856f242f\") " pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:40:54.551677 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.551668    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:40:54.551881 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.551705    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs\") pod \"network-metrics-daemon-nqtxx\" (UID: \"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d\") " pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:40:54.551881 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:54.551821    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:54.551881 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:54.551875    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs podName:6a4ab2cc-af6d-460b-b8dd-9f16078cc11d nodeName:}" failed. No retries permitted until 2026-10-30 00:40:55.551858873 +0000 UTC m=+4.160765872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs") pod "network-metrics-daemon-nqtxx" (UID: "6a4ab2cc-af6d-460b-b8dd-9f16078cc11d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:54.552286 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:54.552269    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:40:54.552363 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:54.552290    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:40:54.552363 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:54.552303    2572 projected.go:196] Error preparing data for projected volume kube-api-access-9jrrr for pod openshift-network-diagnostics/network-check-target-qs6rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:54.552363 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:54.552347    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr podName:93171d29-b786-4586-80aa-9643856f242f nodeName:}" failed. No retries permitted until 2026-10-30 00:40:55.552332318 +0000 UTC m=+4.161239314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9jrrr" (UniqueName: "kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr") pod "network-check-target-qs6rx" (UID: "93171d29-b786-4586-80aa-9643856f242f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:54.552513 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:54.552404    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:54.552513 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:54.552435    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret podName:372da31c-aac8-485a-bcc4-31a830f176a9 nodeName:}" failed. No retries permitted until 2026-10-30 00:40:55.552424558 +0000 UTC m=+4.161331560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret") pod "global-pull-secret-syncer-fw99d" (UID: "372da31c-aac8-485a-bcc4-31a830f176a9") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:54.801893 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.801734    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:35:53 +0000 UTC" deadline="2028-04-28 20:25:49.463351553 +0000 UTC"
Oct 30 00:40:54.801893 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.801797    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13123h44m54.661558442s"
Oct 30 00:40:54.842666 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.842199    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:40:54.842666 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:54.842322    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rx" podUID="93171d29-b786-4586-80aa-9643856f242f"
Oct 30 00:40:54.854205 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.854174    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal" event={"ID":"f92445811ce6c3905e7d9666e6d2a0cd","Type":"ContainerStarted","Data":"9755023c1e94d2ffc36a2ed9a909e373b40628d29cc81304743b69df93c1d1f9"}
Oct 30 00:40:54.862572 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.862270    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j" event={"ID":"57ac048f-898e-4478-87f5-251679ad7bcf","Type":"ContainerStarted","Data":"e11a3dc899194af4a0b1a526c973c17d086a9194b5c6d1eb0f132739c21725a4"}
Oct 30 00:40:54.873044 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.872984    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7zgwc" event={"ID":"f6870c47-e336-47b5-adfa-7143eed6d6ab","Type":"ContainerStarted","Data":"efa04a170026a47686f8371cdc8639e4be1dcfc33c16f8af4f09e48e3680b616"}
Oct 30 00:40:54.886382 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.886358    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5lmgn" event={"ID":"f65424f1-dbb4-4e42-ab0a-a3334cef4dba","Type":"ContainerStarted","Data":"a30e8b1969efd766b01fadee5adb70c7cec6959b6b2fee32c3ff3fadcc6bdbb3"}
Oct 30 00:40:54.893263 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.890979    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7mpt" event={"ID":"098b604a-a614-4584-abc6-bfbe37ae40e9","Type":"ContainerStarted","Data":"ed625f5cad539fbb5aaa93de13ca93d89d6c585b85dbd26f2a472f681f877255"}
Oct 30 00:40:54.909777 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.909751    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nn559" event={"ID":"03bdb0b4-fc37-4886-93eb-9dc6ce910e63","Type":"ContainerStarted","Data":"5e92eaca5a85cf3eef9872798ee6f21661180e698557c3aac3b667276677d3be"}
Oct 30 00:40:54.918906 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.918881    2572 generic.go:356] "Generic (PLEG): container finished" podID="f710fb4e919766c9ee824cfee175df00" containerID="62b18312a6fb1bd8c8ecee96b509655ed85a75932dd3c2c8504f44764d54b43d" exitCode=0
Oct 30 00:40:54.919008 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.918964    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal" event={"ID":"f710fb4e919766c9ee824cfee175df00","Type":"ContainerDied","Data":"62b18312a6fb1bd8c8ecee96b509655ed85a75932dd3c2c8504f44764d54b43d"}
Oct 30 00:40:54.920855 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.920812    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xjww4" event={"ID":"ae89a416-9d4c-4477-bf72-a5371f55701a","Type":"ContainerStarted","Data":"d1ae6d47423fbd414d8969ab8a547ced0df026f778417443a085a723f4b21823"}
Oct 30 00:40:54.928588 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.928499    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857b8" event={"ID":"d1bada76-8261-4ed7-adaf-8de93cf733c9","Type":"ContainerStarted","Data":"46963cd6ffcef17e1b289b99fdc1296214f353768c00c222a500aa7ee2b6ab01"}
Oct 30 00:40:54.948709 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.948637    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w22p8" event={"ID":"79677abd-0ee7-4081-b31e-9f07e913fd2d","Type":"ContainerStarted","Data":"5c0eab2a4a7a80ad6c3562b408163aeb13743bd43038107b928e7441be33ff0c"}
Oct 30 00:40:54.952395 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.952375    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8k6rj" event={"ID":"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414","Type":"ContainerStarted","Data":"01807897f26a162bbd2dbb2979a3b8bf22171bd050a356bb06d7d95550eb041a"}
Oct 30 00:40:54.956998 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:54.956925    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal" podStartSLOduration=2.9569105970000003 podStartE2EDuration="2.956910597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:54.899409469 +0000 UTC m=+3.508316484" watchObservedRunningTime="2026-10-30 00:40:54.956910597 +0000 UTC m=+3.565817613"
Oct 30 00:40:55.560185 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:55.559567    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs\") pod \"network-metrics-daemon-nqtxx\" (UID: \"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d\") " pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:40:55.560185 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:55.559826    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:55.560185 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:55.559849    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrrr\" (UniqueName: \"kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr\") pod \"network-check-target-qs6rx\" (UID: \"93171d29-b786-4586-80aa-9643856f242f\") " pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:40:55.560185 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:55.559897    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs podName:6a4ab2cc-af6d-460b-b8dd-9f16078cc11d nodeName:}" failed. No retries permitted until 2026-10-30 00:40:57.559877141 +0000 UTC m=+6.168784156 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs") pod "network-metrics-daemon-nqtxx" (UID: "6a4ab2cc-af6d-460b-b8dd-9f16078cc11d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:55.560185 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:55.559947    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:40:55.560185 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:55.559969    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:40:55.560185 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:55.559988    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:40:55.560185 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:55.560002    2572 projected.go:196] Error preparing data for projected volume kube-api-access-9jrrr for pod openshift-network-diagnostics/network-check-target-qs6rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:55.560185 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:55.560030    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:55.560185 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:55.560048    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr podName:93171d29-b786-4586-80aa-9643856f242f nodeName:}" failed. No retries permitted until 2026-10-30 00:40:57.560034617 +0000 UTC m=+6.168941630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9jrrr" (UniqueName: "kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr") pod "network-check-target-qs6rx" (UID: "93171d29-b786-4586-80aa-9643856f242f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:55.560185 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:55.560069    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret podName:372da31c-aac8-485a-bcc4-31a830f176a9 nodeName:}" failed. No retries permitted until 2026-10-30 00:40:57.56005946 +0000 UTC m=+6.168966457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret") pod "global-pull-secret-syncer-fw99d" (UID: "372da31c-aac8-485a-bcc4-31a830f176a9") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:55.802413 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:55.802377    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:35:53 +0000 UTC" deadline="2028-08-04 18:08:16.860673852 +0000 UTC"
Oct 30 00:40:55.802413 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:55.802412    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15473h27m21.058266258s"
Oct 30 00:40:55.845956 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:55.845272    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:40:55.845956 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:55.845413    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nqtxx" podUID="6a4ab2cc-af6d-460b-b8dd-9f16078cc11d"
Oct 30 00:40:55.845956 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:55.845794    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:40:55.845956 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:55.845884    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw99d" podUID="372da31c-aac8-485a-bcc4-31a830f176a9"
Oct 30 00:40:55.965037 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:55.964716    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal" event={"ID":"f710fb4e919766c9ee824cfee175df00","Type":"ContainerStarted","Data":"111545754b6b0585f925d00fce2b5397bf4d496e3e9b5f5d4210c6e1af2d5498"}
Oct 30 00:40:56.841544 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:56.841515    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:40:56.841941 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:56.841637    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rx" podUID="93171d29-b786-4586-80aa-9643856f242f"
Oct 30 00:40:56.981016 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:56.980983    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7mpt" event={"ID":"098b604a-a614-4584-abc6-bfbe37ae40e9","Type":"ContainerStarted","Data":"0c775ac4c57b9c4d2bade838b2d0d064541ea1be33ae905a116e603268182bea"}
Oct 30 00:40:57.050952 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:57.049877    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal" podStartSLOduration=5.0498587 podStartE2EDuration="5.0498587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:56.003298703 +0000 UTC m=+4.612205719" watchObservedRunningTime="2026-10-30 00:40:57.0498587 +0000 UTC m=+5.658765718"
Oct 30 00:40:57.576885 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:57.576849    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs\") pod \"network-metrics-daemon-nqtxx\" (UID: \"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d\") " pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:40:57.577048 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:57.576906    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrrr\" (UniqueName: \"kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr\") pod \"network-check-target-qs6rx\" (UID: \"93171d29-b786-4586-80aa-9643856f242f\") " pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:40:57.577048 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:57.576952    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:40:57.577189 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:57.577061    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:57.577189 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:57.577114    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret podName:372da31c-aac8-485a-bcc4-31a830f176a9 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:01.577097604 +0000 UTC m=+10.186004605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret") pod "global-pull-secret-syncer-fw99d" (UID: "372da31c-aac8-485a-bcc4-31a830f176a9") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:40:57.577536 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:57.577517    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:57.577617 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:57.577575    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs podName:6a4ab2cc-af6d-460b-b8dd-9f16078cc11d nodeName:}" failed. No retries permitted until 2026-10-30 00:41:01.577558279 +0000 UTC m=+10.186465276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs") pod "network-metrics-daemon-nqtxx" (UID: "6a4ab2cc-af6d-460b-b8dd-9f16078cc11d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:40:57.577671 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:57.577655    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:40:57.577717 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:57.577672    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:40:57.577717 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:57.577686    2572 projected.go:196] Error preparing data for projected volume kube-api-access-9jrrr for pod openshift-network-diagnostics/network-check-target-qs6rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:57.577813 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:57.577722    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr podName:93171d29-b786-4586-80aa-9643856f242f nodeName:}" failed. No retries permitted until 2026-10-30 00:41:01.577710201 +0000 UTC m=+10.186617195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9jrrr" (UniqueName: "kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr") pod "network-check-target-qs6rx" (UID: "93171d29-b786-4586-80aa-9643856f242f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:40:57.842333 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:57.842270    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:40:57.842771 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:57.842410    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nqtxx" podUID="6a4ab2cc-af6d-460b-b8dd-9f16078cc11d"
Oct 30 00:40:57.842771 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:57.842603    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:40:57.842771 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:57.842698    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw99d" podUID="372da31c-aac8-485a-bcc4-31a830f176a9"
Oct 30 00:40:57.984885 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:57.984848    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5lmgn" event={"ID":"f65424f1-dbb4-4e42-ab0a-a3334cef4dba","Type":"ContainerStarted","Data":"14c8edb8c97c786ee89d1fa2e6a2b67da0cf5434227c564cae5d3ae33a625d85"}
Oct 30 00:40:57.989220 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:57.989191    2572 generic.go:356] "Generic (PLEG): container finished" podID="098b604a-a614-4584-abc6-bfbe37ae40e9" containerID="0c775ac4c57b9c4d2bade838b2d0d064541ea1be33ae905a116e603268182bea" exitCode=0
Oct 30 00:40:57.989345 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:57.989242    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7mpt" event={"ID":"098b604a-a614-4584-abc6-bfbe37ae40e9","Type":"ContainerDied","Data":"0c775ac4c57b9c4d2bade838b2d0d064541ea1be33ae905a116e603268182bea"}
Oct 30 00:40:58.067395 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:58.067335    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5lmgn" podStartSLOduration=2.591421488 podStartE2EDuration="6.067313851s" totalImagesPullingTime="3.475892363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:58.023275422 +0000 UTC m=+6.632182437" watchObservedRunningTime="2026-10-30 00:40:58.067313851 +0000 UTC m=+6.676220867"
Oct 30 00:40:58.841991 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:58.841956    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:40:58.842264 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:58.842090    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rx" podUID="93171d29-b786-4586-80aa-9643856f242f"
Oct 30 00:40:58.994649 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:58.994617    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7zgwc" event={"ID":"f6870c47-e336-47b5-adfa-7143eed6d6ab","Type":"ContainerStarted","Data":"38dcc6aafb3af3a9895b0f1e79c99f684a13b36e129a99b528c9f2356f21df96"}
Oct 30 00:40:59.842405 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:59.842377    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:40:59.842582 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:59.842504    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nqtxx" podUID="6a4ab2cc-af6d-460b-b8dd-9f16078cc11d"
Oct 30 00:40:59.842889 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:59.842871    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:40:59.842978 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:40:59.842961    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw99d" podUID="372da31c-aac8-485a-bcc4-31a830f176a9"
Oct 30 00:40:59.998968 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:40:59.998937    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j" event={"ID":"57ac048f-898e-4478-87f5-251679ad7bcf","Type":"ContainerStarted","Data":"621684ec30bf983b0a7a320f40c55a5781eb60b96fe4fb74a1096e7b3a3c2bca"}
Oct 30 00:41:00.817251 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:00.817192    2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 00:41:00.841711 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:00.841683    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:00.841837 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:00.841809    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rx" podUID="93171d29-b786-4586-80aa-9643856f242f"
Oct 30 00:41:01.004584 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:01.004446    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j" event={"ID":"57ac048f-898e-4478-87f5-251679ad7bcf","Type":"ContainerStarted","Data":"19d5c5eaf3ced8d7d91dfc55aa15e27cabbff28536b6cb726dd0d2dbecaaa125"}
Oct 30 00:41:01.362330 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:01.361670    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5lmgn"
Oct 30 00:41:01.362651 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:01.362387    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5lmgn"
Oct 30 00:41:01.400852 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:01.400791    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7zgwc" podStartSLOduration=5.057778077 podStartE2EDuration="9.400773591s" totalImagesPullingTime="4.342995514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:59.034814038 +0000 UTC m=+7.643721055" watchObservedRunningTime="2026-10-30 00:41:01.400773591 +0000 UTC m=+10.009680607"
Oct 30 00:41:01.609952 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:01.609856    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:01.609952 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:01.609914    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs\") pod \"network-metrics-daemon-nqtxx\" (UID: \"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d\") " pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:01.609952 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:01.609952    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrrr\" (UniqueName: \"kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr\") pod \"network-check-target-qs6rx\" (UID: \"93171d29-b786-4586-80aa-9643856f242f\") " pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:01.610256 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:01.610092    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:41:01.610256 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:01.610110    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:41:01.610256 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:01.610120    2572 projected.go:196] Error preparing data for projected volume kube-api-access-9jrrr for pod openshift-network-diagnostics/network-check-target-qs6rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:41:01.610256 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:01.610198    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr podName:93171d29-b786-4586-80aa-9643856f242f nodeName:}" failed. No retries permitted until 2026-10-30 00:41:09.610179082 +0000 UTC m=+18.219086092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9jrrr" (UniqueName: "kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr") pod "network-check-target-qs6rx" (UID: "93171d29-b786-4586-80aa-9643856f242f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:41:01.610861 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:01.610633    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:41:01.610861 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:01.610687    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret podName:372da31c-aac8-485a-bcc4-31a830f176a9 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:09.61067221 +0000 UTC m=+18.219579216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret") pod "global-pull-secret-syncer-fw99d" (UID: "372da31c-aac8-485a-bcc4-31a830f176a9") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:41:01.610861 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:01.610745    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:41:01.610861 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:01.610783    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs podName:6a4ab2cc-af6d-460b-b8dd-9f16078cc11d nodeName:}" failed. No retries permitted until 2026-10-30 00:41:09.610771368 +0000 UTC m=+18.219678362 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs") pod "network-metrics-daemon-nqtxx" (UID: "6a4ab2cc-af6d-460b-b8dd-9f16078cc11d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:41:01.803671 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:01.803531    2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:41:00.817214982Z","UUID":"4ca68a89-8126-4506-92be-23539a94a904","Handler":null,"Name":"","Endpoint":""}
Oct 30 00:41:01.817832 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:01.817780    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 00:41:01.817970 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:01.817853    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 00:41:01.845608 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:01.845582    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:01.845882 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:01.845834    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:01.846134 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:01.846095    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nqtxx" podUID="6a4ab2cc-af6d-460b-b8dd-9f16078cc11d"
Oct 30 00:41:01.846298 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:01.846209    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw99d" podUID="372da31c-aac8-485a-bcc4-31a830f176a9"
Oct 30 00:41:02.009464 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:02.009428    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8k6rj" event={"ID":"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414","Type":"ContainerStarted","Data":"291757c3aafd2375c5e70bd08f0bb38c5b021676e20164034f53c2f9e627908c"}
Oct 30 00:41:02.012622 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:02.012597    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xjww4" event={"ID":"ae89a416-9d4c-4477-bf72-a5371f55701a","Type":"ContainerStarted","Data":"a549de83d01749c1982847c1c453bde3583a42f1bf813199d62b69181e451730"}
Oct 30 00:41:02.047590 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:02.047526    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-8k6rj" podStartSLOduration=3.652116303 podStartE2EDuration="11.047509913s" totalImagesPullingTime="7.39539361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:02.046998242 +0000 UTC m=+10.655905258" watchObservedRunningTime="2026-10-30 00:41:02.047509913 +0000 UTC m=+10.656416930"
Oct 30 00:41:02.089131 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:02.089091    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-xjww4" podStartSLOduration=4.067776214 podStartE2EDuration="11.08907577s" totalImagesPullingTime="7.021299556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:02.08849992 +0000 UTC m=+10.697406937" watchObservedRunningTime="2026-10-30 00:41:02.08907577 +0000 UTC m=+10.697982784"
Oct 30 00:41:02.842138 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:02.842103    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:02.842340 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:02.842262    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rx" podUID="93171d29-b786-4586-80aa-9643856f242f"
Oct 30 00:41:03.016940 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:03.016905    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j" event={"ID":"57ac048f-898e-4478-87f5-251679ad7bcf","Type":"ContainerStarted","Data":"24bf7c2fdfaf5b4357c9d0b71b39288538f475bc8b29619db0a8bfcacc4c0373"}
Oct 30 00:41:03.020514 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:03.020479    2572 generic.go:356] "Generic (PLEG): container finished" podID="098b604a-a614-4584-abc6-bfbe37ae40e9" containerID="7d7fb7a2251eb53a1ffa0f0199d8f1790df1832e7d2b01d1e7cebf4af29435f1" exitCode=0
Oct 30 00:41:03.020629 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:03.020563    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7mpt" event={"ID":"098b604a-a614-4584-abc6-bfbe37ae40e9","Type":"ContainerDied","Data":"7d7fb7a2251eb53a1ffa0f0199d8f1790df1832e7d2b01d1e7cebf4af29435f1"}
Oct 30 00:41:03.061042 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:03.060995    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j" podStartSLOduration=3.570553468 podStartE2EDuration="11.060979042s" totalImagesPullingTime="7.490425574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:03.060328599 +0000 UTC m=+11.669235623" watchObservedRunningTime="2026-10-30 00:41:03.060979042 +0000 UTC m=+11.669886057"
Oct 30 00:41:03.841578 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:03.841541    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:03.841713 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:03.841679    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nqtxx" podUID="6a4ab2cc-af6d-460b-b8dd-9f16078cc11d"
Oct 30 00:41:03.841772 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:03.841738    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:03.841872 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:03.841848    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw99d" podUID="372da31c-aac8-485a-bcc4-31a830f176a9"
Oct 30 00:41:04.024076 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:04.024038    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w22p8" event={"ID":"79677abd-0ee7-4081-b31e-9f07e913fd2d","Type":"ContainerStarted","Data":"18535eca4fb29fd2a7aee6a1a77483faad423727343597c212639f5cf5f64672"}
Oct 30 00:41:04.026777 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:04.026749    2572 generic.go:356] "Generic (PLEG): container finished" podID="098b604a-a614-4584-abc6-bfbe37ae40e9" containerID="36ea43fc43cea2085f9b8f2c66a56e4ed1949a9b5c1effd86cfbc1025e3aaee4" exitCode=0
Oct 30 00:41:04.026902 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:04.026828    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7mpt" event={"ID":"098b604a-a614-4584-abc6-bfbe37ae40e9","Type":"ContainerDied","Data":"36ea43fc43cea2085f9b8f2c66a56e4ed1949a9b5c1effd86cfbc1025e3aaee4"}
Oct 30 00:41:04.028420 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:04.028391    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nn559" event={"ID":"03bdb0b4-fc37-4886-93eb-9dc6ce910e63","Type":"ContainerStarted","Data":"061a49c82ec75bb0d9735b15076dc858ee4adef35970a4b5753d401e4f53d1b1"}
Oct 30 00:41:04.061385 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:04.061342    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w22p8" podStartSLOduration=5.543942207 podStartE2EDuration="13.061329704s" totalImagesPullingTime="7.517387497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:04.06082573 +0000 UTC m=+12.669732755" watchObservedRunningTime="2026-10-30 00:41:04.061329704 +0000 UTC m=+12.670236718"
Oct 30 00:41:04.147098 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:04.147044    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nn559" podStartSLOduration=3.999823851 podStartE2EDuration="13.147028248s" totalImagesPullingTime="9.147204397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:04.146646322 +0000 UTC m=+12.755553338" watchObservedRunningTime="2026-10-30 00:41:04.147028248 +0000 UTC m=+12.755935262"
Oct 30 00:41:04.841869 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:04.841841    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:04.842007 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:04.841953    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rx" podUID="93171d29-b786-4586-80aa-9643856f242f"
Oct 30 00:41:05.842076 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:05.842037    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:05.842950 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:05.842187    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nqtxx" podUID="6a4ab2cc-af6d-460b-b8dd-9f16078cc11d"
Oct 30 00:41:05.842950 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:05.842249    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:05.842950 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:05.842339    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw99d" podUID="372da31c-aac8-485a-bcc4-31a830f176a9"
Oct 30 00:41:06.035780 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:06.035743    2572 generic.go:356] "Generic (PLEG): container finished" podID="098b604a-a614-4584-abc6-bfbe37ae40e9" containerID="f0e934aeafe61a566b38ed36a03f25dcade0a5f1abd540c06ce5718277ec9e20" exitCode=0
Oct 30 00:41:06.035926 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:06.035794    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7mpt" event={"ID":"098b604a-a614-4584-abc6-bfbe37ae40e9","Type":"ContainerDied","Data":"f0e934aeafe61a566b38ed36a03f25dcade0a5f1abd540c06ce5718277ec9e20"}
Oct 30 00:41:06.841976 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:06.841949    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:06.842139 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:06.842072    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rx" podUID="93171d29-b786-4586-80aa-9643856f242f"
Oct 30 00:41:07.844072 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:07.843958    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:07.844465 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:07.844144    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nqtxx" podUID="6a4ab2cc-af6d-460b-b8dd-9f16078cc11d"
Oct 30 00:41:07.844465 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:07.844265    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:07.844465 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:07.844365    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw99d" podUID="372da31c-aac8-485a-bcc4-31a830f176a9"
Oct 30 00:41:08.044618 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:08.044589    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857b8" event={"ID":"d1bada76-8261-4ed7-adaf-8de93cf733c9","Type":"ContainerStarted","Data":"ef65a1a9ff6e07073a67b47aadb32ff65f286ba9de4efce39a5eece7c19d7f81"}
Oct 30 00:41:08.044725 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:08.044681    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857b8" event={"ID":"d1bada76-8261-4ed7-adaf-8de93cf733c9","Type":"ContainerStarted","Data":"efb38355643b86696eb363df563df6c24e75087480ced6f8bfb3d969aae797ac"}
Oct 30 00:41:08.044769 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:08.044732    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857b8" event={"ID":"d1bada76-8261-4ed7-adaf-8de93cf733c9","Type":"ContainerStarted","Data":"cfab4ba66fd914e9f657ba7a9ab1e9bd11ee654ffe665b503239b170a4fe4963"}
Oct 30 00:41:08.044808 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:08.044781    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857b8" event={"ID":"d1bada76-8261-4ed7-adaf-8de93cf733c9","Type":"ContainerStarted","Data":"6de185b728ebd2389dd18f327c1d40fdc957fde55b9eee8e5789ba7cebf144fd"}
Oct 30 00:41:08.047282 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:08.047261    2572 generic.go:356] "Generic (PLEG): container finished" podID="098b604a-a614-4584-abc6-bfbe37ae40e9" containerID="eb6f6ac2ce1d1900dcab4e3ab000bd89df65e73191aa6e0dd6843490b1ac156b" exitCode=0
Oct 30 00:41:08.047364 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:08.047293    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7mpt" event={"ID":"098b604a-a614-4584-abc6-bfbe37ae40e9","Type":"ContainerDied","Data":"eb6f6ac2ce1d1900dcab4e3ab000bd89df65e73191aa6e0dd6843490b1ac156b"}
Oct 30 00:41:08.841809 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:08.841634    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:08.842030 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:08.841873    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rx" podUID="93171d29-b786-4586-80aa-9643856f242f"
Oct 30 00:41:09.052002 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:09.051973    2572 generic.go:356] "Generic (PLEG): container finished" podID="098b604a-a614-4584-abc6-bfbe37ae40e9" containerID="65f580e412d0e6a61636c78011109d1af59d20c4f618b358be56a0d5b867a9a9" exitCode=0
Oct 30 00:41:09.052475 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:09.052052    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7mpt" event={"ID":"098b604a-a614-4584-abc6-bfbe37ae40e9","Type":"ContainerDied","Data":"65f580e412d0e6a61636c78011109d1af59d20c4f618b358be56a0d5b867a9a9"}
Oct 30 00:41:09.055284 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:09.055260    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857b8" event={"ID":"d1bada76-8261-4ed7-adaf-8de93cf733c9","Type":"ContainerStarted","Data":"0def1453aa39871a4a095a0a2a9c205703e38ed6e187570d2d30994fce731da7"}
Oct 30 00:41:09.055354 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:09.055319    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857b8" event={"ID":"d1bada76-8261-4ed7-adaf-8de93cf733c9","Type":"ContainerStarted","Data":"7913f595ffaabc405a7cc5fc4fdd9833128dbeb6bfad761ece5d92eb6a0996e8"}
Oct 30 00:41:09.667355 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:09.667314    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:09.667355 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:09.667362    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs\") pod \"network-metrics-daemon-nqtxx\" (UID: \"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d\") " pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:09.667612 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:09.667444    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:41:09.667612 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:09.667476    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrrr\" (UniqueName: \"kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr\") pod \"network-check-target-qs6rx\" (UID: \"93171d29-b786-4586-80aa-9643856f242f\") " pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:09.667612 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:09.667490    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:41:09.667612 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:09.667494    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs podName:6a4ab2cc-af6d-460b-b8dd-9f16078cc11d nodeName:}" failed. No retries permitted until 2026-10-30 00:41:25.667481669 +0000 UTC m=+34.276388662 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs") pod "network-metrics-daemon-nqtxx" (UID: "6a4ab2cc-af6d-460b-b8dd-9f16078cc11d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:41:09.667612 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:09.667554    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret podName:372da31c-aac8-485a-bcc4-31a830f176a9 nodeName:}" failed. No retries permitted until 2026-10-30 00:41:25.667539559 +0000 UTC m=+34.276446563 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret") pod "global-pull-secret-syncer-fw99d" (UID: "372da31c-aac8-485a-bcc4-31a830f176a9") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:41:09.667612 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:09.667571    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:41:09.667612 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:09.667585    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:41:09.667612 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:09.667593    2572 projected.go:196] Error preparing data for projected volume kube-api-access-9jrrr for pod openshift-network-diagnostics/network-check-target-qs6rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:41:09.667863 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:09.667625    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr podName:93171d29-b786-4586-80aa-9643856f242f nodeName:}" failed. No retries permitted until 2026-10-30 00:41:25.667616966 +0000 UTC m=+34.276523960 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9jrrr" (UniqueName: "kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr") pod "network-check-target-qs6rx" (UID: "93171d29-b786-4586-80aa-9643856f242f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:41:09.842368 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:09.842337    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:09.842513 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:09.842464    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nqtxx" podUID="6a4ab2cc-af6d-460b-b8dd-9f16078cc11d"
Oct 30 00:41:09.842513 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:09.842498    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:09.842583 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:09.842569    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw99d" podUID="372da31c-aac8-485a-bcc4-31a830f176a9"
Oct 30 00:41:10.060825 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:10.060794    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q7mpt" event={"ID":"098b604a-a614-4584-abc6-bfbe37ae40e9","Type":"ContainerStarted","Data":"1c134344312ce44ad44d5daf6b425942e87ad9ef0b7f6c1614931bab8ca260d0"}
Oct 30 00:41:10.841965 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:10.841936    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:10.842122 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:10.842028    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rx" podUID="93171d29-b786-4586-80aa-9643856f242f"
Oct 30 00:41:11.066440 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:11.066408    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857b8" event={"ID":"d1bada76-8261-4ed7-adaf-8de93cf733c9","Type":"ContainerStarted","Data":"810e32d8b90e7e6c2484abbe4f65cdad8b1a0bcbb822f8d5da4acaa8075e0e42"}
Oct 30 00:41:11.841974 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:11.841940    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:11.842174 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:11.842024    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw99d" podUID="372da31c-aac8-485a-bcc4-31a830f176a9"
Oct 30 00:41:11.842174 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:11.842118    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:11.842304 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:11.842233    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nqtxx" podUID="6a4ab2cc-af6d-460b-b8dd-9f16078cc11d"
Oct 30 00:41:12.841684 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:12.841641    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:12.842331 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:12.841753    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rx" podUID="93171d29-b786-4586-80aa-9643856f242f"
Oct 30 00:41:13.073620 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:13.073589    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-857b8" event={"ID":"d1bada76-8261-4ed7-adaf-8de93cf733c9","Type":"ContainerStarted","Data":"88262b842d435b2ae45e3b44df144b4bc87fa460eae454fa1a8eea7c8990ac16"}
Oct 30 00:41:13.073884 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:13.073866    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:13.073967 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:13.073929    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:13.074021 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:13.073968    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:13.088783 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:13.088627    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:13.088892 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:13.088849    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:13.159253 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:13.159182    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-857b8" podStartSLOduration=8.6726695 podStartE2EDuration="22.159171931s" totalImagesPullingTime="13.486502431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:13.147392981 +0000 UTC m=+21.756299996" watchObservedRunningTime="2026-10-30 00:41:13.159171931 +0000 UTC m=+21.768078937"
Oct 30 00:41:13.159459 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:13.159441    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q7mpt" podStartSLOduration=12.20428707 podStartE2EDuration="22.159435836s" totalImagesPullingTime="9.955148766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:40:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:10.127681822 +0000 UTC m=+18.736588826" watchObservedRunningTime="2026-10-30 00:41:13.159435836 +0000 UTC m=+21.768342850"
Oct 30 00:41:13.842264 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:13.842234    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:13.842714 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:13.842388    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw99d" podUID="372da31c-aac8-485a-bcc4-31a830f176a9"
Oct 30 00:41:13.842714 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:13.842422    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:13.842714 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:13.842500    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nqtxx" podUID="6a4ab2cc-af6d-460b-b8dd-9f16078cc11d"
Oct 30 00:41:14.377700 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:14.377636    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fw99d"]
Oct 30 00:41:14.377851 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:14.377725    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:14.377851 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:14.377800    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw99d" podUID="372da31c-aac8-485a-bcc4-31a830f176a9"
Oct 30 00:41:14.380042 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:14.380013    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qs6rx"]
Oct 30 00:41:14.380187 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:14.380132    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:14.380283 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:14.380261    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rx" podUID="93171d29-b786-4586-80aa-9643856f242f"
Oct 30 00:41:14.380605 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:14.380587    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nqtxx"]
Oct 30 00:41:14.380690 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:14.380673    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:14.380786 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:14.380769    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nqtxx" podUID="6a4ab2cc-af6d-460b-b8dd-9f16078cc11d"
Oct 30 00:41:15.841950 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:15.841922    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:15.841950 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:15.841935    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:15.842450 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:15.842052    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:15.842450 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:15.842055    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rx" podUID="93171d29-b786-4586-80aa-9643856f242f"
Oct 30 00:41:15.842450 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:15.842148    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nqtxx" podUID="6a4ab2cc-af6d-460b-b8dd-9f16078cc11d"
Oct 30 00:41:15.842450 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:15.842242    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw99d" podUID="372da31c-aac8-485a-bcc4-31a830f176a9"
Oct 30 00:41:17.841537 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:17.841507    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:17.841537 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:17.841521    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:17.842038 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:17.841518    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:17.842038 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:17.841597    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nqtxx" podUID="6a4ab2cc-af6d-460b-b8dd-9f16078cc11d"
Oct 30 00:41:17.842038 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:17.841660    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs6rx" podUID="93171d29-b786-4586-80aa-9643856f242f"
Oct 30 00:41:17.842038 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:17.841717    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fw99d" podUID="372da31c-aac8-485a-bcc4-31a830f176a9"
Oct 30 00:41:19.630929 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.630903    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeReady"
Oct 30 00:41:19.631237 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.631017    2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 00:41:19.687938 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.687915    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5b69x"]
Oct 30 00:41:19.689607 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.689592    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5b69x"
Oct 30 00:41:19.691499 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.691482    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-66wbz"]
Oct 30 00:41:19.692912 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.692896    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:19.703692 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.703669    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 00:41:19.703788 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.703748    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 00:41:19.703851 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.703817    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x7m5x\""
Oct 30 00:41:19.703851 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.703828    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 00:41:19.706239 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.706224    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 00:41:19.706311 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.706285    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 00:41:19.706387 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.706345    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l8pvs\""
Oct 30 00:41:19.715587 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.715568    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5b69x"]
Oct 30 00:41:19.717777 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.717760    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-66wbz"]
Oct 30 00:41:19.742002 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.741979    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:19.742075 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.742033    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ff8a2c-7788-4791-8212-84a9a683320c-config-volume\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:19.742075 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.742048    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v7fm\" (UniqueName: \"kubernetes.io/projected/e0ff8a2c-7788-4791-8212-84a9a683320c-kube-api-access-5v7fm\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:19.742195 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.742175    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4be6fff-4a26-4188-8ca7-80125df121d0-cert\") pod \"ingress-canary-5b69x\" (UID: \"e4be6fff-4a26-4188-8ca7-80125df121d0\") " pod="openshift-ingress-canary/ingress-canary-5b69x"
Oct 30 00:41:19.742233 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.742208    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fswtt\" (UniqueName: \"kubernetes.io/projected/e4be6fff-4a26-4188-8ca7-80125df121d0-kube-api-access-fswtt\") pod \"ingress-canary-5b69x\" (UID: \"e4be6fff-4a26-4188-8ca7-80125df121d0\") " pod="openshift-ingress-canary/ingress-canary-5b69x"
Oct 30 00:41:19.742233 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.742229    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ff8a2c-7788-4791-8212-84a9a683320c-tmp-dir\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:19.841853 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.841782    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:19.841853 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.841817    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:19.842013 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.841924    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:19.842525 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.842502    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:19.842617 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.842556    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ff8a2c-7788-4791-8212-84a9a683320c-config-volume\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:19.842673 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:19.842638    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:41:19.842718 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.842675    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v7fm\" (UniqueName: \"kubernetes.io/projected/e0ff8a2c-7788-4791-8212-84a9a683320c-kube-api-access-5v7fm\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:19.842718 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:19.842684    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls podName:e0ff8a2c-7788-4791-8212-84a9a683320c nodeName:}" failed. No retries permitted until 2026-10-30 00:41:20.34267243 +0000 UTC m=+28.951579427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls") pod "dns-default-66wbz" (UID: "e0ff8a2c-7788-4791-8212-84a9a683320c") : secret "dns-default-metrics-tls" not found
Oct 30 00:41:19.842835 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.842738    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4be6fff-4a26-4188-8ca7-80125df121d0-cert\") pod \"ingress-canary-5b69x\" (UID: \"e4be6fff-4a26-4188-8ca7-80125df121d0\") " pod="openshift-ingress-canary/ingress-canary-5b69x"
Oct 30 00:41:19.842835 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.842768    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fswtt\" (UniqueName: \"kubernetes.io/projected/e4be6fff-4a26-4188-8ca7-80125df121d0-kube-api-access-fswtt\") pod \"ingress-canary-5b69x\" (UID: \"e4be6fff-4a26-4188-8ca7-80125df121d0\") " pod="openshift-ingress-canary/ingress-canary-5b69x"
Oct 30 00:41:19.842835 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.842805    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ff8a2c-7788-4791-8212-84a9a683320c-tmp-dir\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:19.843129 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.843108    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ff8a2c-7788-4791-8212-84a9a683320c-tmp-dir\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:19.843230 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.843200    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ff8a2c-7788-4791-8212-84a9a683320c-config-volume\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:19.846757 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.846739    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4be6fff-4a26-4188-8ca7-80125df121d0-cert\") pod \"ingress-canary-5b69x\" (UID: \"e4be6fff-4a26-4188-8ca7-80125df121d0\") " pod="openshift-ingress-canary/ingress-canary-5b69x"
Oct 30 00:41:19.856211 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.856191    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 00:41:19.856306 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.856215    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 00:41:19.856306 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.856212    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 00:41:19.856417 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.856401    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z4qxk\""
Oct 30 00:41:19.856417 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.856413    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wwjm6\""
Oct 30 00:41:19.856499 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.856453    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wm8np\""
Oct 30 00:41:19.856499 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.856491    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 00:41:19.865655 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.865636    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v7fm\" (UniqueName: \"kubernetes.io/projected/e0ff8a2c-7788-4791-8212-84a9a683320c-kube-api-access-5v7fm\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:19.865787 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:19.865771    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fswtt\" (UniqueName: \"kubernetes.io/projected/e4be6fff-4a26-4188-8ca7-80125df121d0-kube-api-access-fswtt\") pod \"ingress-canary-5b69x\" (UID: \"e4be6fff-4a26-4188-8ca7-80125df121d0\") " pod="openshift-ingress-canary/ingress-canary-5b69x"
Oct 30 00:41:20.001604 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:20.001583    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5b69x"
Oct 30 00:41:20.300267 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:20.300076    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5b69x"]
Oct 30 00:41:20.303943 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:41:20.303917    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4be6fff_4a26_4188_8ca7_80125df121d0.slice/crio-8978da243611b66666b2d1025fdc3b3ee0185e684cf76dd7dcff2a933702ac8d WatchSource:0}: Error finding container 8978da243611b66666b2d1025fdc3b3ee0185e684cf76dd7dcff2a933702ac8d: Status 404 returned error can't find the container with id 8978da243611b66666b2d1025fdc3b3ee0185e684cf76dd7dcff2a933702ac8d
Oct 30 00:41:20.347397 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:20.347377    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:20.347534 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:20.347520    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:41:20.347577 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:20.347569    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls podName:e0ff8a2c-7788-4791-8212-84a9a683320c nodeName:}" failed. No retries permitted until 2026-10-30 00:41:21.347555901 +0000 UTC m=+29.956462894 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls") pod "dns-default-66wbz" (UID: "e0ff8a2c-7788-4791-8212-84a9a683320c") : secret "dns-default-metrics-tls" not found
Oct 30 00:41:21.091663 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:21.091622    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5b69x" event={"ID":"e4be6fff-4a26-4188-8ca7-80125df121d0","Type":"ContainerStarted","Data":"8978da243611b66666b2d1025fdc3b3ee0185e684cf76dd7dcff2a933702ac8d"}
Oct 30 00:41:21.353819 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:21.353739    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:21.353970 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:21.353917    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:41:21.354022 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:21.353973    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls podName:e0ff8a2c-7788-4791-8212-84a9a683320c nodeName:}" failed. No retries permitted until 2026-10-30 00:41:23.353958772 +0000 UTC m=+31.962865765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls") pod "dns-default-66wbz" (UID: "e0ff8a2c-7788-4791-8212-84a9a683320c") : secret "dns-default-metrics-tls" not found
Oct 30 00:41:21.538710 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:21.538667    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5lmgn"
Oct 30 00:41:21.538871 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:21.538812    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:41:21.539337 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:21.539314    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5lmgn"
Oct 30 00:41:22.940921 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:22.940894    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jz4ff"]
Oct 30 00:41:22.978886 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:22.978866    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jz4ff"]
Oct 30 00:41:22.978993 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:22.978975    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:23.064721 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.064692    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:23.064852 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.064746    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz7mb\" (UniqueName: \"kubernetes.io/projected/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-api-access-sz7mb\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:23.064852 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.064773    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3822276d-acd5-4f16-ac86-32e8978c1f9f-crio-socket\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:23.064852 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.064822    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:23.064968 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.064859    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3822276d-acd5-4f16-ac86-32e8978c1f9f-data-volume\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:23.065527 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.065511    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 00:41:23.065676 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.065665    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 00:41:23.065707 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.065689    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 00:41:23.065896 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.065878    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 00:41:23.065976 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.065960    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qs9gb\""
Oct 30 00:41:23.097698 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.097661    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5b69x" event={"ID":"e4be6fff-4a26-4188-8ca7-80125df121d0","Type":"ContainerStarted","Data":"2ff8114d889a1f8325c5b8664e5037c2362859f85def6c3a3abfc9f1759ac64a"}
Oct 30 00:41:23.137039 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.136977    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5b69x" podStartSLOduration=1.506618049 podStartE2EDuration="4.136958662s" totalImagesPullingTime="2.630340613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:23.135732294 +0000 UTC m=+31.744639303" watchObservedRunningTime="2026-10-30 00:41:23.136958662 +0000 UTC m=+31.745865680"
Oct 30 00:41:23.165484 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.165447    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:23.165633 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.165554    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz7mb\" (UniqueName: \"kubernetes.io/projected/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-api-access-sz7mb\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:23.165633 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.165581    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3822276d-acd5-4f16-ac86-32e8978c1f9f-crio-socket\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:23.165633 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:23.165596    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:23.165633 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.165605    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:23.165633 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.165630    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3822276d-acd5-4f16-ac86-32e8978c1f9f-data-volume\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:23.165907 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:23.165668    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls podName:3822276d-acd5-4f16-ac86-32e8978c1f9f nodeName:}" failed. No retries permitted until 2026-10-30 00:41:23.665648672 +0000 UTC m=+32.274555666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jz4ff" (UID: "3822276d-acd5-4f16-ac86-32e8978c1f9f") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:23.165907 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.165806    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3822276d-acd5-4f16-ac86-32e8978c1f9f-crio-socket\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:23.165907 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:23.165905    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:41:23.166070 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:23.165956    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm podName:3822276d-acd5-4f16-ac86-32e8978c1f9f nodeName:}" failed. No retries permitted until 2026-10-30 00:41:23.665941239 +0000 UTC m=+32.274848266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jz4ff" (UID: "3822276d-acd5-4f16-ac86-32e8978c1f9f") : configmap "kube-rbac-proxy" not found
Oct 30 00:41:23.166070 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.165974    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3822276d-acd5-4f16-ac86-32e8978c1f9f-data-volume\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:23.188541 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.188482    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz7mb\" (UniqueName: \"kubernetes.io/projected/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-api-access-sz7mb\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:23.353547 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.353518    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xkblh"]
Oct 30 00:41:23.366901 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.366875    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:23.366992 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:23.366982    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:41:23.367039 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:23.367030    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls podName:e0ff8a2c-7788-4791-8212-84a9a683320c nodeName:}" failed. No retries permitted until 2026-10-30 00:41:27.367015748 +0000 UTC m=+35.975922741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls") pod "dns-default-66wbz" (UID: "e0ff8a2c-7788-4791-8212-84a9a683320c") : secret "dns-default-metrics-tls" not found
Oct 30 00:41:23.370242 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.370223    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkblh"
Oct 30 00:41:23.376740 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.376721    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xkblh"]
Oct 30 00:41:23.388009 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.387991    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 00:41:23.388131 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.388114    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:23.388303 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.388287    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kdklf\""
Oct 30 00:41:23.467293 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.467244    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqngb\" (UniqueName: \"kubernetes.io/projected/cb1e136c-a300-45e4-9031-ea39904518a5-kube-api-access-bqngb\") pod \"migrator-66df5d496b-xkblh\" (UID: \"cb1e136c-a300-45e4-9031-ea39904518a5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkblh"
Oct 30 00:41:23.568603 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.568579    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqngb\" (UniqueName: \"kubernetes.io/projected/cb1e136c-a300-45e4-9031-ea39904518a5-kube-api-access-bqngb\") pod \"migrator-66df5d496b-xkblh\" (UID: \"cb1e136c-a300-45e4-9031-ea39904518a5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkblh"
Oct 30 00:41:23.590863 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.590841    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqngb\" (UniqueName: \"kubernetes.io/projected/cb1e136c-a300-45e4-9031-ea39904518a5-kube-api-access-bqngb\") pod \"migrator-66df5d496b-xkblh\" (UID: \"cb1e136c-a300-45e4-9031-ea39904518a5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkblh"
Oct 30 00:41:23.669282 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.669263    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:23.669363 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.669308    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:23.669422 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:23.669404    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:23.669468 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:23.669459    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls podName:3822276d-acd5-4f16-ac86-32e8978c1f9f nodeName:}" failed. No retries permitted until 2026-10-30 00:41:24.669446769 +0000 UTC m=+33.278353762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jz4ff" (UID: "3822276d-acd5-4f16-ac86-32e8978c1f9f") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:23.669508 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:23.669410    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:41:23.669545 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:23.669537    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm podName:3822276d-acd5-4f16-ac86-32e8978c1f9f nodeName:}" failed. No retries permitted until 2026-10-30 00:41:24.669524908 +0000 UTC m=+33.278431917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jz4ff" (UID: "3822276d-acd5-4f16-ac86-32e8978c1f9f") : configmap "kube-rbac-proxy" not found
Oct 30 00:41:23.692413 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.692391    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkblh"
Oct 30 00:41:23.995974 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:23.995946    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xkblh"]
Oct 30 00:41:23.999261 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:41:23.999234    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1e136c_a300_45e4_9031_ea39904518a5.slice/crio-1889d2bf99a8f6863b3403d36d4885593394551741115b3026723649b9bf013f WatchSource:0}: Error finding container 1889d2bf99a8f6863b3403d36d4885593394551741115b3026723649b9bf013f: Status 404 returned error can't find the container with id 1889d2bf99a8f6863b3403d36d4885593394551741115b3026723649b9bf013f
Oct 30 00:41:24.100719 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:24.100690    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkblh" event={"ID":"cb1e136c-a300-45e4-9031-ea39904518a5","Type":"ContainerStarted","Data":"1889d2bf99a8f6863b3403d36d4885593394551741115b3026723649b9bf013f"}
Oct 30 00:41:24.678280 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:24.678205    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:24.678280 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:24.678258    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:24.678551 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:24.678326    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:41:24.678551 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:24.678353    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:24.678551 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:24.678377    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm podName:3822276d-acd5-4f16-ac86-32e8978c1f9f nodeName:}" failed. No retries permitted until 2026-10-30 00:41:26.678365111 +0000 UTC m=+35.287272105 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jz4ff" (UID: "3822276d-acd5-4f16-ac86-32e8978c1f9f") : configmap "kube-rbac-proxy" not found
Oct 30 00:41:24.678551 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:24.678404    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls podName:3822276d-acd5-4f16-ac86-32e8978c1f9f nodeName:}" failed. No retries permitted until 2026-10-30 00:41:26.678389804 +0000 UTC m=+35.287296827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jz4ff" (UID: "3822276d-acd5-4f16-ac86-32e8978c1f9f") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:25.687501 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:25.687446    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:25.687501 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:25.687481    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs\") pod \"network-metrics-daemon-nqtxx\" (UID: \"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d\") " pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:25.687836 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:25.687511    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrrr\" (UniqueName: \"kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr\") pod \"network-check-target-qs6rx\" (UID: \"93171d29-b786-4586-80aa-9643856f242f\") " pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:25.687836 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:25.687597    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:41:25.687836 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:25.687666    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs podName:6a4ab2cc-af6d-460b-b8dd-9f16078cc11d nodeName:}" failed. No retries permitted until 2026-10-30 00:41:57.68764711 +0000 UTC m=+66.296554118 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs") pod "network-metrics-daemon-nqtxx" (UID: "6a4ab2cc-af6d-460b-b8dd-9f16078cc11d") : secret "metrics-daemon-secret" not found
Oct 30 00:41:25.689735 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:25.689713    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jrrr\" (UniqueName: \"kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr\") pod \"network-check-target-qs6rx\" (UID: \"93171d29-b786-4586-80aa-9643856f242f\") " pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:25.689825 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:25.689813    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:25.856686 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:25.856663    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:25.862817 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:25.862799    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:26.106609 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:26.106573    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkblh" event={"ID":"cb1e136c-a300-45e4-9031-ea39904518a5","Type":"ContainerStarted","Data":"96056eb36aa6aae99117c817b8f48f47682327ff1025745961392ba10f068f12"}
Oct 30 00:41:26.145368 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:26.145330    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkblh" podStartSLOduration=1.710170873 podStartE2EDuration="3.145315627s" totalImagesPullingTime="1.435144754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:26.144477368 +0000 UTC m=+34.753384383" watchObservedRunningTime="2026-10-30 00:41:26.145315627 +0000 UTC m=+34.754222641"
Oct 30 00:41:26.181802 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:26.181776    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fw99d"]
Oct 30 00:41:26.184253 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:41:26.184225    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372da31c_aac8_485a_bcc4_31a830f176a9.slice/crio-090aee5914485b8a7823a90d1cb665f9e8275a963aa2addf26545f81c8534d46 WatchSource:0}: Error finding container 090aee5914485b8a7823a90d1cb665f9e8275a963aa2addf26545f81c8534d46: Status 404 returned error can't find the container with id 090aee5914485b8a7823a90d1cb665f9e8275a963aa2addf26545f81c8534d46
Oct 30 00:41:26.221020 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:26.220974    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qs6rx"]
Oct 30 00:41:26.223508 ip-10-0-133-32 kubenswrapper[2572]: W1030 00:41:26.223485    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93171d29_b786_4586_80aa_9643856f242f.slice/crio-5938488a15873da121c350bd2a0573176e777b9f49964e3872708c721fad2ae9 WatchSource:0}: Error finding container 5938488a15873da121c350bd2a0573176e777b9f49964e3872708c721fad2ae9: Status 404 returned error can't find the container with id 5938488a15873da121c350bd2a0573176e777b9f49964e3872708c721fad2ae9
Oct 30 00:41:26.695251 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:26.695213    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:26.695634 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:26.695305    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:26.695634 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:26.695343    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:41:26.695634 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:26.695396    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:26.695634 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:26.695421    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm podName:3822276d-acd5-4f16-ac86-32e8978c1f9f nodeName:}" failed. No retries permitted until 2026-10-30 00:41:30.695401157 +0000 UTC m=+39.304308152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jz4ff" (UID: "3822276d-acd5-4f16-ac86-32e8978c1f9f") : configmap "kube-rbac-proxy" not found
Oct 30 00:41:26.695634 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:26.695441    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls podName:3822276d-acd5-4f16-ac86-32e8978c1f9f nodeName:}" failed. No retries permitted until 2026-10-30 00:41:30.695429306 +0000 UTC m=+39.304336299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jz4ff" (UID: "3822276d-acd5-4f16-ac86-32e8978c1f9f") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:27.110935 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:27.110899    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qs6rx" event={"ID":"93171d29-b786-4586-80aa-9643856f242f","Type":"ContainerStarted","Data":"5938488a15873da121c350bd2a0573176e777b9f49964e3872708c721fad2ae9"}
Oct 30 00:41:27.112663 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:27.112633    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fw99d" event={"ID":"372da31c-aac8-485a-bcc4-31a830f176a9","Type":"ContainerStarted","Data":"090aee5914485b8a7823a90d1cb665f9e8275a963aa2addf26545f81c8534d46"}
Oct 30 00:41:27.400600 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:27.400522    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:27.400769 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:27.400702    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:41:27.400831 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:27.400797    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls podName:e0ff8a2c-7788-4791-8212-84a9a683320c nodeName:}" failed. No retries permitted until 2026-10-30 00:41:35.400758402 +0000 UTC m=+44.009665396 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls") pod "dns-default-66wbz" (UID: "e0ff8a2c-7788-4791-8212-84a9a683320c") : secret "dns-default-metrics-tls" not found
Oct 30 00:41:29.805483 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:29.805454    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8k6rj_f8b80e7e-75db-4e4e-9c1b-0d2f4b754414/dns-node-resolver/0.log"
Oct 30 00:41:30.122077 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:30.121991    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qs6rx" event={"ID":"93171d29-b786-4586-80aa-9643856f242f","Type":"ContainerStarted","Data":"d4c5e040bc646a22e2c6936329fba8b31c577a9e025519c456d8adf113c00ed9"}
Oct 30 00:41:30.122245 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:30.122179    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:30.724690 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:30.724659    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:30.724849 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:30.724707    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:30.724849 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:30.724815    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:30.724929 ip-10-0-133-32 kubenswrapper[2572]: E1030 00:41:30.724884    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls podName:3822276d-acd5-4f16-ac86-32e8978c1f9f nodeName:}" failed. No retries permitted until 2026-10-30 00:41:38.724866743 +0000 UTC m=+47.333773737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jz4ff" (UID: "3822276d-acd5-4f16-ac86-32e8978c1f9f") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:30.725173 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:30.725140    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:30.812425 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:30.812403    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7zgwc_f6870c47-e336-47b5-adfa-7143eed6d6ab/node-ca/0.log"
Oct 30 00:41:31.126070 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:31.126028    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fw99d" event={"ID":"372da31c-aac8-485a-bcc4-31a830f176a9","Type":"ContainerStarted","Data":"74ec5bd4f49b7ddd5e3e0a6b7685bb90dfd8c605b48270a62be8e25f55bdb068"}
Oct 30 00:41:31.173692 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:31.173646    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qs6rx" podStartSLOduration=35.51801204 podStartE2EDuration="39.17363147s" totalImagesPullingTime="3.65561943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:30.179585132 +0000 UTC m=+38.788492147" watchObservedRunningTime="2026-10-30 00:41:31.17363147 +0000 UTC m=+39.782538484"
Oct 30 00:41:31.182120 ip-10-0-133-32 kubenswrapper[2572]: I1030 00:41:31.181976    2572 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:41:31.182129 ip-10-0-133-32 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 00:41:31.197481 ip-10-0-133-32 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 00:41:31.197644 ip-10-0-133-32 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 00:41:31.197689 ip-10-0-133-32 systemd[1]: kubelet.service: Consumed 2.145s CPU time.
Oct 30 00:41:31.215609 ip-10-0-133-32 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 00:41:31.337677 ip-10-0-133-32 kubenswrapper[4425]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:41:31.337677 ip-10-0-133-32 kubenswrapper[4425]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 00:41:31.337677 ip-10-0-133-32 kubenswrapper[4425]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:41:31.337677 ip-10-0-133-32 kubenswrapper[4425]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:41:31.339974 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.339894    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:31.339974 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.339969    4425 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:31.339974 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.339974    4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:31.339974 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.339978    4425 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.339982    4425 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.339986    4425 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.339990    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.339993    4425 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.339997    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340001    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340005    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340008    4425 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340012    4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340016    4425 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340020    4425 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340025    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340029    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340033    4425 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340037    4425 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340041    4425 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340055    4425 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340059    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:31.340245 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340062    4425 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340066    4425 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340070    4425 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340074    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340078    4425 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340081    4425 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340085    4425 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340089    4425 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340093    4425 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340097    4425 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340101    4425 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340108    4425 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340112    4425 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340117    4425 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340122    4425 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340128    4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340132    4425 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340140    4425 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340145    4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340149    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:31.341043 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340171    4425 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340177    4425 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340183    4425 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340188    4425 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340192    4425 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340196    4425 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340200    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340204    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340208    4425 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340213    4425 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340217    4425 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340221    4425 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340225    4425 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340230    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340235    4425 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340240    4425 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340244    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340248    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340252    4425 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340256    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:31.342183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340260    4425 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340265    4425 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340269    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340274    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340278    4425 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340282    4425 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340287    4425 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340292    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340296    4425 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340300    4425 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340308    4425 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340315    4425 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340320    4425 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340326    4425 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340330    4425 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340335    4425 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340339    4425 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340344    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340348    4425 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:31.343101 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340352    4425 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340358    4425 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340362    4425 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340366    4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340371    4425 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340375    4425 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340380    4425 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340384    4425 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340388    4425 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340392    4425 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340396    4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340400    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340404    4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340408    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340413    4425 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340416    4425 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340421    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340425    4425 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340429    4425 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340434    4425 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:31.343937 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340438    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340442    4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340447    4425 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340451    4425 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.340456    4425 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341655    4425 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341667    4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341675    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341679    4425 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341684    4425 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341688    4425 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341692    4425 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341696    4425 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341700    4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341704    4425 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341709    4425 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341713    4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341717    4425 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341722    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341726    4425 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:31.344808 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341730    4425 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341734    4425 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341738    4425 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341742    4425 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341746    4425 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341750    4425 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341754    4425 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341758    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341762    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341767    4425 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341771    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341775    4425 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341780    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341784    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341788    4425 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341793    4425 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341797    4425 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341802    4425 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341806    4425 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341812    4425 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:31.345700 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341817    4425 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341821    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341825    4425 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341829    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341834    4425 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341838    4425 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341842    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341846    4425 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341850    4425 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341854    4425 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341858    4425 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341862    4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341866    4425 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341870    4425 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341875    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341879    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341883    4425 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341887    4425 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341891    4425 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:31.346410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341895    4425 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341900    4425 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341904    4425 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341908    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341912    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341916    4425 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341922    4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341926    4425 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341933    4425 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341938    4425 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341942    4425 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341951    4425 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341958    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341962    4425 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341967    4425 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341971    4425 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341975    4425 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341979    4425 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341983    4425 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:31.346872 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341987    4425 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341992    4425 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.341995    4425 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342000    4425 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342004    4425 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342009    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342013    4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342017    4425 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342021    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342025    4425 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342029    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342034    4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342040    4425 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342046    4425 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342053    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342057    4425 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342061    4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342066    4425 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342070    4425 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:31.347346 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342075    4425 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342079    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342082    4425 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342086    4425 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342090    4425 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342095    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342109    4425 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342113    4425 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342117    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342121    4425 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342125    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342129    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342135    4425 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342140    4425 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342978    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342987    4425 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342992    4425 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.342996    4425 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343001    4425 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343005    4425 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:31.347860 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343010    4425 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343014    4425 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343018    4425 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343022    4425 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343027    4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343032    4425 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343036    4425 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343040    4425 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343044    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343048    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343052    4425 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343056    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343060    4425 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343064    4425 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343069    4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343074    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343078    4425 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343082    4425 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343087    4425 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343091    4425 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:31.348640 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343096    4425 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343099    4425 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343103    4425 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343107    4425 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343111    4425 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343115    4425 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343122    4425 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343127    4425 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343133    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343138    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343142    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343147    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343168    4425 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343173    4425 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343177    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343181    4425 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343186    4425 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343190    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343194    4425 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:31.349399 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343198    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343202    4425 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343206    4425 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343209    4425 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343214    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343218    4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343222    4425 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343226    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343231    4425 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343235    4425 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343239    4425 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343243    4425 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343248    4425 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343253    4425 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343257    4425 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343262    4425 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343267    4425 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343272    4425 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343276    4425 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:31.350202 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343280    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343284    4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343288    4425 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343293    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343297    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343301    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343306    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343310    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343314    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343318    4425 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343322    4425 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343326    4425 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343330    4425 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343337    4425 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343344    4425 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343349    4425 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343353    4425 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343357    4425 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343361    4425 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:31.351062 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343365    4425 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343374    4425 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343378    4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343383    4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343387    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343391    4425 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343395    4425 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343401    4425 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343405    4425 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343409    4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343413    4425 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343420    4425 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343424    4425 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343428    4425 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343432    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343436    4425 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343440    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343445    4425 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343449    4425 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343453    4425 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343457    4425 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:31.351905 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343461    4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.343465    4425 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343601    4425 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343612    4425 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343621    4425 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343628    4425 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343635    4425 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343640    4425 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343647    4425 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343653    4425 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343658    4425 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343663    4425 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343671    4425 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343676    4425 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343681    4425 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343685    4425 flags.go:64] FLAG: --cgroup-root=""
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343690    4425 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343694    4425 flags.go:64] FLAG: --client-ca-file=""
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343700    4425 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343705    4425 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343711    4425 flags.go:64] FLAG: --cluster-domain=""
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343715    4425 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343720    4425 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343725    4425 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 00:41:31.352847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343730    4425 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343737    4425 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343742    4425 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343747    4425 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343752    4425 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343756    4425 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343761    4425 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343766    4425 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343770    4425 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343775    4425 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343782    4425 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343787    4425 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343791    4425 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343796    4425 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343801    4425 flags.go:64] FLAG: --enable-server="true"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343807    4425 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343814    4425 flags.go:64] FLAG: --event-burst="100"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343819    4425 flags.go:64] FLAG: --event-qps="50"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343824    4425 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343829    4425 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343836    4425 flags.go:64] FLAG: --eviction-hard=""
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343842    4425 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343847    4425 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343852    4425 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343857    4425 flags.go:64] FLAG: --eviction-soft=""
Oct 30 00:41:31.353892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343861    4425 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343866    4425 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343871    4425 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343876    4425 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343881    4425 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343885    4425 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343890    4425 flags.go:64] FLAG: --feature-gates=""
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343896    4425 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343901    4425 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343906    4425 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343911    4425 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343916    4425 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343921    4425 flags.go:64] FLAG: --help="false"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343926    4425 flags.go:64] FLAG: --hostname-override="ip-10-0-133-32.ec2.internal"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343931    4425 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343936    4425 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343940    4425 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343945    4425 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343950    4425 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343954    4425 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343958    4425 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343963    4425 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343970    4425 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 00:41:31.354971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343975    4425 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343980    4425 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343984    4425 flags.go:64] FLAG: --kube-reserved=""
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343989    4425 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.343995    4425 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344000    4425 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344004    4425 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344009    4425 flags.go:64] FLAG: --lock-file=""
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344013    4425 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344018    4425 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344023    4425 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344031    4425 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344035    4425 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344041    4425 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344045    4425 flags.go:64] FLAG: --logging-format="text"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344050    4425 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344055    4425 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344060    4425 flags.go:64] FLAG: --manifest-url=""
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344065    4425 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344072    4425 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344077    4425 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344082    4425 flags.go:64] FLAG: --max-pods="110"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344087    4425 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344092    4425 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344097    4425 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 00:41:31.355934 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344101    4425 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344105    4425 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344110    4425 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344115    4425 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344126    4425 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344130    4425 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344137    4425 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344143    4425 flags.go:64] FLAG: --pod-cidr=""
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344147    4425 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344167    4425 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344173    4425 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344180    4425 flags.go:64] FLAG: --port="10250"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344184    4425 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344189    4425 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-088f2a0bb20a306cc"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344195    4425 flags.go:64] FLAG: --qos-reserved=""
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344199    4425 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344204    4425 flags.go:64] FLAG: --register-node="true"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344209    4425 flags.go:64] FLAG: --register-with-taints=""
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344214    4425 flags.go:64] FLAG: --registry-burst="10"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344219    4425 flags.go:64] FLAG: --registry-qps="5"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344224    4425 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344234    4425 flags.go:64] FLAG: --reserved-memory=""
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344249    4425 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344255    4425 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344259    4425 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344264    4425 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 00:41:31.357146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344269    4425 flags.go:64] FLAG: --runonce="false"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344273    4425 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344278    4425 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344283    4425 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344288    4425 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344292    4425 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344297    4425 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344302    4425 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344307    4425 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344311    4425 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344316    4425 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344320    4425 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344327    4425 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344332    4425 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344337    4425 flags.go:64] FLAG: --system-cgroups=""
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344341    4425 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344350    4425 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344356    4425 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344360    4425 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344367    4425 flags.go:64] FLAG: --tls-min-version=""
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344371    4425 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344377    4425 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344381    4425 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344386    4425 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344391    4425 flags.go:64] FLAG: --v="2"
Oct 30 00:41:31.357993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344397    4425 flags.go:64] FLAG: --version="false"
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344403    4425 flags.go:64] FLAG: --vmodule=""
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344410    4425 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.344416    4425 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344620    4425 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344627    4425 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344633    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344637    4425 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344641    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344646    4425 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344650    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344654    4425 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344659    4425 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344663    4425 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344667    4425 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344671    4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344676    4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344680    4425 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344683    4425 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344689    4425 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:31.358727 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344694    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344699    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344703    4425 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344707    4425 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344714    4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344718    4425 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344722    4425 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344726    4425 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344730    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344736    4425 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344743    4425 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344748    4425 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344752    4425 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344757    4425 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344761    4425 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344765    4425 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344770    4425 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344774    4425 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344779    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344783    4425 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:31.359549 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344788    4425 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344792    4425 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344796    4425 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344800    4425 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344804    4425 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344808    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344812    4425 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344816    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344820    4425 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344823    4425 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344827    4425 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344833    4425 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344838    4425 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344842    4425 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344846    4425 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344851    4425 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344856    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344861    4425 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344865    4425 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:31.360384 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344869    4425 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344873    4425 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344877    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344882    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344886    4425 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344891    4425 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344895    4425 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344899    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344904    4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344908    4425 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344913    4425 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344917    4425 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344922    4425 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344926    4425 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344930    4425 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344935    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344939    4425 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344943    4425 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344947    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:31.360999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344951    4425 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344955    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344959    4425 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344963    4425 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344968    4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344974    4425 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344978    4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344984    4425 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344989    4425 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.344994    4425 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345001    4425 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345006    4425 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345010    4425 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345014    4425 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345018    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345023    4425 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345027    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345031    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345035    4425 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:31.361527 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345039    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:31.361993 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345044    4425 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:31.361993 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345048    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:31.361993 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345052    4425 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:31.361993 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345056    4425 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:31.361993 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345061    4425 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:31.361993 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345065    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:31.361993 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345069    4425 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:31.361993 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345073    4425 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:31.361993 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345077    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:31.361993 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345081    4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:31.361993 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345085    4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:31.361993 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345090    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:31.361993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.345135    4425 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:41:31.361993 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345212    4425 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:31.361993 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345220    4425 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:31.361993 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345225    4425 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345230    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345234    4425 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345243    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345248    4425 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345252    4425 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345257    4425 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345263    4425 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345267    4425 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345271    4425 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345275    4425 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345279    4425 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345283    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345288    4425 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345292    4425 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345296    4425 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345301    4425 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345305    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345309    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:31.362394 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345313    4425 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345317    4425 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345321    4425 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345327    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345331    4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345335    4425 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345339    4425 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345344    4425 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345348    4425 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345352    4425 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345356    4425 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345359    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345364    4425 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345368    4425 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345372    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345376    4425 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345383    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345387    4425 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345391    4425 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345395    4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:31.362853 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345401    4425 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345406    4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345413    4425 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345419    4425 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345423    4425 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345428    4425 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345432    4425 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345436    4425 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345440    4425 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345444    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345448    4425 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345452    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345457    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345462    4425 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345466    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345470    4425 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345479    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345483    4425 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345487    4425 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345491    4425 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:31.363395 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345495    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345499    4425 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345504    4425 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345508    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345512    4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345516    4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345520    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345524    4425 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345530    4425 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345536    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345541    4425 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345545    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345551    4425 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345555    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345559    4425 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345564    4425 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345568    4425 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345572    4425 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345576    4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345580    4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:31.363868 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345584    4425 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345589    4425 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345593    4425 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345598    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345602    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345606    4425 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345610    4425 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345614    4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345619    4425 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345624    4425 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345628    4425 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345633    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345639    4425 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345646    4425 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345650    4425 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345656    4425 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345661    4425 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345666    4425 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345670    4425 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345674    4425 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:31.364353 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345680    4425 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345684    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345690    4425 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345694    4425 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.345699    4425 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.351042    4425 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.351060    4425 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.351083    4425 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.351091    4425 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351196    4425 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351205    4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351210    4425 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351214    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351219    4425 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351224    4425 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351228    4425 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351233    4425 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351237    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351241    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:31.364831 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351245    4425 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351249    4425 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351254    4425 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351258    4425 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351262    4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351266    4425 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351270    4425 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351275    4425 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351279    4425 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351283    4425 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351288    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351292    4425 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351296    4425 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351301    4425 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351306    4425 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351310    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351314    4425 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351319    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351324    4425 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351328    4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:31.365296 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351333    4425 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351337    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351341    4425 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351345    4425 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351350    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351354    4425 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351358    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351362    4425 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351366    4425 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351370    4425 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351375    4425 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351379    4425 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351383    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351387    4425 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351392    4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351396    4425 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351401    4425 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351405    4425 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351411    4425 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:31.365761 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351417    4425 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351422    4425 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351427    4425 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351432    4425 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351436    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351440    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351445    4425 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351450    4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351454    4425 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351458    4425 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351462    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351466    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351471    4425 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351475    4425 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351480    4425 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351484    4425 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351488    4425 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351492    4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351496    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:31.366218 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351500    4425 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351505    4425 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351510    4425 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351514    4425 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351518    4425 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351522    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351526    4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351531    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351535    4425 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351539    4425 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351543    4425 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351548    4425 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351553    4425 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351557    4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351563    4425 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351569    4425 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351573    4425 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351577    4425 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351581    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:31.366706 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351585    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351590    4425 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351594    4425 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351598    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351602    4425 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351606    4425 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351611    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351615    4425 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351620    4425 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351624    4425 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351628    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351632    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351636    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351641    4425 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351645    4425 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351649    4425 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351653    4425 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351657    4425 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:31.367146 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351661    4425 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:31.367582 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.351703    4425 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:41:31.367582 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351946    4425 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:31.367582 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351956    4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:31.367582 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351962    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:31.367582 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351966    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:31.367582 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351971    4425 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:31.367582 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351976    4425 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:31.367582 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351981    4425 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:31.367582 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351985    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:31.367582 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351989    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:31.367582 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351993    4425 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:31.367582 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.351997    4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:31.367582 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352001    4425 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:31.367582 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352005    4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:31.367582 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352010    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:31.367582 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352014    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352018    4425 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352022    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352027    4425 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352031    4425 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352035    4425 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352039    4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352043    4425 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352047    4425 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352051    4425 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352056    4425 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352060    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352065    4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352070    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352074    4425 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352078    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352082    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352086    4425 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352090    4425 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:31.367957 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352094    4425 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352099    4425 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352103    4425 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352108    4425 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352111    4425 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352118    4425 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352124    4425 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352129    4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352133    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352137    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352141    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352145    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352150    4425 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352170    4425 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352174    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352179    4425 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352183    4425 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352187    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352191    4425 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352195    4425 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:31.368421 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352199    4425 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352203    4425 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352208    4425 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352212    4425 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352216    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352220    4425 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352224    4425 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352228    4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352233    4425 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352237    4425 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352241    4425 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352245    4425 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352249    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352252    4425 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352256    4425 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352260    4425 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352264    4425 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352268    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352274    4425 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352279    4425 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:31.369183 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352283    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352287    4425 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352291    4425 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352294    4425 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352299    4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352303    4425 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352307    4425 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352311    4425 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352318    4425 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352325    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352329    4425 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352334    4425 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352342    4425 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352348    4425 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352352    4425 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352357    4425 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352361    4425 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352366    4425 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352370    4425 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:31.369798 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352375    4425 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:31.370406 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352380    4425 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:31.370406 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352384    4425 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:31.370406 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352388    4425 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:31.370406 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352393    4425 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:31.370406 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352397    4425 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:31.370406 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352401    4425 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:31.370406 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352405    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:31.370406 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352409    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:31.370406 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352413    4425 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:31.370406 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352418    4425 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:31.370406 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352422    4425 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:31.370406 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352427    4425 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:31.370406 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352432    4425 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:31.370406 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.352469    4425 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:41:31.370406 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352520    4425 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:41:31.370406 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352527    4425 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352532    4425 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352537    4425 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352543    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352548    4425 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352553    4425 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352558    4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352562    4425 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352566    4425 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352570    4425 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352576    4425 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352581    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352585    4425 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352589    4425 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352593    4425 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352597    4425 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352601    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352605    4425 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352609    4425 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352615    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:41:31.370792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352620    4425 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352624    4425 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352628    4425 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352632    4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352636    4425 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352641    4425 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352645    4425 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352652    4425 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352657    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352663    4425 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352667    4425 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352671    4425 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352675    4425 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352679    4425 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352683    4425 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352687    4425 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352692    4425 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352696    4425 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352700    4425 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:41:31.371397 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352704    4425 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352708    4425 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352712    4425 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352718    4425 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352722    4425 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352726    4425 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352731    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352735    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352739    4425 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352744    4425 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352748    4425 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352752    4425 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352757    4425 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352762    4425 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352766    4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352771    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352775    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352779    4425 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352783    4425 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:41:31.372088 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352787    4425 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352792    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352796    4425 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352800    4425 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352805    4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352809    4425 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352814    4425 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352818    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352822    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352826    4425 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352831    4425 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352835    4425 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352839    4425 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352843    4425 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352847    4425 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352851    4425 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352858    4425 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352862    4425 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352866    4425 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:41:31.372555 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352870    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352874    4425 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352878    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352883    4425 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352887    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352891    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352896    4425 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352901    4425 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352905    4425 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352909    4425 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352914    4425 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352918    4425 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352922    4425 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352929    4425 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352935    4425 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352940    4425 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352945    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352950    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352955    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352959    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:41:31.372999 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352964    4425 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:41:31.373544 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352970    4425 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:41:31.373544 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352974    4425 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:41:31.373544 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352979    4425 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:41:31.373544 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352983    4425 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:41:31.373544 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352987    4425 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:41:31.373544 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352994    4425 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:41:31.373544 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:31.352998    4425 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:41:31.373544 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.353068    4425 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 00:41:31.373544 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.354554    4425 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 00:41:31.373544 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.354627    4425 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 00:41:31.373544 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.354894    4425 server.go:1014] "Starting client certificate rotation"
Oct 30 00:41:31.373544 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.355025    4425 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:41:31.373544 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.355079    4425 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:35:51 +0000 UTC" deadline="2028-05-27 00:35:33.927259733 +0000 UTC"
Oct 30 00:41:31.373544 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.355105    4425 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13799h54m2.572156827s"
Oct 30 00:41:31.373544 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.355590    4425 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:41:31.373544 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.357240    4425 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:41:31.373926 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.358632    4425 log.go:25] "Validated CRI v1 runtime API"
Oct 30 00:41:31.373926 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.360430    4425 log.go:25] "Validated CRI v1 image API"
Oct 30 00:41:31.373926 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.361136    4425 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 00:41:31.373926 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.362493    4425 fs.go:135] Filesystem UUIDs: map[6f0a5da8-8e2e-4ee6-99e8-3e1d6cbd6a2f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b22f0fda-b965-41a8-948e-a1c224d47f38:/dev/nvme0n1p4]
Oct 30 00:41:31.374039 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.362511    4425 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01807897f26a162bbd2dbb2979a3b8bf22171bd050a356bb06d7d95550eb041a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01807897f26a162bbd2dbb2979a3b8bf22171bd050a356bb06d7d95550eb041a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/090aee5914485b8a7823a90d1cb665f9e8275a963aa2addf26545f81c8534d46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/090aee5914485b8a7823a90d1cb665f9e8275a963aa2addf26545f81c8534d46/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1889d2bf99a8f6863b3403d36d4885593394551741115b3026723649b9bf013f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1889d2bf99a8f6863b3403d36d4885593394551741115b3026723649b9bf013f/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46963cd6ffcef17e1b289b99fdc1296214f353768c00c222a500aa7ee2b6ab01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46963cd6ffcef17e1b289b99fdc1296214f353768c00c222a500aa7ee2b6ab01/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5938488a15873da121c350bd2a0573176e777b9f49964e3872708c721fad2ae9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5938488a15873da121c350bd2a0573176e777b9f49964e3872708c721fad2ae9/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c0eab2a4a7a80ad6c3562b408163aeb13743bd43038107b928e7441be33ff0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c0eab2a4a7a80ad6c3562b408163aeb13743bd43038107b928e7441be33ff0c/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8978da243611b66666b2d1025fdc3b3ee0185e684cf76dd7dcff2a933702ac8d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8978da243611b66666b2d1025fdc3b3ee0185e684cf76dd7dcff2a933702ac8d/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a30e8b1969efd766b01fadee5adb70c7cec6959b6b2fee32c3ff3fadcc6bdbb3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a30e8b1969efd766b01fadee5adb70c7cec6959b6b2fee32c3ff3fadcc6bdbb3/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf9b95b5a46c6d0a903419448e97bfadf4b3f5abc5a2d89b079dac9001e7c953/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf9b95b5a46c6d0a903419448e97bfadf4b3f5abc5a2d89b079dac9001e7c953/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1ae6d47423fbd414d8969ab8a547ced0df026f778417443a085a723f4b21823/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1ae6d47423fbd414d8969ab8a547ced0df026f778417443a085a723f4b21823/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db088824e75d4d815e976a2beaf1ff299248af53e5d9b89145f60e2232895bba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db088824e75d4d815e976a2beaf1ff299248af53e5d9b89145f60e2232895bba/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e11a3dc899194af4a0b1a526c973c17d086a9194b5c6d1eb0f132739c21725a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e11a3dc899194af4a0b1a526c973c17d086a9194b5c6d1eb0f132739c21725a4/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed625f5cad539fbb5aaa93de13ca93d89d6c585b85dbd26f2a472f681f877255/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed625f5cad539fbb5aaa93de13ca93d89d6c585b85dbd26f2a472f681f877255/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/efa04a170026a47686f8371cdc8639e4be1dcfc33c16f8af4f09e48e3680b616/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/efa04a170026a47686f8371cdc8639e4be1dcfc33c16f8af4f09e48e3680b616/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03bdb0b4-fc37-4886-93eb-9dc6ce910e63/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/03bdb0b4-fc37-4886-93eb-9dc6ce910e63/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03bdb0b4-fc37-4886-93eb-9dc6ce910e63/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/03bdb0b4-fc37-4886-93eb-9dc6ce910e63/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03bdb0b4-fc37-4886-93eb-9dc6ce910e63/volumes/kubernetes.io~projected/kube-api-access-75h4s:{mountpoint:/var/lib/kubelet/pods/03bdb0b4-fc37-4886-93eb-9dc6ce910e63/volumes/kubernetes.io~projected/kube-api-access-75h4s major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/098b604a-a614-4584-abc6-bfbe37ae40e9/volumes/kubernetes.io~projected/kube-api-access-wn7ts:{mountpoint:/var/lib/kubelet/pods/098b604a-a614-4584-abc6-bfbe37ae40e9/volumes/kubernetes.io~projected/kube-api-access-wn7ts major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/372da31c-aac8-485a-bcc4-31a830f176a9/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/372da31c-aac8-485a-bcc4-31a830f176a9/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3822276d-acd5-4f16-ac86-32e8978c1f9f/volumes/kubernetes.io~projected/kube-api-access-sz7mb:{mountpoint:/var/lib/kubelet/pods/3822276d-acd5-4f16-ac86-32e8978c1f9f/volumes/kubernetes.io~projected/kube-api-access-sz7mb major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57ac048f-898e-4478-87f5-251679ad7bcf/volumes/kubernetes.io~projected/kube-api-access-l7qg2:{mountpoint:/var/lib/kubelet/pods/57ac048f-898e-4478-87f5-251679ad7bcf/volumes/kubernetes.io~projected/kube-api-access-l7qg2 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d/volumes/kubernetes.io~projected/kube-api-access-qhbvm:{mountpoint:/var/lib/kubelet/pods/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d/volumes/kubernetes.io~projected/kube-api-access-qhbvm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79677abd-0ee7-4081-b31e-9f07e913fd2d/volumes/kubernetes.io~projected/kube-api-access-zn77j:{mountpoint:/var/lib/kubelet/pods/79677abd-0ee7-4081-b31e-9f07e913fd2d/volumes/kubernetes.io~projected/kube-api-access-zn77j major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93171d29-b786-4586-80aa-9643856f242f/volumes/kubernetes.io~projected/kube-api-access-9jrrr:{mountpoint:/var/lib/kubelet/pods/93171d29-b786-4586-80aa-9643856f242f/volumes/kubernetes.io~projected/kube-api-access-9jrrr major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae89a416-9d4c-4477-bf72-a5371f55701a/volumes/kubernetes.io~projected/kube-api-access-l57jg:{mountpoint:/var/lib/kubelet/pods/ae89a416-9d4c-4477-bf72-a5371f55701a/volumes/kubernetes.io~projected/kube-api-access-l57jg major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb1e136c-a300-45e4-9031-ea39904518a5/volumes/kubernetes.io~projected/kube-api-access-bqngb:{mountpoint:/var/lib/kubelet/pods/cb1e136c-a300-45e4-9031-ea39904518a5/volumes/kubernetes.io~projected/kube-api-access-bqngb major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1bada76-8261-4ed7-adaf-8de93cf733c9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d1bada76-8261-4ed7-adaf-8de93cf733c9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1bada76-8261-4ed7-adaf-8de93cf733c9/volumes/kubernetes.io~projected/kube-api-access-6t7j2:{mountpoint:/var/lib/kubelet/pods/d1bada76-8261-4ed7-adaf-8de93cf733c9/volumes/kubernetes.io~projected/kube-api-access-6t7j2 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1bada76-8261-4ed7-adaf-8de93cf733c9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d1bada76-8261-4ed7-adaf-8de93cf733c9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0ff8a2c-7788-4791-8212-84a9a683320c/volumes/kubernetes.io~projected/kube-api-access-5v7fm:{mountpoint:/var/lib/kubelet/pods/e0ff8a2c-7788-4791-8212-84a9a683320c/volumes/kubernetes.io~projected/kube-api-access-5v7fm major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4be6fff-4a26-4188-8ca7-80125df121d0/volumes/kubernetes.io~projected/kube-api-access-fswtt:{mountpoint:/var/lib/kubelet/pods/e4be6fff-4a26-4188-8ca7-80125df121d0/volumes/kubernetes.io~projected/kube-api-access-fswtt major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4be6fff-4a26-4188-8ca7-80125df121d0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e4be6fff-4a26-4188-8ca7-80125df121d0/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f65424f1-dbb4-4e42-ab0a-a3334cef4dba/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/f65424f1-dbb4-4e42-ab0a-a3334cef4dba/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6870c47-e336-47b5-adfa-7143eed6d6ab/volumes/kubernetes.io~projected/kube-api-access-8pq68:{mountpoint:/var/lib/kubelet/pods/f6870c47-e336-47b5-adfa-7143eed6d6ab/volumes/kubernetes.io~projected/kube-api-access-8pq68 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414/volumes/kubernetes.io~projected/kube-api-access-76pzc:{mountpoint:/var/lib/kubelet/pods/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414/volumes/kubernetes.io~projected/kube-api-access-76pzc major:0 minor:61 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/05f917b60eacabbbdb34ee285b2275c7e9b714509f76d09bf14236c29658fa72/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9478964cd2a7568c4fd864010617deff4d6b7602d950b90f0352b6d79b01012c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5956bc8f047c057eb3a817db11bac9ff41a5f39b82eac17099f8673ad76d629e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/0f191008c6f9d1bd01a80260bdd5ba37e85d65562ac6a520336684245d6cdee4/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6ad62667e44945a7863c55a02e20183024211f9cd4c99ffecdb6f3075cb728d3/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/fac318c6ab78aa86060e0ab0a062027ae33ec76d4c24b3a5cb69164185272835/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/06e31f77b3d58cebb7836fdde3b6cac38075edd98873fd528c4c37f84cece4cd/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2dcda5eab5734dfcd4f99858dc9bae46f1872631170575ff0b637630445d4729/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/490c0749ef24ec639c187e269c261cf368a87a3185efa8f1ed9fb0bc9dca9e50/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/8d09af2a35558472637d54b2e60f57f8cc6705ac743ecaf22fbba376ecf5011d/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/cd71d337782d557807cbddeb86890d518fd4619dcf816adbc1a8a4a09b9d4a1e/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/bc502e68a6eeaae06eb20326014cf33f2bad090ef599938f56e5449b253e25e0/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/39d05d66dcd90d9e8bf0b1eef68a74cb633f96ab747b47c8b0cac62105cc09e4/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/ec74f5f8182317d9f5629521561757cb1cd0c84b224964a2f93a515129f367df/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7d2275a218c40c45b70bea1ca87371916d60f0150b76dc75d90045c5afe888f0/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/5354f3092f73722030ba7b622b0fe082d6020f80b57b8b275d3095b50ec34734/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/0037b9f7fabc7cc3123aa0d02c6eef9fa32585b4b6b87ee325d960a9f764d222/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/da0ee2bb87fc777da51634b0ca93ea9c97b0a5ed8d7851d6dd62f60f1d0d70a2/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-183:{mountpoint:/var/lib/containers/storage/overlay/c9eef15e010551f1db4be0741215bbf0a9c10f2d348afb4b061117fe71687650/merged major:0 minor:183 fsType:overlay blockSize:0} overlay_0-185:{mountpoint:/var/lib/containers/storage/overlay/57b0c2907fff15350aa2a480bd34b071b50105a5ccaa236463df0e3426960b78/merged major:0 minor:185 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2f80d3f2472ec35b5a2cfff5cdbededc60df9588a263f9263264e46399773b12/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/cbdd421988b4d3c22c3674e57ac192dc2a522fa4aea62684b7af3f6c20492317/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/c90741b3718a0a0db4e358d33bffe0591b1de71d782f6f0debf31ec35d6589d7/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/1142be97d3f3c55573b6d032eef3229d2dff5437d34662360fd997f008eb1804/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/9741cc187f0219ab2402ded9338c5b47eecb89ef379e967654f9c53eb5a6b2c7/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/b2eca034822c140ef705273a57842c260b6ba905990810e5403464cb98bd1898/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/5ff009681ca895357d7a9d7e9af078717bb9f29601362984edbede5218372209/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/75b21504c02c0008eb1f174e4442f1a7bed994437f404e01db50891aeb801f4d/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/37554bb97a5140c420c2253c5fe2ea8fec1a26afe455d5d98d8fe6378c5f766b/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/11fc3a76286a181a8b0295de09cf770ea50b18f5e7b7a7c2a1e0a1a47b047efb/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/10e1726deceeece55b65b05536eb078afecf9e81fe8e3c200a11714ea6a7a1c0/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/61e3f128dc00a64846af5f6ef46e999f4e960803df758a156d66a9aa8bf07f37/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/679526f3decdcc4f81cfe8d299e9cfbe30bea3cc36d7fe8bc665532f36e29fb2/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/bf158d57014946746cf165728e42e30fee02e46d02f26c288f0dbc8f4ddc4ad0/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/35bb1d76e0621dbbee6667dd7b31b9daf4b6919b6b1fcc71db6dcabccd356091/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/da08fe00508913c6214ee558c5c907815c7fafd2476a5eb4bd8b3e0557f5cb30/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/9deccbbff3a025db6a5118aa0fe76c4b8355ab3e4a50608e8e49e84d39c06142/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/2f7764d1d18836e3394049c0b1bef3566af25ef08d2566cddaea0d4f2fcc0180/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/74e1736a268aeced0f41b360677787cd5953e2032e585cefd4006b32a576a697/merged major:0 minor:97 fsType:overlay blockSize:0}]
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.369372    4425 manager.go:217] Machine: {Timestamp:2026-10-30 00:41:31.368472808 +0000 UTC m=+0.118128107 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3201044 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26eea458e551a80bead520c7e6f516 SystemUUID:ec26eea4-58e5-51a8-0bea-d520c7e6f516 BootID:ff1937be-1857-4d00-bf2d-3fcf44b7c139 Filesystems:[{Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01807897f26a162bbd2dbb2979a3b8bf22171bd050a356bb06d7d95550eb041a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/372da31c-aac8-485a-bcc4-31a830f176a9/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4be6fff-4a26-4188-8ca7-80125df121d0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414/volumes/kubernetes.io~projected/kube-api-access-76pzc DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c0eab2a4a7a80ad6c3562b408163aeb13743bd43038107b928e7441be33ff0c/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8978da243611b66666b2d1025fdc3b3ee0185e684cf76dd7dcff2a933702ac8d/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1889d2bf99a8f6863b3403d36d4885593394551741115b3026723649b9bf013f/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed625f5cad539fbb5aaa93de13ca93d89d6c585b85dbd26f2a472f681f877255/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5938488a15873da121c350bd2a0573176e777b9f49964e3872708c721fad2ae9/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db088824e75d4d815e976a2beaf1ff299248af53e5d9b89145f60e2232895bba/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d1bada76-8261-4ed7-adaf-8de93cf733c9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/03bdb0b4-fc37-4886-93eb-9dc6ce910e63/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46963cd6ffcef17e1b289b99fdc1296214f353768c00c222a500aa7ee2b6ab01/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1bada76-8261-4ed7-adaf-8de93cf733c9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-183 DeviceMajor:0 DeviceMinor:183 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3822276d-acd5-4f16-ac86-32e8978c1f9f/volumes/kubernetes.io~projected/kube-api-access-sz7mb DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-185 DeviceMajor:0 DeviceMinor:185 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03bdb0b4-fc37-4886-93eb-9dc6ce910e63/volumes/kubernetes.io~projected/kube-api-access-75h4s DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d/volumes/kubernetes.io~projected/kube-api-access-qhbvm DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae89a416-9d4c-4477-bf72-a5371f55701a/volumes/kubernetes.io~projected/kube-api-access-l57jg DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/efa04a170026a47686f8371cdc8639e4be1dcfc33c16f8af4f09e48e3680b616/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e11a3dc899194af4a0b1a526c973c17d086a9194b5c6d1eb0f132739c21725a4/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/79677abd-0ee7-4081-b31e-9f07e913fd2d/volumes/kubernetes.io~projected/kube-api-access-zn77j DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb1e136c-a300-45e4-9031-ea39904518a5/volumes/kubernetes.io~projected/kube-api-access-bqngb DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/090aee5914485b8a7823a90d1cb665f9e8275a963aa2addf26545f81c8534d46/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/f65424f1-dbb4-4e42-ab0a-a3334cef4dba/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d1bada76-8261-4ed7-adaf-8de93cf733c9/volumes/kubernetes.io~projected/kube-api-access-6t7j2 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03bdb0b4-fc37-4886-93eb-9dc6ce910e63/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e4be6fff-4a26-4188-8ca7-80125df121d0/volumes/kubernetes.io~projected/kube-api-access-fswtt DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/93171d29-b786-4586-80aa-9643856f242f/volumes/kubernetes.io~projected/kube-api-access-9jrrr DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a30e8b1969efd766b01fadee5adb70c7cec6959b6b2fee32c3ff3fadcc6bdbb3/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0ff8a2c-7788-4791-8212-84a9a683320c/volumes/kubernetes.io~projected/kube-api-access-5v7fm DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf9b95b5a46c6d0a903419448e97bfadf4b3f5abc5a2d89b079dac9001e7c953/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57ac048f-898e-4478-87f5-251679ad7bcf/volumes/kubernetes.io~projected/kube-api-access-l7qg2 DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/f6870c47-e336-47b5-adfa-7143eed6d6ab/volumes/kubernetes.io~projected/kube-api-access-8pq68 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/098b604a-a614-4584-abc6-bfbe37ae40e9/volumes/kubernetes.io~projected/kube-api-access-wn7ts DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1ae6d47423fbd414d8969ab8a547ced0df026f778417443a085a723f4b21823/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:090aee5914485b8 MacAddress:aa:74:12:6d:c1:d3 Speed:10000 Mtu:8901} {Name:1889d2bf99a8f68 MacAddress:e2:bf:87:34:34:c2 Speed:10000 Mtu:8901} {Name:5938488a15873da MacAddress:4e:de:c5:a9:2c:3c Speed:10000 Mtu:8901} {Name:8978da243611b66 MacAddress:da:0c:40:66:42:35 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e2:0f:ce:bd Speed:0 Mtu:9001} {Name:br-int MacAddress:5e:77:35:d8:53:4a Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:e2:0f:ce:bd Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:f2:16:d0:ee:9e:d2 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:3a:4e:e7:45:18:80 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.370243    4425 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.370298    4425 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.370563    4425 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.370583    4425 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-32.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.370700    4425 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.370706    4425 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.370748    4425 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.370768    4425 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.370847    4425 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.371003    4425 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.371103    4425 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.371136    4425 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.371147    4425 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.371171    4425 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.371179    4425 kubelet.go:405] "Adding apiserver pod source"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.371186    4425 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.371669    4425 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.371678    4425 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.372071    4425 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.373909    4425 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.373919    4425 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.374001    4425 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 00:41:31.374183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.374171    4425 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 00:41:31.374966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.374198    4425 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 00:41:31.374966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.374280    4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 00:41:31.374966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.374291    4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 00:41:31.374966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.374296    4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 00:41:31.374966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.374301    4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 00:41:31.374966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.374306    4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 00:41:31.374966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.374311    4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 00:41:31.374966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.374317    4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 00:41:31.374966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.374322    4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 00:41:31.374966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.374328    4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 00:41:31.374966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.374334    4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 00:41:31.374966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.374348    4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 00:41:31.374966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.374358    4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 00:41:31.374966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.374380    4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 00:41:31.374966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.374386    4425 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 00:41:31.377399 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.377386    4425 server.go:1263] "Started kubelet"
Oct 30 00:41:31.377806 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.377451    4425 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 00:41:31.377806 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.377483    4425 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 00:41:31.377806 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.377554    4425 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 00:41:31.377864 ip-10-0-133-32 systemd[1]: Started Kubernetes Kubelet.
Oct 30 00:41:31.378265 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.378213    4425 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 00:41:31.378549 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.378533    4425 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 00:41:31.380596 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.380046    4425 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 00:41:31.380596 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.380067    4425 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 00:41:31.380596 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.380104    4425 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:35:53 +0000 UTC" deadline="2028-07-22 17:56:27.811057222 +0000 UTC"
Oct 30 00:41:31.380596 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.380133    4425 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15161h14m56.430927368s"
Oct 30 00:41:31.380596 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.380190    4425 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 00:41:31.380596 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.380199    4425 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 00:41:31.380596 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.380274    4425 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 00:41:31.380596 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:41:31.380256    4425 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-32.ec2.internal\" not found"
Oct 30 00:41:31.382141 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.382043    4425 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 00:41:31.382141 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.382123    4425 factory.go:55] Registering systemd factory
Oct 30 00:41:31.382141 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.382134    4425 factory.go:223] Registration of the systemd container factory successfully
Oct 30 00:41:31.383114 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.383098    4425 factory.go:153] Registering CRI-O factory
Oct 30 00:41:31.383237 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.383132    4425 factory.go:223] Registration of the crio container factory successfully
Oct 30 00:41:31.383237 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.383150    4425 factory.go:103] Registering Raw factory
Oct 30 00:41:31.383237 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.383184    4425 manager.go:1199] Started watching for new ooms in manager
Oct 30 00:41:31.383723 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.383710    4425 manager.go:319] Starting recovery of all containers
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389313    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0ff8a2c-7788-4791-8212-84a9a683320c" volumeName="kubernetes.io/empty-dir/e0ff8a2c-7788-4791-8212-84a9a683320c-tmp-dir" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389369    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae89a416-9d4c-4477-bf72-a5371f55701a" volumeName="kubernetes.io/configmap/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-daemon-config" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389388    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1bada76-8261-4ed7-adaf-8de93cf733c9" volumeName="kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovnkube-config" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389400    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f65424f1-dbb4-4e42-ab0a-a3334cef4dba" volumeName="kubernetes.io/secret/f65424f1-dbb4-4e42-ab0a-a3334cef4dba-agent-certs" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389415    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6870c47-e336-47b5-adfa-7143eed6d6ab" volumeName="kubernetes.io/configmap/f6870c47-e336-47b5-adfa-7143eed6d6ab-serviceca" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389426    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="098b604a-a614-4584-abc6-bfbe37ae40e9" volumeName="kubernetes.io/projected/098b604a-a614-4584-abc6-bfbe37ae40e9-kube-api-access-wn7ts" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389453    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79677abd-0ee7-4081-b31e-9f07e913fd2d" volumeName="kubernetes.io/configmap/79677abd-0ee7-4081-b31e-9f07e913fd2d-iptables-alerter-script" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389466    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93171d29-b786-4586-80aa-9643856f242f" volumeName="kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389494    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1bada76-8261-4ed7-adaf-8de93cf733c9" volumeName="kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389507    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1bada76-8261-4ed7-adaf-8de93cf733c9" volumeName="kubernetes.io/secret/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389519    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="098b604a-a614-4584-abc6-bfbe37ae40e9" volumeName="kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-cni-binary-copy" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389531    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="098b604a-a614-4584-abc6-bfbe37ae40e9" volumeName="kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389543    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0ff8a2c-7788-4791-8212-84a9a683320c" volumeName="kubernetes.io/configmap/e0ff8a2c-7788-4791-8212-84a9a683320c-config-volume" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389555    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0ff8a2c-7788-4791-8212-84a9a683320c" volumeName="kubernetes.io/projected/e0ff8a2c-7788-4791-8212-84a9a683320c-kube-api-access-5v7fm" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389571    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f65424f1-dbb4-4e42-ab0a-a3334cef4dba" volumeName="kubernetes.io/configmap/f65424f1-dbb4-4e42-ab0a-a3334cef4dba-konnectivity-ca" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389614    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b80e7e-75db-4e4e-9c1b-0d2f4b754414" volumeName="kubernetes.io/empty-dir/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-tmp-dir" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389628    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1bada76-8261-4ed7-adaf-8de93cf733c9" volumeName="kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-env-overrides" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389640    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1bada76-8261-4ed7-adaf-8de93cf733c9" volumeName="kubernetes.io/projected/d1bada76-8261-4ed7-adaf-8de93cf733c9-kube-api-access-6t7j2" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389661    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03bdb0b4-fc37-4886-93eb-9dc6ce910e63" volumeName="kubernetes.io/empty-dir/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-tuned" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389672    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03bdb0b4-fc37-4886-93eb-9dc6ce910e63" volumeName="kubernetes.io/empty-dir/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-tmp" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389684    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="098b604a-a614-4584-abc6-bfbe37ae40e9" volumeName="kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389696    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3822276d-acd5-4f16-ac86-32e8978c1f9f" volumeName="kubernetes.io/projected/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-api-access-sz7mb" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389707    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57ac048f-898e-4478-87f5-251679ad7bcf" volumeName="kubernetes.io/projected/57ac048f-898e-4478-87f5-251679ad7bcf-kube-api-access-l7qg2" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389721    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb1e136c-a300-45e4-9031-ea39904518a5" volumeName="kubernetes.io/projected/cb1e136c-a300-45e4-9031-ea39904518a5-kube-api-access-bqngb" seLinuxMountContext=""
Oct 30 00:41:31.389716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389732    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3822276d-acd5-4f16-ac86-32e8978c1f9f" volumeName="kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 00:41:31.390413 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389742    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79677abd-0ee7-4081-b31e-9f07e913fd2d" volumeName="kubernetes.io/projected/79677abd-0ee7-4081-b31e-9f07e913fd2d-kube-api-access-zn77j" seLinuxMountContext=""
Oct 30 00:41:31.390413 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389756    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4be6fff-4a26-4188-8ca7-80125df121d0" volumeName="kubernetes.io/secret/e4be6fff-4a26-4188-8ca7-80125df121d0-cert" seLinuxMountContext=""
Oct 30 00:41:31.390413 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389765    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03bdb0b4-fc37-4886-93eb-9dc6ce910e63" volumeName="kubernetes.io/projected/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-kube-api-access-75h4s" seLinuxMountContext=""
Oct 30 00:41:31.390413 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389791    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="372da31c-aac8-485a-bcc4-31a830f176a9" volumeName="kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret" seLinuxMountContext=""
Oct 30 00:41:31.390413 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389800    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a4ab2cc-af6d-460b-b8dd-9f16078cc11d" volumeName="kubernetes.io/projected/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-kube-api-access-qhbvm" seLinuxMountContext=""
Oct 30 00:41:31.390413 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389811    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae89a416-9d4c-4477-bf72-a5371f55701a" volumeName="kubernetes.io/configmap/ae89a416-9d4c-4477-bf72-a5371f55701a-cni-binary-copy" seLinuxMountContext=""
Oct 30 00:41:31.390413 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389822    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae89a416-9d4c-4477-bf72-a5371f55701a" volumeName="kubernetes.io/projected/ae89a416-9d4c-4477-bf72-a5371f55701a-kube-api-access-l57jg" seLinuxMountContext=""
Oct 30 00:41:31.390413 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389836    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4be6fff-4a26-4188-8ca7-80125df121d0" volumeName="kubernetes.io/projected/e4be6fff-4a26-4188-8ca7-80125df121d0-kube-api-access-fswtt" seLinuxMountContext=""
Oct 30 00:41:31.390413 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389847    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b80e7e-75db-4e4e-9c1b-0d2f4b754414" volumeName="kubernetes.io/projected/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-kube-api-access-76pzc" seLinuxMountContext=""
Oct 30 00:41:31.390413 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389860    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3822276d-acd5-4f16-ac86-32e8978c1f9f" volumeName="kubernetes.io/empty-dir/3822276d-acd5-4f16-ac86-32e8978c1f9f-data-volume" seLinuxMountContext=""
Oct 30 00:41:31.390413 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389871    4425 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6870c47-e336-47b5-adfa-7143eed6d6ab" volumeName="kubernetes.io/projected/f6870c47-e336-47b5-adfa-7143eed6d6ab-kube-api-access-8pq68" seLinuxMountContext=""
Oct 30 00:41:31.390413 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389884    4425 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 00:41:31.390413 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.389892    4425 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 00:41:31.411583 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.411542    4425 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 00:41:31.414264 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.414243    4425 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 00:41:31.414264 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.414261    4425 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 00:41:31.414419 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.414276    4425 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 00:41:31.414419 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:41:31.414305    4425 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 00:41:31.425064 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.425044    4425 generic.go:356] "Generic (PLEG): container finished" podID="f710fb4e919766c9ee824cfee175df00" containerID="62b18312a6fb1bd8c8ecee96b509655ed85a75932dd3c2c8504f44764d54b43d" exitCode=0
Oct 30 00:41:31.426512 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.426484    4425 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:41:31.430562 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.430543    4425 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:41:31.430746 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.430729    4425 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 00:41:31.432463 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.432442    4425 generic.go:356] "Generic (PLEG): container finished" podID="098b604a-a614-4584-abc6-bfbe37ae40e9" containerID="65f580e412d0e6a61636c78011109d1af59d20c4f618b358be56a0d5b867a9a9" exitCode=0
Oct 30 00:41:31.432463 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.432463    4425 generic.go:356] "Generic (PLEG): container finished" podID="098b604a-a614-4584-abc6-bfbe37ae40e9" containerID="eb6f6ac2ce1d1900dcab4e3ab000bd89df65e73191aa6e0dd6843490b1ac156b" exitCode=0
Oct 30 00:41:31.432598 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.432472    4425 generic.go:356] "Generic (PLEG): container finished" podID="098b604a-a614-4584-abc6-bfbe37ae40e9" containerID="f0e934aeafe61a566b38ed36a03f25dcade0a5f1abd540c06ce5718277ec9e20" exitCode=0
Oct 30 00:41:31.432598 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.432480    4425 generic.go:356] "Generic (PLEG): container finished" podID="098b604a-a614-4584-abc6-bfbe37ae40e9" containerID="36ea43fc43cea2085f9b8f2c66a56e4ed1949a9b5c1effd86cfbc1025e3aaee4" exitCode=0
Oct 30 00:41:31.432598 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.432487    4425 generic.go:356] "Generic (PLEG): container finished" podID="098b604a-a614-4584-abc6-bfbe37ae40e9" containerID="7d7fb7a2251eb53a1ffa0f0199d8f1790df1832e7d2b01d1e7cebf4af29435f1" exitCode=0
Oct 30 00:41:31.432598 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.432494    4425 generic.go:356] "Generic (PLEG): container finished" podID="098b604a-a614-4584-abc6-bfbe37ae40e9" containerID="0c775ac4c57b9c4d2bade838b2d0d064541ea1be33ae905a116e603268182bea" exitCode=0
Oct 30 00:41:31.440491 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.440471    4425 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:41:31.447286 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.447264    4425 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:41:31.448439 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.448424    4425 manager.go:324] Recovery completed
Oct 30 00:41:31.462340 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.462323    4425 cpu_manager.go:226] "Starting" policy="none"
Oct 30 00:41:31.462340 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.462335    4425 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 00:41:31.462481 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.462350    4425 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 00:41:31.462481 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.462454    4425 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 00:41:31.462481 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.462464    4425 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 00:41:31.462565 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.462484    4425 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 00:41:31.462565 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.462492    4425 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 00:41:31.462565 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.462498    4425 policy_none.go:50] "Start"
Oct 30 00:41:31.462565 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.462506    4425 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 00:41:31.462565 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.462515    4425 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 00:41:31.462698 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.462633    4425 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 00:41:31.462698 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.462640    4425 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 00:41:31.462698 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.462644    4425 policy_none.go:44] "Start"
Oct 30 00:41:31.466341 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.466314    4425 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 00:41:31.468779 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.468766    4425 manager.go:341] "Starting Device Plugin manager"
Oct 30 00:41:31.469266 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:41:31.468793    4425 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 00:41:31.469347 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.469270    4425 server.go:87] "Starting device plugin registration server"
Oct 30 00:41:31.469560 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.469547    4425 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 00:41:31.469631 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.469560    4425 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 00:41:31.469685 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.469667    4425 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 00:41:31.469802 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.469768    4425 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 00:41:31.469870 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.469812    4425 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 00:41:31.469870 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.469819    4425 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 00:41:31.469954 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.469883    4425 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:41:31.469801168Z","UUID":"559abcf8-1d0f-4deb-9cdf-c113c86791cb","Handler":null,"Name":"","Endpoint":""}
Oct 30 00:41:31.471135 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:41:31.471118    4425 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 00:41:31.471317 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.471300    4425 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 00:41:31.471381 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.471335    4425 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 00:41:31.514382 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.514349    4425 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"]
Oct 30 00:41:31.514809 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.514778    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal" event={"ID":"f710fb4e919766c9ee824cfee175df00","Type":"ContainerStarted","Data":"111545754b6b0585f925d00fce2b5397bf4d496e3e9b5f5d4210c6e1af2d5498"}
Oct 30 00:41:31.514855 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.514822    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal" event={"ID":"f710fb4e919766c9ee824cfee175df00","Type":"ContainerDied","Data":"62b18312a6fb1bd8c8ecee96b509655ed85a75932dd3c2c8504f44764d54b43d"}
Oct 30 00:41:31.514855 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.514845    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal" event={"ID":"f710fb4e919766c9ee824cfee175df00","Type":"ContainerStarted","Data":"db088824e75d4d815e976a2beaf1ff299248af53e5d9b89145f60e2232895bba"}
Oct 30 00:41:31.514939 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.514882    4425 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:41:31.514939 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.514912    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal" event={"ID":"f92445811ce6c3905e7d9666e6d2a0cd","Type":"ContainerStarted","Data":"9755023c1e94d2ffc36a2ed9a909e373b40628d29cc81304743b69df93c1d1f9"}
Oct 30 00:41:31.514939 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.514930    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal" event={"ID":"f92445811ce6c3905e7d9666e6d2a0cd","Type":"ContainerStarted","Data":"bf9b95b5a46c6d0a903419448e97bfadf4b3f5abc5a2d89b079dac9001e7c953"}
Oct 30 00:41:31.515024 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.514985    4425 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Oct 30 00:41:31.535753 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.535737    4425 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:41:31.535810 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:41:31.535773    4425 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-32.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Oct 30 00:41:31.536402 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.536391    4425 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:41:31.536441 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:41:31.536413    4425 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:41:31.570525 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.570510    4425 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:41:31.571480 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.571466    4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:41:31.571539 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.571489    4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:41:31.571539 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.571498    4425 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-32.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:41:31.571539 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.571521    4425 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-32.ec2.internal"
Oct 30 00:41:31.604027 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.603991    4425 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-32.ec2.internal"
Oct 30 00:41:31.604094 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.604031    4425 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-32.ec2.internal"
Oct 30 00:41:31.631474 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.631456    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f710fb4e919766c9ee824cfee175df00-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"f710fb4e919766c9ee824cfee175df00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:41:31.631547 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.631480    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f710fb4e919766c9ee824cfee175df00-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"f710fb4e919766c9ee824cfee175df00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:41:31.631547 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.631496    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f92445811ce6c3905e7d9666e6d2a0cd-config\") pod \"kube-apiserver-proxy-ip-10-0-133-32.ec2.internal\" (UID: \"f92445811ce6c3905e7d9666e6d2a0cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Oct 30 00:41:31.732382 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.732365    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f710fb4e919766c9ee824cfee175df00-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"f710fb4e919766c9ee824cfee175df00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:41:31.732477 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.732389    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f710fb4e919766c9ee824cfee175df00-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"f710fb4e919766c9ee824cfee175df00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:41:31.732477 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.732404    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f92445811ce6c3905e7d9666e6d2a0cd-config\") pod \"kube-apiserver-proxy-ip-10-0-133-32.ec2.internal\" (UID: \"f92445811ce6c3905e7d9666e6d2a0cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Oct 30 00:41:31.732477 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.732448    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f92445811ce6c3905e7d9666e6d2a0cd-config\") pod \"kube-apiserver-proxy-ip-10-0-133-32.ec2.internal\" (UID: \"f92445811ce6c3905e7d9666e6d2a0cd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Oct 30 00:41:31.732563 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.732478    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f710fb4e919766c9ee824cfee175df00-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"f710fb4e919766c9ee824cfee175df00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:41:31.732563 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:31.732479    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f710fb4e919766c9ee824cfee175df00-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" (UID: \"f710fb4e919766c9ee824cfee175df00\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:41:32.372362 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.372334    4425 apiserver.go:52] "Watching apiserver"
Oct 30 00:41:32.408534 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.408507    4425 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 00:41:32.408909 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.408891    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-w22p8","openshift-ingress-canary/ingress-canary-5b69x","openshift-network-diagnostics/network-check-target-qs6rx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j","openshift-multus/multus-additional-cni-plugins-q7mpt","openshift-dns/node-resolver-8k6rj","kube-system/konnectivity-agent-5lmgn","openshift-image-registry/node-ca-7zgwc","kube-system/global-pull-secret-syncer-fw99d","openshift-multus/network-metrics-daemon-nqtxx","openshift-dns/dns-default-66wbz","kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-xkblh","openshift-cluster-node-tuning-operator/tuned-nn559","openshift-ovn-kubernetes/ovnkube-node-857b8","openshift-insights/insights-runtime-extractor-jz4ff","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal","openshift-multus/multus-xjww4"]
Oct 30 00:41:32.411074 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.411051    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:32.414430 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.414406    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:32.416073 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.416048    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:32.423711 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.423691    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 00:41:32.423802 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.423751    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 00:41:32.423802 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.423756    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 00:41:32.424046 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.424028    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 00:41:32.424142 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.424051    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 00:41:32.424142 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.424085    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 00:41:32.424142 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.424089    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 00:41:32.424142 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.424098    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-szv55\""
Oct 30 00:41:32.424142 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.424089    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 00:41:32.424142 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.424124    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cg6zp\""
Oct 30 00:41:32.424418 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.424275    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 00:41:32.424418 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.424277    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7htjs\""
Oct 30 00:41:32.424418 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.424325    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 00:41:32.424539 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.424522    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:32.429805 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.429790    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 00:41:32.429805 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.429793    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:32.429921 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.429815    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 00:41:32.430267 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.430254    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 00:41:32.430412 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.430397    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 00:41:32.432569 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.432555    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7sn5q\""
Oct 30 00:41:32.432705 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.432693    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z4qxk\""
Oct 30 00:41:32.432770 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.432730    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-92qkq\""
Oct 30 00:41:32.433772 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.433756    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wwjm6\""
Oct 30 00:41:32.433841 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.433827    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5b9kk\""
Oct 30 00:41:32.433841 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.433834    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 00:41:32.433939 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.433915    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 00:41:32.433976 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.433949    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 00:41:32.434043 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434031    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 00:41:32.434123 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434109    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 00:41:32.434260 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434246    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 00:41:32.434318 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434272    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kdklf\""
Oct 30 00:41:32.434318 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434312    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 00:41:32.434392 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434319    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jgsn4\""
Oct 30 00:41:32.434392 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434382    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l8pvs\""
Oct 30 00:41:32.434480 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434426    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f5gxs\""
Oct 30 00:41:32.434480 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434470    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qs9gb\""
Oct 30 00:41:32.434564 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434525    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 00:41:32.434564 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434538    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wm8np\""
Oct 30 00:41:32.434564 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434541    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 00:41:32.434564 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434554    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 00:41:32.434713 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434626    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 00:41:32.434713 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434648    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 00:41:32.434803 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434722    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 00:41:32.434803 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434783    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 00:41:32.434803 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434800    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 00:41:32.434927 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434801    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 00:41:32.434927 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434813    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 00:41:32.434927 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434857    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 00:41:32.434927 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434869    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 00:41:32.434927 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434888    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 00:41:32.434927 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434905    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 00:41:32.434927 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434912    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 00:41:32.434927 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434924    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 00:41:32.435168 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434935    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 00:41:32.435168 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.434954    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 00:41:32.435168 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.435011    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 00:41:32.435168 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.435067    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x7m5x\""
Oct 30 00:41:32.435168 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.435107    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z5p7p\""
Oct 30 00:41:32.447482 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.447466    4425 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:41:32.447596 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.447584    4425 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Oct 30 00:41:32.468390 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.468373    4425 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:41:32.468456 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:41:32.468410    4425 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-32.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-32.ec2.internal"
Oct 30 00:41:32.468977 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.468964    4425 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:41:32.469015 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:41:32.468995    4425 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-32.ec2.internal"
Oct 30 00:41:32.481974 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.481958    4425 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 00:41:32.505026 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.504989    4425 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lmgn_f65424f1-dbb4-4e42-ab0a-a3334cef4dba/konnectivity-agent/0.log" limit=1024
Oct 30 00:41:32.536103 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536079    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-cni-multus\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.536204 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536106    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-kubelet\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.536204 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536121    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-etc-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.536204 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536140    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-ovn\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.536204 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536184    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovn-node-metrics-cert\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.536366 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536207    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysctl-conf\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.536366 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536311    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-cni-bin\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.536366 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536335    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-node-log\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.536366 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536352    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-log-socket\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.536519 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536404    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-cnibin\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:41:32.536519 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536448    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:41:32.536606 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536539    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:41:32.536606 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536558    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovn-node-metrics-cert\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.536606 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536576    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f65424f1-dbb4-4e42-ab0a-a3334cef4dba-konnectivity-ca\") pod \"konnectivity-agent-5lmgn\" (UID: \"f65424f1-dbb4-4e42-ab0a-a3334cef4dba\") " pod="kube-system/konnectivity-agent-5lmgn"
Oct 30 00:41:32.536606 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536595    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3822276d-acd5-4f16-ac86-32e8978c1f9f-crio-socket\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:32.536761 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536626    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.536761 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536720    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ff8a2c-7788-4791-8212-84a9a683320c-config-volume\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:32.536761 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536742    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-daemon-config\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.536879 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536770    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f65424f1-dbb4-4e42-ab0a-a3334cef4dba-konnectivity-ca\") pod \"konnectivity-agent-5lmgn\" (UID: \"f65424f1-dbb4-4e42-ab0a-a3334cef4dba\") " pod="kube-system/konnectivity-agent-5lmgn"
Oct 30 00:41:32.536879 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536793    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovnkube-config\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.536879 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536795    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:41:32.537001 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536880    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:41:32.537001 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536901    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz7mb\" (UniqueName: \"kubernetes.io/projected/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-api-access-sz7mb\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:32.537001 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536937    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79677abd-0ee7-4081-b31e-9f07e913fd2d-iptables-alerter-script\") pod \"iptables-alerter-w22p8\" (UID: \"79677abd-0ee7-4081-b31e-9f07e913fd2d\") " pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:41:32.537001 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536951    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0ff8a2c-7788-4791-8212-84a9a683320c-config-volume\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:32.537001 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.536969    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-daemon-config\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.537435 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537025    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovnkube-config\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.537435 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537055    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-socket-dir-parent\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.537435 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537073    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-systemd-units\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.537435 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537092    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t7j2\" (UniqueName: \"kubernetes.io/projected/d1bada76-8261-4ed7-adaf-8de93cf733c9-kube-api-access-6t7j2\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.537435 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537110    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-sys\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.537435 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537189    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79677abd-0ee7-4081-b31e-9f07e913fd2d-iptables-alerter-script\") pod \"iptables-alerter-w22p8\" (UID: \"79677abd-0ee7-4081-b31e-9f07e913fd2d\") " pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:41:32.537435 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537306    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-tuned\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.537722 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537145    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-tuned\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.537722 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537572    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f65424f1-dbb4-4e42-ab0a-a3334cef4dba-agent-certs\") pod \"konnectivity-agent-5lmgn\" (UID: \"f65424f1-dbb4-4e42-ab0a-a3334cef4dba\") " pod="kube-system/konnectivity-agent-5lmgn"
Oct 30 00:41:32.537722 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537588    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ff8a2c-7788-4791-8212-84a9a683320c-tmp-dir\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:32.537722 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537605    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-hostroot\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.537722 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537618    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-multus-certs\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.537722 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537632    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-host\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.537722 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537661    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqngb\" (UniqueName: \"kubernetes.io/projected/cb1e136c-a300-45e4-9031-ea39904518a5-kube-api-access-bqngb\") pod \"migrator-66df5d496b-xkblh\" (UID: \"cb1e136c-a300-45e4-9031-ea39904518a5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkblh"
Oct 30 00:41:32.537722 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537680    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ff8a2c-7788-4791-8212-84a9a683320c-tmp-dir\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:32.537722 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537704    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/372da31c-aac8-485a-bcc4-31a830f176a9-kubelet-config\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:32.538082 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537733    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/372da31c-aac8-485a-bcc4-31a830f176a9-dbus\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:32.538082 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537782    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrrr\" (UniqueName: \"kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr\") pod \"network-check-target-qs6rx\" (UID: \"93171d29-b786-4586-80aa-9643856f242f\") " pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:32.538082 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537809    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-sys-fs\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.538082 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537832    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-cni-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.538082 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537864    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f65424f1-dbb4-4e42-ab0a-a3334cef4dba-agent-certs\") pod \"konnectivity-agent-5lmgn\" (UID: \"f65424f1-dbb4-4e42-ab0a-a3334cef4dba\") " pod="kube-system/konnectivity-agent-5lmgn"
Oct 30 00:41:32.538082 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537910    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-cnibin\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.538082 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537946    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-var-lib-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.538082 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.537986    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fswtt\" (UniqueName: \"kubernetes.io/projected/e4be6fff-4a26-4188-8ca7-80125df121d0-kube-api-access-fswtt\") pod \"ingress-canary-5b69x\" (UID: \"e4be6fff-4a26-4188-8ca7-80125df121d0\") " pod="openshift-ingress-canary/ingress-canary-5b69x"
Oct 30 00:41:32.538082 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538018    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:32.538082 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538038    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhbvm\" (UniqueName: \"kubernetes.io/projected/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-kube-api-access-qhbvm\") pod \"network-metrics-daemon-nqtxx\" (UID: \"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d\") " pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:32.538460 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538140    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-kubelet\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.538460 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538255    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-conf-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.538460 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538291    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-etc-kubernetes\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.538460 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538294    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:32.538460 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538327    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-slash\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.538460 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538375    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75h4s\" (UniqueName: \"kubernetes.io/projected/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-kube-api-access-75h4s\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.538460 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538389    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn7ts\" (UniqueName: \"kubernetes.io/projected/098b604a-a614-4584-abc6-bfbe37ae40e9-kube-api-access-wn7ts\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:41:32.538460 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538436    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v7fm\" (UniqueName: \"kubernetes.io/projected/e0ff8a2c-7788-4791-8212-84a9a683320c-kube-api-access-5v7fm\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:32.538460 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538453    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovnkube-script-lib\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.538882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538469    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-modprobe-d\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.538882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538484    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76pzc\" (UniqueName: \"kubernetes.io/projected/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-kube-api-access-76pzc\") pod \"node-resolver-8k6rj\" (UID: \"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414\") " pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:41:32.538882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538500    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn77j\" (UniqueName: \"kubernetes.io/projected/79677abd-0ee7-4081-b31e-9f07e913fd2d-kube-api-access-zn77j\") pod \"iptables-alerter-w22p8\" (UID: \"79677abd-0ee7-4081-b31e-9f07e913fd2d\") " pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:41:32.538882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538522    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-systemd\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.538882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538568    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-run\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.538882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538592    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7qg2\" (UniqueName: \"kubernetes.io/projected/57ac048f-898e-4478-87f5-251679ad7bcf-kube-api-access-l7qg2\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.538882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538629    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae89a416-9d4c-4477-bf72-a5371f55701a-cni-binary-copy\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.538882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538659    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-systemd\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.538882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538693    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-ovnkube-script-lib\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.538882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538683    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-lib-modules\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.538882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538743    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:41:32.538882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538785    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-tmp-dir\") pod \"node-resolver-8k6rj\" (UID: \"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414\") " pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:41:32.538882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538814    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f6870c47-e336-47b5-adfa-7143eed6d6ab-serviceca\") pod \"node-ca-7zgwc\" (UID: \"f6870c47-e336-47b5-adfa-7143eed6d6ab\") " pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:41:32.538882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538830    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ae89a416-9d4c-4477-bf72-a5371f55701a-cni-binary-copy\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.538882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538840    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:32.538882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538863    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-netns\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.538882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538879    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-run-netns\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.539648 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538893    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-tmp-dir\") pod \"node-resolver-8k6rj\" (UID: \"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414\") " pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:41:32.539648 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.538970    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-device-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.539648 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539339    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79677abd-0ee7-4081-b31e-9f07e913fd2d-host-slash\") pod \"iptables-alerter-w22p8\" (UID: \"79677abd-0ee7-4081-b31e-9f07e913fd2d\") " pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:41:32.539648 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539374    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-cni-bin\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.539648 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539398    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-var-lib-kubelet\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.539648 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539413    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6870c47-e336-47b5-adfa-7143eed6d6ab-host\") pod \"node-ca-7zgwc\" (UID: \"f6870c47-e336-47b5-adfa-7143eed6d6ab\") " pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:41:32.539648 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539444    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs\") pod \"network-metrics-daemon-nqtxx\" (UID: \"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d\") " pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:32.539648 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539461    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-etc-selinux\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.539648 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539491    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-system-cni-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.539648 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539503    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f6870c47-e336-47b5-adfa-7143eed6d6ab-serviceca\") pod \"node-ca-7zgwc\" (UID: \"f6870c47-e336-47b5-adfa-7143eed6d6ab\") " pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:41:32.539648 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539567    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-k8s-cni-cncf-io\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.539648 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539604    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pq68\" (UniqueName: \"kubernetes.io/projected/f6870c47-e336-47b5-adfa-7143eed6d6ab-kube-api-access-8pq68\") pod \"node-ca-7zgwc\" (UID: \"f6870c47-e336-47b5-adfa-7143eed6d6ab\") " pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:41:32.539648 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539636    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.540043 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539659    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.540043 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539684    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-cni-netd\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.540043 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539706    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysctl-d\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.540043 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539730    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:41:32.540043 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539764    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-hosts-file\") pod \"node-resolver-8k6rj\" (UID: \"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414\") " pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:41:32.540043 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539877    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4be6fff-4a26-4188-8ca7-80125df121d0-cert\") pod \"ingress-canary-5b69x\" (UID: \"e4be6fff-4a26-4188-8ca7-80125df121d0\") " pod="openshift-ingress-canary/ingress-canary-5b69x"
Oct 30 00:41:32.540043 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539880    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/098b604a-a614-4584-abc6-bfbe37ae40e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:41:32.540043 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539910    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.540043 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.539935    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-env-overrides\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.540043 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.540039    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysconfig\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.540442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.540070    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-kubernetes\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.540442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.540092    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4be6fff-4a26-4188-8ca7-80125df121d0-cert\") pod \"ingress-canary-5b69x\" (UID: \"e4be6fff-4a26-4188-8ca7-80125df121d0\") " pod="openshift-ingress-canary/ingress-canary-5b69x"
Oct 30 00:41:32.540442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.540095    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-tmp\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.540442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.540141    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3822276d-acd5-4f16-ac86-32e8978c1f9f-data-volume\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:32.540442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.540167    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-tmp\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.540442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.540146    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d1bada76-8261-4ed7-adaf-8de93cf733c9-env-overrides\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.540442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.540213    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3822276d-acd5-4f16-ac86-32e8978c1f9f-data-volume\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:32.540442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.540231    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-registration-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.540442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.540258    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l57jg\" (UniqueName: \"kubernetes.io/projected/ae89a416-9d4c-4477-bf72-a5371f55701a-kube-api-access-l57jg\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.540442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.540282    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-system-cni-dir\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:41:32.540442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.540330    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:32.540442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.540382    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:32.540789 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.540484    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-socket-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.540789 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.540587    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/372da31c-aac8-485a-bcc4-31a830f176a9-original-pull-secret\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:32.559731 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.559709    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t7j2\" (UniqueName: \"kubernetes.io/projected/d1bada76-8261-4ed7-adaf-8de93cf733c9-kube-api-access-6t7j2\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.561453 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.561427    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn7ts\" (UniqueName: \"kubernetes.io/projected/098b604a-a614-4584-abc6-bfbe37ae40e9-kube-api-access-wn7ts\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:41:32.561509 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.561427    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75h4s\" (UniqueName: \"kubernetes.io/projected/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-kube-api-access-75h4s\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.561509 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.561497    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz7mb\" (UniqueName: \"kubernetes.io/projected/3822276d-acd5-4f16-ac86-32e8978c1f9f-kube-api-access-sz7mb\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:32.561564 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.561546    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fswtt\" (UniqueName: \"kubernetes.io/projected/e4be6fff-4a26-4188-8ca7-80125df121d0-kube-api-access-fswtt\") pod \"ingress-canary-5b69x\" (UID: \"e4be6fff-4a26-4188-8ca7-80125df121d0\") " pod="openshift-ingress-canary/ingress-canary-5b69x"
Oct 30 00:41:32.565676 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.565657    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn77j\" (UniqueName: \"kubernetes.io/projected/79677abd-0ee7-4081-b31e-9f07e913fd2d-kube-api-access-zn77j\") pod \"iptables-alerter-w22p8\" (UID: \"79677abd-0ee7-4081-b31e-9f07e913fd2d\") " pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:41:32.565831 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.565815    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhbvm\" (UniqueName: \"kubernetes.io/projected/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-kube-api-access-qhbvm\") pod \"network-metrics-daemon-nqtxx\" (UID: \"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d\") " pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:32.566300 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.566285    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7qg2\" (UniqueName: \"kubernetes.io/projected/57ac048f-898e-4478-87f5-251679ad7bcf-kube-api-access-l7qg2\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.566450 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.566429    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pq68\" (UniqueName: \"kubernetes.io/projected/f6870c47-e336-47b5-adfa-7143eed6d6ab-kube-api-access-8pq68\") pod \"node-ca-7zgwc\" (UID: \"f6870c47-e336-47b5-adfa-7143eed6d6ab\") " pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:41:32.566522 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.566509    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v7fm\" (UniqueName: \"kubernetes.io/projected/e0ff8a2c-7788-4791-8212-84a9a683320c-kube-api-access-5v7fm\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:32.567009 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.566991    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jrrr\" (UniqueName: \"kubernetes.io/projected/93171d29-b786-4586-80aa-9643856f242f-kube-api-access-9jrrr\") pod \"network-check-target-qs6rx\" (UID: \"93171d29-b786-4586-80aa-9643856f242f\") " pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:32.567139 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.567121    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l57jg\" (UniqueName: \"kubernetes.io/projected/ae89a416-9d4c-4477-bf72-a5371f55701a-kube-api-access-l57jg\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.567404 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.567388    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqngb\" (UniqueName: \"kubernetes.io/projected/cb1e136c-a300-45e4-9031-ea39904518a5-kube-api-access-bqngb\") pod \"migrator-66df5d496b-xkblh\" (UID: \"cb1e136c-a300-45e4-9031-ea39904518a5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkblh"
Oct 30 00:41:32.571366 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.571349    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76pzc\" (UniqueName: \"kubernetes.io/projected/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-kube-api-access-76pzc\") pod \"node-resolver-8k6rj\" (UID: \"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414\") " pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:41:32.641766 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.641700    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-modprobe-d\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.641766 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.641729    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-systemd\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.641766 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.641743    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-run\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.641766 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.641757    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-systemd\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.641955 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.641833    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-run\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.641955 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.641833    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-systemd\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.641955 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.641870    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-systemd\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.641955 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.641868    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-modprobe-d\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.641955 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.641882    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-lib-modules\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.641955 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.641917    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:41:32.641955 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.641939    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.641961    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-netns\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.641970    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.641987    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-run-netns\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642009    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-netns\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.641971    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-lib-modules\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642010    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-device-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642050    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-run-netns\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642077    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79677abd-0ee7-4081-b31e-9f07e913fd2d-host-slash\") pod \"iptables-alerter-w22p8\" (UID: \"79677abd-0ee7-4081-b31e-9f07e913fd2d\") " pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642095    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-cni-bin\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642046    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-device-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642109    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-var-lib-kubelet\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642127    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6870c47-e336-47b5-adfa-7143eed6d6ab-host\") pod \"node-ca-7zgwc\" (UID: \"f6870c47-e336-47b5-adfa-7143eed6d6ab\") " pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642169    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs\") pod \"network-metrics-daemon-nqtxx\" (UID: \"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d\") " pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642184    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79677abd-0ee7-4081-b31e-9f07e913fd2d-host-slash\") pod \"iptables-alerter-w22p8\" (UID: \"79677abd-0ee7-4081-b31e-9f07e913fd2d\") " pod="openshift-network-operator/iptables-alerter-w22p8"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642187    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-var-lib-kubelet\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642193    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-etc-selinux\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642185    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-cni-bin\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642195    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6870c47-e336-47b5-adfa-7143eed6d6ab-host\") pod \"node-ca-7zgwc\" (UID: \"f6870c47-e336-47b5-adfa-7143eed6d6ab\") " pod="openshift-image-registry/node-ca-7zgwc"
Oct 30 00:41:32.642209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642216    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-system-cni-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642229    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-etc-selinux\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642236    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-k8s-cni-cncf-io\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642251    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642266    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642286    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-cni-netd\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642291    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-k8s-cni-cncf-io\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642304    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysctl-d\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642296    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-system-cni-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642305    4425 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642328    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-hosts-file\") pod \"node-resolver-8k6rj\" (UID: \"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414\") " pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642334    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-cni-netd\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642352    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642354    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642391    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysconfig\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642393    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8b80e7e-75db-4e4e-9c1b-0d2f4b754414-hosts-file\") pod \"node-resolver-8k6rj\" (UID: \"f8b80e7e-75db-4e4e-9c1b-0d2f4b754414\") " pod="openshift-dns/node-resolver-8k6rj"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642391    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642414    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-kubernetes\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642433    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysconfig\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642434    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysctl-d\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642305    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642459    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-kubernetes\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642463    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-registration-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642485    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-system-cni-dir\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642507    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642525    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-socket-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642547    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-cni-multus\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642548    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-registration-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642563    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-system-cni-dir\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642574    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-kubelet\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642595    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-etc-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642609    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-kubelet\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642608    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-cni-multus\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:41:32.642621    4425 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642643    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-etc-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642650    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-ovn\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642618    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-run-ovn\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:41:32.642685    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls podName:3822276d-acd5-4f16-ac86-32e8978c1f9f nodeName:}" failed. No retries permitted until 2026-10-30 00:41:33.142667431 +0000 UTC m=+1.892322740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jz4ff" (UID: "3822276d-acd5-4f16-ac86-32e8978c1f9f") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642702    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysctl-conf\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642723    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-cni-bin\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642745    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-node-log\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642768    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-log-socket\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642790    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-cnibin\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642812    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3822276d-acd5-4f16-ac86-32e8978c1f9f-crio-socket\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642819    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-node-log\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642824    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-etc-sysctl-conf\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642833    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642844    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-cni-bin\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642861    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-socket-dir-parent\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642868    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/098b604a-a614-4584-abc6-bfbe37ae40e9-cnibin\") pod \"multus-additional-cni-plugins-q7mpt\" (UID: \"098b604a-a614-4584-abc6-bfbe37ae40e9\") " pod="openshift-multus/multus-additional-cni-plugins-q7mpt"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642873    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-log-socket\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642879    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3822276d-acd5-4f16-ac86-32e8978c1f9f-crio-socket\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642884    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-systemd-units\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642912    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642918    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-systemd-units\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642918    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-sys\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642920    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-socket-dir-parent\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642959    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-sys\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642985    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-hostroot\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.642990    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-socket-dir\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643006    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-multus-certs\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643022    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-host\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643043    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/372da31c-aac8-485a-bcc4-31a830f176a9-kubelet-config\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643060    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/372da31c-aac8-485a-bcc4-31a830f176a9-dbus\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643071    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-run-multus-certs\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643083    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-sys-fs\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643105    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-cni-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643060    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-hostroot\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643112    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/372da31c-aac8-485a-bcc4-31a830f176a9-dbus\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643084    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03bdb0b4-fc37-4886-93eb-9dc6ce910e63-host\") pod \"tuned-nn559\" (UID: \"03bdb0b4-fc37-4886-93eb-9dc6ce910e63\") " pod="openshift-cluster-node-tuning-operator/tuned-nn559"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643098    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/372da31c-aac8-485a-bcc4-31a830f176a9-kubelet-config\") pod \"global-pull-secret-syncer-fw99d\" (UID: \"372da31c-aac8-485a-bcc4-31a830f176a9\") " pod="kube-system/global-pull-secret-syncer-fw99d"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643141    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-cnibin\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643169    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-cni-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.643102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643123    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57ac048f-898e-4478-87f5-251679ad7bcf-sys-fs\") pod \"aws-ebs-csi-driver-node-7kb9j\" (UID: \"57ac048f-898e-4478-87f5-251679ad7bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kb9j"
Oct 30 00:41:32.645562 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643182    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-var-lib-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.645562 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643207    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-var-lib-openvswitch\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.645562 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643210    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-cnibin\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.645562 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643213    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-kubelet\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.645562 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643240    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-host-var-lib-kubelet\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.645562 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643247    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-conf-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.645562 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643269    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-etc-kubernetes\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.645562 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643294    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-slash\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.645562 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643341    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-multus-conf-dir\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.645562 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643343    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ae89a416-9d4c-4477-bf72-a5371f55701a-etc-kubernetes\") pod \"multus-xjww4\" (UID: \"ae89a416-9d4c-4477-bf72-a5371f55701a\") " pod="openshift-multus/multus-xjww4"
Oct 30 00:41:32.645562 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.643452    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1bada76-8261-4ed7-adaf-8de93cf733c9-host-slash\") pod \"ovnkube-node-857b8\" (UID: \"d1bada76-8261-4ed7-adaf-8de93cf733c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:32.645562 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.644527    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0ff8a2c-7788-4791-8212-84a9a683320c-metrics-tls\") pod \"dns-default-66wbz\" (UID: \"e0ff8a2c-7788-4791-8212-84a9a683320c\") " pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:32.645562 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.644690    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a4ab2cc-af6d-460b-b8dd-9f16078cc11d-metrics-certs\") pod \"network-metrics-daemon-nqtxx\" (UID: \"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d\") " pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:32.682521 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.682497    4425 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lmgn_f65424f1-dbb4-4e42-ab0a-a3334cef4dba/konnectivity-agent/0.log" limit=1024
Oct 30 00:41:32.711474 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.711456    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nqtxx"
Oct 30 00:41:32.714711 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.714695    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:32.899194 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.899106    4425 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lmgn_f65424f1-dbb4-4e42-ab0a-a3334cef4dba/konnectivity-agent/0.log" limit=1024
Oct 30 00:41:32.906518 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:32.906490    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:33.044752 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:33.044716    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nqtxx"]
Oct 30 00:41:33.048766 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:33.048742    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4ab2cc_af6d_460b_b8dd_9f16078cc11d.slice/crio-f5c5bc292358a5f7d752786ae6935a4b1dd2df7a59039baef253f542d8b9e5e6 WatchSource:0}: Error finding container f5c5bc292358a5f7d752786ae6935a4b1dd2df7a59039baef253f542d8b9e5e6: Status 404 returned error can't find the container with id f5c5bc292358a5f7d752786ae6935a4b1dd2df7a59039baef253f542d8b9e5e6
Oct 30 00:41:33.050852 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:33.050833    4425 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:41:33.070613 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:33.070589    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-66wbz"]
Oct 30 00:41:33.073956 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:33.073931    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ff8a2c_7788_4791_8212_84a9a683320c.slice/crio-75e392027da44087136eb5e120a0997593b3d48d2407523bfdd6f40b874c8caf WatchSource:0}: Error finding container 75e392027da44087136eb5e120a0997593b3d48d2407523bfdd6f40b874c8caf: Status 404 returned error can't find the container with id 75e392027da44087136eb5e120a0997593b3d48d2407523bfdd6f40b874c8caf
Oct 30 00:41:33.101593 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:33.101568    4425 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lmgn_f65424f1-dbb4-4e42-ab0a-a3334cef4dba/konnectivity-agent/0.log" limit=1024
Oct 30 00:41:33.147449 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:33.147418    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:33.147603 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:41:33.147552    4425 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:33.147661 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:41:33.147620    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls podName:3822276d-acd5-4f16-ac86-32e8978c1f9f nodeName:}" failed. No retries permitted until 2026-10-30 00:41:34.147601789 +0000 UTC m=+2.897257089 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jz4ff" (UID: "3822276d-acd5-4f16-ac86-32e8978c1f9f") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:33.450559 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:33.450524    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-66wbz" event={"ID":"e0ff8a2c-7788-4791-8212-84a9a683320c","Type":"ContainerStarted","Data":"75e392027da44087136eb5e120a0997593b3d48d2407523bfdd6f40b874c8caf"}
Oct 30 00:41:33.451658 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:33.451631    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nqtxx" event={"ID":"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d","Type":"ContainerStarted","Data":"f5c5bc292358a5f7d752786ae6935a4b1dd2df7a59039baef253f542d8b9e5e6"}
Oct 30 00:41:34.154985 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:34.153405    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:34.154985 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:41:34.153575    4425 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:34.154985 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:41:34.153643    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls podName:3822276d-acd5-4f16-ac86-32e8978c1f9f nodeName:}" failed. No retries permitted until 2026-10-30 00:41:36.15362931 +0000 UTC m=+4.903284608 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jz4ff" (UID: "3822276d-acd5-4f16-ac86-32e8978c1f9f") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:34.456087 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:34.456000    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nqtxx" event={"ID":"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d","Type":"ContainerStarted","Data":"c19e8c7f775f2e3091dc38e2e394a7a1d0f3b38ee286be1955099c02fa7ae7aa"}
Oct 30 00:41:34.456472 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:34.456098    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nqtxx" event={"ID":"6a4ab2cc-af6d-460b-b8dd-9f16078cc11d","Type":"ContainerStarted","Data":"73d9ea8883b30de7565c7a9d617ca4e94f3c6ecc39ba0ca737348a55b3953497"}
Oct 30 00:41:35.460901 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:35.460866    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-66wbz" event={"ID":"e0ff8a2c-7788-4791-8212-84a9a683320c","Type":"ContainerStarted","Data":"bd986673b597a10d6dd9f8482b3964ea8ddc661216f2d44897d6df7fd2fe8ebc"}
Oct 30 00:41:35.461385 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:35.460917    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-66wbz" event={"ID":"e0ff8a2c-7788-4791-8212-84a9a683320c","Type":"ContainerStarted","Data":"20db4535960329faea58278d2d5fef21997667df60b87c0145e8cbeeac146c3b"}
Oct 30 00:41:36.163479 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:36.163431    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:36.163671 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:41:36.163575    4425 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:36.163671 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:41:36.163652    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls podName:3822276d-acd5-4f16-ac86-32e8978c1f9f nodeName:}" failed. No retries permitted until 2026-10-30 00:41:40.163635434 +0000 UTC m=+8.913290786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jz4ff" (UID: "3822276d-acd5-4f16-ac86-32e8978c1f9f") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:41:36.464211 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:36.464134    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:39.204988 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:39.204954    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:40.181449 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:40.181411    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:40.183758 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:40.183730    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3822276d-acd5-4f16-ac86-32e8978c1f9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz4ff\" (UID: \"3822276d-acd5-4f16-ac86-32e8978c1f9f\") " pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:40.216617 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:40.216592    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jz4ff"
Oct 30 00:41:40.508123 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:40.508045    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jz4ff"]
Oct 30 00:41:40.512077 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:40.512049    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3822276d_acd5_4f16_ac86_32e8978c1f9f.slice/crio-7331ef5bbfbd7ee2e603f0c3497f480afac52e18da33ad97f97b0b8516081eca WatchSource:0}: Error finding container 7331ef5bbfbd7ee2e603f0c3497f480afac52e18da33ad97f97b0b8516081eca: Status 404 returned error can't find the container with id 7331ef5bbfbd7ee2e603f0c3497f480afac52e18da33ad97f97b0b8516081eca
Oct 30 00:41:41.482073 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:41.481989    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jz4ff" event={"ID":"3822276d-acd5-4f16-ac86-32e8978c1f9f","Type":"ContainerStarted","Data":"8e478c711f65cb5e1830d907d1dcd37bf556f598fadc30fc3197842a0a2f6fb9"}
Oct 30 00:41:41.482073 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:41.482047    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jz4ff" event={"ID":"3822276d-acd5-4f16-ac86-32e8978c1f9f","Type":"ContainerStarted","Data":"a95f36fadd123f354c48f2f6751e7e11831c3a677007db2398e785e37965a81e"}
Oct 30 00:41:41.482073 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:41.482065    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jz4ff" event={"ID":"3822276d-acd5-4f16-ac86-32e8978c1f9f","Type":"ContainerStarted","Data":"7331ef5bbfbd7ee2e603f0c3497f480afac52e18da33ad97f97b0b8516081eca"}
Oct 30 00:41:41.893088 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:41.893054    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:41.893336 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:41.893306    4425 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:41:43.490029 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:43.489995    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jz4ff" event={"ID":"3822276d-acd5-4f16-ac86-32e8978c1f9f","Type":"ContainerStarted","Data":"9aaf865893df7b908c08dc6fbf6b7298395549fdcd26b1ed0380f13145c3c4ab"}
Oct 30 00:41:43.515946 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:43.515921    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5lmgn"
Oct 30 00:41:46.393740 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.393710    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-fptfc"]
Oct 30 00:41:46.395727 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.395701    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-fptfc"
Oct 30 00:41:46.403240 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.403217    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t5vgx"]
Oct 30 00:41:46.404854 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.404839    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t5vgx"
Oct 30 00:41:46.413858 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.413842    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-nbwsc\""
Oct 30 00:41:46.415620 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.415606    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 00:41:46.424534 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.424520    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 00:41:46.440019 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.440002    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t5vgx"]
Oct 30 00:41:46.441638 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.441620    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-fptfc"]
Oct 30 00:41:46.470684 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.470662    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-66wbz"
Oct 30 00:41:46.514877 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.514851    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-s7nbs\""
Oct 30 00:41:46.514986 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.514967    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 00:41:46.522240 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.522218    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22756\" (UniqueName: \"kubernetes.io/projected/75d61c2d-5a46-45a7-b9d7-fa072c685bde-kube-api-access-22756\") pod \"downloads-6dd789c8b7-fptfc\" (UID: \"75d61c2d-5a46-45a7-b9d7-fa072c685bde\") " pod="openshift-console/downloads-6dd789c8b7-fptfc"
Oct 30 00:41:46.522316 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.522252    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f64d1922-82b0-4b57-8ec5-032d8ec8da4f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-t5vgx\" (UID: \"f64d1922-82b0-4b57-8ec5-032d8ec8da4f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t5vgx"
Oct 30 00:41:46.522316 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.522280    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75d61c2d-5a46-45a7-b9d7-fa072c685bde-tmp\") pod \"downloads-6dd789c8b7-fptfc\" (UID: \"75d61c2d-5a46-45a7-b9d7-fa072c685bde\") " pod="openshift-console/downloads-6dd789c8b7-fptfc"
Oct 30 00:41:46.622769 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.622744    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75d61c2d-5a46-45a7-b9d7-fa072c685bde-tmp\") pod \"downloads-6dd789c8b7-fptfc\" (UID: \"75d61c2d-5a46-45a7-b9d7-fa072c685bde\") " pod="openshift-console/downloads-6dd789c8b7-fptfc"
Oct 30 00:41:46.622918 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.622803    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22756\" (UniqueName: \"kubernetes.io/projected/75d61c2d-5a46-45a7-b9d7-fa072c685bde-kube-api-access-22756\") pod \"downloads-6dd789c8b7-fptfc\" (UID: \"75d61c2d-5a46-45a7-b9d7-fa072c685bde\") " pod="openshift-console/downloads-6dd789c8b7-fptfc"
Oct 30 00:41:46.622918 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.622837    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f64d1922-82b0-4b57-8ec5-032d8ec8da4f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-t5vgx\" (UID: \"f64d1922-82b0-4b57-8ec5-032d8ec8da4f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t5vgx"
Oct 30 00:41:46.623201 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.623180    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75d61c2d-5a46-45a7-b9d7-fa072c685bde-tmp\") pod \"downloads-6dd789c8b7-fptfc\" (UID: \"75d61c2d-5a46-45a7-b9d7-fa072c685bde\") " pod="openshift-console/downloads-6dd789c8b7-fptfc"
Oct 30 00:41:46.625121 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.625094    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f64d1922-82b0-4b57-8ec5-032d8ec8da4f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-t5vgx\" (UID: \"f64d1922-82b0-4b57-8ec5-032d8ec8da4f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t5vgx"
Oct 30 00:41:46.650997 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.650938    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22756\" (UniqueName: \"kubernetes.io/projected/75d61c2d-5a46-45a7-b9d7-fa072c685bde-kube-api-access-22756\") pod \"downloads-6dd789c8b7-fptfc\" (UID: \"75d61c2d-5a46-45a7-b9d7-fa072c685bde\") " pod="openshift-console/downloads-6dd789c8b7-fptfc"
Oct 30 00:41:46.705573 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.705541    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-fptfc"
Oct 30 00:41:46.713250 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:46.713233    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t5vgx"
Oct 30 00:41:47.008112 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:47.008040    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t5vgx"]
Oct 30 00:41:47.011636 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:47.011605    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64d1922_82b0_4b57_8ec5_032d8ec8da4f.slice/crio-b602f0e2d34be5b85f122bfca896ccf0ec21037a42855027584d5dcf90bdff38 WatchSource:0}: Error finding container b602f0e2d34be5b85f122bfca896ccf0ec21037a42855027584d5dcf90bdff38: Status 404 returned error can't find the container with id b602f0e2d34be5b85f122bfca896ccf0ec21037a42855027584d5dcf90bdff38
Oct 30 00:41:47.011723 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:47.011666    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-fptfc"]
Oct 30 00:41:47.014835 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:47.014814    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d61c2d_5a46_45a7_b9d7_fa072c685bde.slice/crio-b7ffb07c0eb7302f4e964b66e51088ffe561d77342fe4b9db722ae85b2661061 WatchSource:0}: Error finding container b7ffb07c0eb7302f4e964b66e51088ffe561d77342fe4b9db722ae85b2661061: Status 404 returned error can't find the container with id b7ffb07c0eb7302f4e964b66e51088ffe561d77342fe4b9db722ae85b2661061
Oct 30 00:41:47.508701 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:47.508660    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t5vgx" event={"ID":"f64d1922-82b0-4b57-8ec5-032d8ec8da4f","Type":"ContainerStarted","Data":"b602f0e2d34be5b85f122bfca896ccf0ec21037a42855027584d5dcf90bdff38"}
Oct 30 00:41:47.511290 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:47.511257    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-fptfc" event={"ID":"75d61c2d-5a46-45a7-b9d7-fa072c685bde","Type":"ContainerStarted","Data":"b7ffb07c0eb7302f4e964b66e51088ffe561d77342fe4b9db722ae85b2661061"}
Oct 30 00:41:48.516768 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:48.516726    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t5vgx" event={"ID":"f64d1922-82b0-4b57-8ec5-032d8ec8da4f","Type":"ContainerStarted","Data":"e614f02782ac375cedb1dba797db0b42326d3fa8e47596e17141c8ea63835d65"}
Oct 30 00:41:48.517223 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:48.516932    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t5vgx"
Oct 30 00:41:48.522373 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:48.522350    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t5vgx"
Oct 30 00:41:48.552094 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:48.552031    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t5vgx" podStartSLOduration=1.613399526 podStartE2EDuration="2.552014714s" totalImagesPullingTime="938.615188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:48.551127682 +0000 UTC m=+17.300782992" watchObservedRunningTime="2026-10-30 00:41:48.552014714 +0000 UTC m=+17.301670026"
Oct 30 00:41:49.595906 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.595873    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4"]
Oct 30 00:41:49.598621 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.598596    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4"
Oct 30 00:41:49.611272 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.611250    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 00:41:49.621191 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.621100    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 00:41:49.621191 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.621100    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 00:41:49.621191 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.621111    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 00:41:49.621441 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.621207    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ffl8s\""
Oct 30 00:41:49.621441 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.621254    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 00:41:49.621991 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.621972    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4"]
Oct 30 00:41:49.746880 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.746830    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxkw\" (UniqueName: \"kubernetes.io/projected/4a2f55a8-9612-47a7-97cd-aeeb955b6659-kube-api-access-hzxkw\") pod \"prometheus-operator-9cd89b74c-j5kh4\" (UID: \"4a2f55a8-9612-47a7-97cd-aeeb955b6659\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4"
Oct 30 00:41:49.747041 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.747012    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a2f55a8-9612-47a7-97cd-aeeb955b6659-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-j5kh4\" (UID: \"4a2f55a8-9612-47a7-97cd-aeeb955b6659\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4"
Oct 30 00:41:49.747110 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.747088    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a2f55a8-9612-47a7-97cd-aeeb955b6659-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-j5kh4\" (UID: \"4a2f55a8-9612-47a7-97cd-aeeb955b6659\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4"
Oct 30 00:41:49.747195 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.747128    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a2f55a8-9612-47a7-97cd-aeeb955b6659-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-j5kh4\" (UID: \"4a2f55a8-9612-47a7-97cd-aeeb955b6659\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4"
Oct 30 00:41:49.847905 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.847826    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxkw\" (UniqueName: \"kubernetes.io/projected/4a2f55a8-9612-47a7-97cd-aeeb955b6659-kube-api-access-hzxkw\") pod \"prometheus-operator-9cd89b74c-j5kh4\" (UID: \"4a2f55a8-9612-47a7-97cd-aeeb955b6659\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4"
Oct 30 00:41:49.847905 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.847882    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a2f55a8-9612-47a7-97cd-aeeb955b6659-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-j5kh4\" (UID: \"4a2f55a8-9612-47a7-97cd-aeeb955b6659\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4"
Oct 30 00:41:49.847905 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.847908    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a2f55a8-9612-47a7-97cd-aeeb955b6659-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-j5kh4\" (UID: \"4a2f55a8-9612-47a7-97cd-aeeb955b6659\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4"
Oct 30 00:41:49.848198 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.847932    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a2f55a8-9612-47a7-97cd-aeeb955b6659-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-j5kh4\" (UID: \"4a2f55a8-9612-47a7-97cd-aeeb955b6659\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4"
Oct 30 00:41:49.848635 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.848604    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a2f55a8-9612-47a7-97cd-aeeb955b6659-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-j5kh4\" (UID: \"4a2f55a8-9612-47a7-97cd-aeeb955b6659\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4"
Oct 30 00:41:49.850875 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.850849    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a2f55a8-9612-47a7-97cd-aeeb955b6659-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-j5kh4\" (UID: \"4a2f55a8-9612-47a7-97cd-aeeb955b6659\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4"
Oct 30 00:41:49.850995 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.850889    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a2f55a8-9612-47a7-97cd-aeeb955b6659-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-j5kh4\" (UID: \"4a2f55a8-9612-47a7-97cd-aeeb955b6659\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4"
Oct 30 00:41:49.869040 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.869017    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxkw\" (UniqueName: \"kubernetes.io/projected/4a2f55a8-9612-47a7-97cd-aeeb955b6659-kube-api-access-hzxkw\") pod \"prometheus-operator-9cd89b74c-j5kh4\" (UID: \"4a2f55a8-9612-47a7-97cd-aeeb955b6659\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4"
Oct 30 00:41:49.913101 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:49.913078    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4"
Oct 30 00:41:50.209616 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:50.209548    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4"]
Oct 30 00:41:50.213561 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:50.213530    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2f55a8_9612_47a7_97cd_aeeb955b6659.slice/crio-65b71e3e28534bec9029375a8a287f7d2c0cd3e5d23ebacfb5f5458ff288bd5d WatchSource:0}: Error finding container 65b71e3e28534bec9029375a8a287f7d2c0cd3e5d23ebacfb5f5458ff288bd5d: Status 404 returned error can't find the container with id 65b71e3e28534bec9029375a8a287f7d2c0cd3e5d23ebacfb5f5458ff288bd5d
Oct 30 00:41:50.526117 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:50.526080    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4" event={"ID":"4a2f55a8-9612-47a7-97cd-aeeb955b6659","Type":"ContainerStarted","Data":"65b71e3e28534bec9029375a8a287f7d2c0cd3e5d23ebacfb5f5458ff288bd5d"}
Oct 30 00:41:51.538730 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:51.538699    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4" event={"ID":"4a2f55a8-9612-47a7-97cd-aeeb955b6659","Type":"ContainerStarted","Data":"1884c31419b46a9b57b4fd4a208eeb6bc7c73ff97b7c5c73d299267ddfd5b5a8"}
Oct 30 00:41:51.539104 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:51.538789    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4" event={"ID":"4a2f55a8-9612-47a7-97cd-aeeb955b6659","Type":"ContainerStarted","Data":"6a381f6e8500370ee3d0e6661faf2f33e73b24fa3a7acf3202d68f314095618e"}
Oct 30 00:41:52.126821 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.126756    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j5kh4" podStartSLOduration=1.9349101069999999 podStartE2EDuration="3.126736089s" totalImagesPullingTime="1.191825982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:51.580978068 +0000 UTC m=+20.330633381" watchObservedRunningTime="2026-10-30 00:41:52.126736089 +0000 UTC m=+20.876391602"
Oct 30 00:41:52.127020 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.127004    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-85cd688b89-8tksl"]
Oct 30 00:41:52.130441 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.130420    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.145186 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.143285    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 00:41:52.145186 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.143427    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 00:41:52.145186 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.143481    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7qn49\""
Oct 30 00:41:52.152183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.152146    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 00:41:52.152274 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.152204    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 00:41:52.153056 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.153035    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 00:41:52.153717 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.153686    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85cd688b89-8tksl"]
Oct 30 00:41:52.262474 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.262440    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1d10b60-20ba-442d-815f-01653be20945-console-oauth-config\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.262904 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.262844    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1d10b60-20ba-442d-815f-01653be20945-console-serving-cert\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.263519 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.263122    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-oauth-serving-cert\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.263519 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.263257    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-service-ca\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.263519 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.263286    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1d10b60-20ba-442d-815f-01653be20945-tmp\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.263519 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.263309    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-console-config\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.263519 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.263357    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5plcv\" (UniqueName: \"kubernetes.io/projected/d1d10b60-20ba-442d-815f-01653be20945-kube-api-access-5plcv\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.363695 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.363657    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1d10b60-20ba-442d-815f-01653be20945-console-oauth-config\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.363877 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.363704    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1d10b60-20ba-442d-815f-01653be20945-console-serving-cert\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.363877 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.363726    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-oauth-serving-cert\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.363877 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.363752    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-service-ca\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.363877 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.363774    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1d10b60-20ba-442d-815f-01653be20945-tmp\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.363877 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.363799    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-console-config\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.363877 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.363844    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5plcv\" (UniqueName: \"kubernetes.io/projected/d1d10b60-20ba-442d-815f-01653be20945-kube-api-access-5plcv\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.364646 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.364621    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1d10b60-20ba-442d-815f-01653be20945-tmp\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.364933 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.364912    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-console-config\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.365002 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.364910    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-service-ca\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.365002 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.364917    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-oauth-serving-cert\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.366456 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.366435    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1d10b60-20ba-442d-815f-01653be20945-console-oauth-config\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.366777 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.366759    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1d10b60-20ba-442d-815f-01653be20945-console-serving-cert\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.385082 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.385027    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5plcv\" (UniqueName: \"kubernetes.io/projected/d1d10b60-20ba-442d-815f-01653be20945-kube-api-access-5plcv\") pod \"console-85cd688b89-8tksl\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") " pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.443576 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.443539    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:41:52.745293 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:52.745211    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85cd688b89-8tksl"]
Oct 30 00:41:52.749199 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:52.749168    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d10b60_20ba_442d_815f_01653be20945.slice/crio-236a91cf73293893662e96421b503d2ffd6e19a8a1db92af788ec71613185073 WatchSource:0}: Error finding container 236a91cf73293893662e96421b503d2ffd6e19a8a1db92af788ec71613185073: Status 404 returned error can't find the container with id 236a91cf73293893662e96421b503d2ffd6e19a8a1db92af788ec71613185073
Oct 30 00:41:53.548035 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:53.548001    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85cd688b89-8tksl" event={"ID":"d1d10b60-20ba-442d-815f-01653be20945","Type":"ContainerStarted","Data":"236a91cf73293893662e96421b503d2ffd6e19a8a1db92af788ec71613185073"}
Oct 30 00:41:54.307014 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.305806    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr"]
Oct 30 00:41:54.309241 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.309144    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr"
Oct 30 00:41:54.326451 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.326425    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 00:41:54.337275 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.337243    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr"]
Oct 30 00:41:54.337429 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.337408    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 00:41:54.337543 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.337497    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-txkq8\""
Oct 30 00:41:54.343773 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.343750    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-857b8"
Oct 30 00:41:54.372370 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.371705    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8g6mn"]
Oct 30 00:41:54.375642 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.375615    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.392795 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.391537    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 00:41:54.426809 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.426778    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 00:41:54.426809 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.426778    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6r6tq\""
Oct 30 00:41:54.427032 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.426790    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 00:41:54.479651 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.479608    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95f31ac0-37f2-455c-a543-c7feba94a43e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.479813 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.479675    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/95f31ac0-37f2-455c-a543-c7feba94a43e-node-exporter-textfile\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.479813 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.479719    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs2fv\" (UniqueName: \"kubernetes.io/projected/95f31ac0-37f2-455c-a543-c7feba94a43e-kube-api-access-xs2fv\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.479813 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.479745    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/95f31ac0-37f2-455c-a543-c7feba94a43e-node-exporter-tls\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.479813 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.479774    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be4dc5c1-48ab-4001-a35e-725eb618b395-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-gdsjr\" (UID: \"be4dc5c1-48ab-4001-a35e-725eb618b395\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr"
Oct 30 00:41:54.479813 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.479803    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z78dj\" (UniqueName: \"kubernetes.io/projected/be4dc5c1-48ab-4001-a35e-725eb618b395-kube-api-access-z78dj\") pod \"openshift-state-metrics-7bd7489957-gdsjr\" (UID: \"be4dc5c1-48ab-4001-a35e-725eb618b395\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr"
Oct 30 00:41:54.480014 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.479848    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be4dc5c1-48ab-4001-a35e-725eb618b395-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-gdsjr\" (UID: \"be4dc5c1-48ab-4001-a35e-725eb618b395\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr"
Oct 30 00:41:54.480014 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.479863    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95f31ac0-37f2-455c-a543-c7feba94a43e-metrics-client-ca\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.480014 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.479902    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/95f31ac0-37f2-455c-a543-c7feba94a43e-node-exporter-accelerators-collector-config\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.480014 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.479929    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95f31ac0-37f2-455c-a543-c7feba94a43e-sys\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.480014 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.479962    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/be4dc5c1-48ab-4001-a35e-725eb618b395-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-gdsjr\" (UID: \"be4dc5c1-48ab-4001-a35e-725eb618b395\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr"
Oct 30 00:41:54.480014 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.479986    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/95f31ac0-37f2-455c-a543-c7feba94a43e-node-exporter-wtmp\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.480265 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.480036    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/95f31ac0-37f2-455c-a543-c7feba94a43e-root\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.580967 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.580889    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/95f31ac0-37f2-455c-a543-c7feba94a43e-node-exporter-textfile\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.580967 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.580942    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs2fv\" (UniqueName: \"kubernetes.io/projected/95f31ac0-37f2-455c-a543-c7feba94a43e-kube-api-access-xs2fv\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.581200 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.580968    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/95f31ac0-37f2-455c-a543-c7feba94a43e-node-exporter-tls\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.581200 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.580995    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be4dc5c1-48ab-4001-a35e-725eb618b395-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-gdsjr\" (UID: \"be4dc5c1-48ab-4001-a35e-725eb618b395\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr"
Oct 30 00:41:54.581200 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.581024    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z78dj\" (UniqueName: \"kubernetes.io/projected/be4dc5c1-48ab-4001-a35e-725eb618b395-kube-api-access-z78dj\") pod \"openshift-state-metrics-7bd7489957-gdsjr\" (UID: \"be4dc5c1-48ab-4001-a35e-725eb618b395\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr"
Oct 30 00:41:54.581200 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.581057    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be4dc5c1-48ab-4001-a35e-725eb618b395-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-gdsjr\" (UID: \"be4dc5c1-48ab-4001-a35e-725eb618b395\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr"
Oct 30 00:41:54.581200 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.581080    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95f31ac0-37f2-455c-a543-c7feba94a43e-metrics-client-ca\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.581200 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.581114    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/95f31ac0-37f2-455c-a543-c7feba94a43e-node-exporter-accelerators-collector-config\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.581200 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.581151    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95f31ac0-37f2-455c-a543-c7feba94a43e-sys\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.581542 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.581204    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/be4dc5c1-48ab-4001-a35e-725eb618b395-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-gdsjr\" (UID: \"be4dc5c1-48ab-4001-a35e-725eb618b395\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr"
Oct 30 00:41:54.581542 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.581227    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/95f31ac0-37f2-455c-a543-c7feba94a43e-node-exporter-wtmp\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.581542 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.581260    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/95f31ac0-37f2-455c-a543-c7feba94a43e-root\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.581542 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.581301    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95f31ac0-37f2-455c-a543-c7feba94a43e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.581542 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.581344    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/95f31ac0-37f2-455c-a543-c7feba94a43e-node-exporter-textfile\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.582920 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.581819    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95f31ac0-37f2-455c-a543-c7feba94a43e-metrics-client-ca\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.582920 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.582253    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/95f31ac0-37f2-455c-a543-c7feba94a43e-node-exporter-wtmp\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.582920 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.582312    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/95f31ac0-37f2-455c-a543-c7feba94a43e-root\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.582920 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.582759    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/95f31ac0-37f2-455c-a543-c7feba94a43e-node-exporter-accelerators-collector-config\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.582920 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.582821    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95f31ac0-37f2-455c-a543-c7feba94a43e-sys\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.583398 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.583375    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be4dc5c1-48ab-4001-a35e-725eb618b395-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-gdsjr\" (UID: \"be4dc5c1-48ab-4001-a35e-725eb618b395\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr"
Oct 30 00:41:54.584188 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.584118    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95f31ac0-37f2-455c-a543-c7feba94a43e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.584562 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.584516    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/be4dc5c1-48ab-4001-a35e-725eb618b395-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-gdsjr\" (UID: \"be4dc5c1-48ab-4001-a35e-725eb618b395\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr"
Oct 30 00:41:54.584659 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.584647    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/95f31ac0-37f2-455c-a543-c7feba94a43e-node-exporter-tls\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.586329 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.586301    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be4dc5c1-48ab-4001-a35e-725eb618b395-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-gdsjr\" (UID: \"be4dc5c1-48ab-4001-a35e-725eb618b395\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr"
Oct 30 00:41:54.602690 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.602665    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs2fv\" (UniqueName: \"kubernetes.io/projected/95f31ac0-37f2-455c-a543-c7feba94a43e-kube-api-access-xs2fv\") pod \"node-exporter-8g6mn\" (UID: \"95f31ac0-37f2-455c-a543-c7feba94a43e\") " pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.602979 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.602953    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z78dj\" (UniqueName: \"kubernetes.io/projected/be4dc5c1-48ab-4001-a35e-725eb618b395-kube-api-access-z78dj\") pod \"openshift-state-metrics-7bd7489957-gdsjr\" (UID: \"be4dc5c1-48ab-4001-a35e-725eb618b395\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr"
Oct 30 00:41:54.627850 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.627814    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr"
Oct 30 00:41:54.695600 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.695568    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8g6mn"
Oct 30 00:41:54.710444 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:54.710408    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f31ac0_37f2_455c_a543_c7feba94a43e.slice/crio-398d833db4ff4f498067e498ceeb3d90a70ba9c76ecc28a6c740e83715ef4d72 WatchSource:0}: Error finding container 398d833db4ff4f498067e498ceeb3d90a70ba9c76ecc28a6c740e83715ef4d72: Status 404 returned error can't find the container with id 398d833db4ff4f498067e498ceeb3d90a70ba9c76ecc28a6c740e83715ef4d72
Oct 30 00:41:54.947008 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:54.946925    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr"]
Oct 30 00:41:54.951365 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:54.951336    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4dc5c1_48ab_4001_a35e_725eb618b395.slice/crio-c78629b6a43bc0943ab2fce1aa7576767efef437a3eccf5490a5523058f6596b WatchSource:0}: Error finding container c78629b6a43bc0943ab2fce1aa7576767efef437a3eccf5490a5523058f6596b: Status 404 returned error can't find the container with id c78629b6a43bc0943ab2fce1aa7576767efef437a3eccf5490a5523058f6596b
Oct 30 00:41:55.558174 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.558126    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr" event={"ID":"be4dc5c1-48ab-4001-a35e-725eb618b395","Type":"ContainerStarted","Data":"c69e724dd640ef7c84dea9090b9e1648d24db078ec1f85ef06e62b817a619db7"}
Oct 30 00:41:55.558639 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.558207    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr" event={"ID":"be4dc5c1-48ab-4001-a35e-725eb618b395","Type":"ContainerStarted","Data":"870aeadc3347dffbfae5de0f1112fd453916042b6baa24d5bbcbe2dc3a4f9cb5"}
Oct 30 00:41:55.558639 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.558227    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr" event={"ID":"be4dc5c1-48ab-4001-a35e-725eb618b395","Type":"ContainerStarted","Data":"c78629b6a43bc0943ab2fce1aa7576767efef437a3eccf5490a5523058f6596b"}
Oct 30 00:41:55.560234 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.560205    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8g6mn" event={"ID":"95f31ac0-37f2-455c-a543-c7feba94a43e","Type":"ContainerStarted","Data":"398d833db4ff4f498067e498ceeb3d90a70ba9c76ecc28a6c740e83715ef4d72"}
Oct 30 00:41:55.741999 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.741951    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:41:55.748553 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.747813    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.763095 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.761662    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 00:41:55.763095 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.761792    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 00:41:55.763095 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.761830    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 00:41:55.763095 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.761863    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 00:41:55.763095 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.761895    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 00:41:55.763095 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.762275    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k2z8g\""
Oct 30 00:41:55.763095 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.762387    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 00:41:55.763095 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.762433    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 00:41:55.763095 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.762480    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 00:41:55.772668 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.772643    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 00:41:55.780348 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.780324    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:41:55.894787 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.894753    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.894915 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.894873    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pglgd\" (UniqueName: \"kubernetes.io/projected/b4591819-3c83-4361-bbe4-977b5f4fd3c3-kube-api-access-pglgd\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.894972 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.894929    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4591819-3c83-4361-bbe4-977b5f4fd3c3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.894972 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.894956    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.895068 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.894977    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.895068 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.895015    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4591819-3c83-4361-bbe4-977b5f4fd3c3-config-out\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.895068 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.895039    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-config-volume\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.895229 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.895065    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.895229 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.895084    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4591819-3c83-4361-bbe4-977b5f4fd3c3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.895229 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.895107    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b4591819-3c83-4361-bbe4-977b5f4fd3c3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.895229 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.895125    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4591819-3c83-4361-bbe4-977b5f4fd3c3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.895229 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.895146    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.895229 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.895181    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-web-config\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.997057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.996371    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pglgd\" (UniqueName: \"kubernetes.io/projected/b4591819-3c83-4361-bbe4-977b5f4fd3c3-kube-api-access-pglgd\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.997057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.996411    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4591819-3c83-4361-bbe4-977b5f4fd3c3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.997057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.996782    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.997057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.996842    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.997057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.996906    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4591819-3c83-4361-bbe4-977b5f4fd3c3-config-out\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.997057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.996934    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-config-volume\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.997057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.996959    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.997057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.996986    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4591819-3c83-4361-bbe4-977b5f4fd3c3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.997057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.997014    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b4591819-3c83-4361-bbe4-977b5f4fd3c3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.997057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.997040    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4591819-3c83-4361-bbe4-977b5f4fd3c3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.997057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.997073    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.997772 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.997098    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-web-config\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.997772 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.997138    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.997772 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.997763    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b4591819-3c83-4361-bbe4-977b5f4fd3c3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.998570 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.998457    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4591819-3c83-4361-bbe4-977b5f4fd3c3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:55.998995 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:55.998960    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4591819-3c83-4361-bbe4-977b5f4fd3c3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:56.001997 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.001972    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:56.002553 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.002105    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:56.002553 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.002514    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4591819-3c83-4361-bbe4-977b5f4fd3c3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:56.004456 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.004431    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-web-config\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:56.004456 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.004444    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:56.005872 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.005350    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4591819-3c83-4361-bbe4-977b5f4fd3c3-config-out\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:56.005970 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.005892    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:56.007065 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.007042    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-config-volume\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:56.007489 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.007465    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:56.020693 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.020664    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pglgd\" (UniqueName: \"kubernetes.io/projected/b4591819-3c83-4361-bbe4-977b5f4fd3c3-kube-api-access-pglgd\") pod \"alertmanager-main-0\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:56.076631 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.076596    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:41:56.415840 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.415771    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:41:56.419967 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:56.419939    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4591819_3c83_4361_bbe4_977b5f4fd3c3.slice/crio-ab9035eae016d96cf49f6c509c13215331675895ccba6cabc641d6a68cefd58c WatchSource:0}: Error finding container ab9035eae016d96cf49f6c509c13215331675895ccba6cabc641d6a68cefd58c: Status 404 returned error can't find the container with id ab9035eae016d96cf49f6c509c13215331675895ccba6cabc641d6a68cefd58c
Oct 30 00:41:56.565190 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.563593    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"]
Oct 30 00:41:56.567784 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.567757    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.568577 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.568548    4425 generic.go:356] "Generic (PLEG): container finished" podID="95f31ac0-37f2-455c-a543-c7feba94a43e" containerID="458117ee7eb3d8021020cf156b5ae8f3f3115e6ab20c517aef02ee6d0e910b79" exitCode=0
Oct 30 00:41:56.568706 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.568636    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8g6mn" event={"ID":"95f31ac0-37f2-455c-a543-c7feba94a43e","Type":"ContainerDied","Data":"458117ee7eb3d8021020cf156b5ae8f3f3115e6ab20c517aef02ee6d0e910b79"}
Oct 30 00:41:56.570319 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.570296    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4591819-3c83-4361-bbe4-977b5f4fd3c3","Type":"ContainerStarted","Data":"ab9035eae016d96cf49f6c509c13215331675895ccba6cabc641d6a68cefd58c"}
Oct 30 00:41:56.572098 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.572071    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85cd688b89-8tksl" event={"ID":"d1d10b60-20ba-442d-815f-01653be20945","Type":"ContainerStarted","Data":"cbd27d488d92814ca10860d928d67bf58d7a2d06c01dc181c90a00b3116d392f"}
Oct 30 00:41:56.580051 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.580022    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr" event={"ID":"be4dc5c1-48ab-4001-a35e-725eb618b395","Type":"ContainerStarted","Data":"3ea7d7bfe5ce2f24f96bef3c11e547000bb36c07d4ef4513983d9bf7850f32ff"}
Oct 30 00:41:56.583633 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.583611    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 00:41:56.583839 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.583816    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 00:41:56.583942 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.583826    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 00:41:56.583942 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.583872    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 00:41:56.583942 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.583901    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1orobfv88n52g\""
Oct 30 00:41:56.584094 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.583991    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 00:41:56.584094 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.584022    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7zs5t\""
Oct 30 00:41:56.590420 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.590397    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"]
Oct 30 00:41:56.660181 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.659771    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-85cd688b89-8tksl" podStartSLOduration=1.225917243 podStartE2EDuration="4.659757592s" totalImagesPullingTime="3.433840349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:56.659413773 +0000 UTC m=+25.409069082" watchObservedRunningTime="2026-10-30 00:41:56.659757592 +0000 UTC m=+25.409412902"
Oct 30 00:41:56.696320 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.696253    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gdsjr" podStartSLOduration=1.6115793699999998 podStartE2EDuration="2.69623544s" totalImagesPullingTime="1.08465607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:56.695889649 +0000 UTC m=+25.445544958" watchObservedRunningTime="2026-10-30 00:41:56.69623544 +0000 UTC m=+25.445890751"
Oct 30 00:41:56.703760 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.703724    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-tmp\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.703871 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.703819    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-metrics-client-ca\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.703871 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.703840    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-grpc-tls\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.703974 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.703869    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.703974 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.703923    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.703974 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.703950    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.704170 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.703997    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.704170 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.704116    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-thanos-querier-tls\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.704258 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.704175    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5hh6\" (UniqueName: \"kubernetes.io/projected/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-kube-api-access-j5hh6\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.805145 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.805095    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-tmp\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.805349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.805181    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-metrics-client-ca\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.805349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.805213    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-grpc-tls\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.805349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.805248    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.805349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.805280    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.805349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.805324    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.805632 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.805365    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.805632 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.805416    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-thanos-querier-tls\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.805632 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.805444    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5hh6\" (UniqueName: \"kubernetes.io/projected/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-kube-api-access-j5hh6\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.805632 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.805555    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-tmp\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.806600 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.805946    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-metrics-client-ca\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.808419 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.808392    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.808556 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.808522    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.808618 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.808552    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.808731 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.808705    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.809097 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.809078    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-grpc-tls\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.809616 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.809594    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-secret-thanos-querier-tls\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.826463 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.826438    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5hh6\" (UniqueName: \"kubernetes.io/projected/ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0-kube-api-access-j5hh6\") pod \"thanos-querier-7bf64bb4d4-jhfpw\" (UID: \"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0\") " pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:56.894773 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:56.894736    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:41:57.222225 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:57.222122    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"]
Oct 30 00:41:57.229231 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:57.229192    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7d3dc4_01b6_4d9d_a66d_cf31bdeab3e0.slice/crio-07b971c28f2b6176af04df4b3c21d2651c443c8e98c090b7aa6679c326e5af6e WatchSource:0}: Error finding container 07b971c28f2b6176af04df4b3c21d2651c443c8e98c090b7aa6679c326e5af6e: Status 404 returned error can't find the container with id 07b971c28f2b6176af04df4b3c21d2651c443c8e98c090b7aa6679c326e5af6e
Oct 30 00:41:57.586788 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:57.586739    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8g6mn" event={"ID":"95f31ac0-37f2-455c-a543-c7feba94a43e","Type":"ContainerStarted","Data":"6d8455edb7148484d8d299380ce1a5e24fb0821ab693ef00f3566c20442a564b"}
Oct 30 00:41:57.587216 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:57.586819    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8g6mn" event={"ID":"95f31ac0-37f2-455c-a543-c7feba94a43e","Type":"ContainerStarted","Data":"1c1ea506d63ef44a70345ee23674a85167c2951ccfa01cda5f08d33eaa4aecaa"}
Oct 30 00:41:57.588912 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:57.588769    4425 generic.go:356] "Generic (PLEG): container finished" podID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerID="f0be405cd48ea0923fb509b440facaa3fe8cb52081d666d0390800a3a2bbb6dd" exitCode=0
Oct 30 00:41:57.588912 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:57.588839    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4591819-3c83-4361-bbe4-977b5f4fd3c3","Type":"ContainerDied","Data":"f0be405cd48ea0923fb509b440facaa3fe8cb52081d666d0390800a3a2bbb6dd"}
Oct 30 00:41:57.591370 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:57.591349    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw" event={"ID":"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0","Type":"ContainerStarted","Data":"07b971c28f2b6176af04df4b3c21d2651c443c8e98c090b7aa6679c326e5af6e"}
Oct 30 00:41:57.625748 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:57.625687    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8g6mn" podStartSLOduration=2.393778911 podStartE2EDuration="3.625672762s" totalImagesPullingTime="1.157374312s" totalInitContainerRuntime="74.519539ms" isStatefulPod=true podCreationTimestamp="2026-10-30 00:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:57.625144553 +0000 UTC m=+26.374799868" watchObservedRunningTime="2026-10-30 00:41:57.625672762 +0000 UTC m=+26.375328069"
Oct 30 00:41:58.787854 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:58.787815    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"]
Oct 30 00:41:58.791854 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:58.791818    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:58.804718 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:58.804686    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 00:41:58.805591 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:58.805571    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 00:41:58.818087 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:58.818058    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"]
Oct 30 00:41:58.878457 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:58.878409    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 00:41:58.878632 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:58.878602    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 00:41:58.878708 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:58.878693    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hrqll\""
Oct 30 00:41:58.878780 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:58.878767    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-q06b1jsjtb7o\""
Oct 30 00:41:58.924608 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:58.924257    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2602823e-b519-4ecd-80f5-c8c60472145b-secret-metrics-server-client-certs\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:58.924608 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:58.924318    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2602823e-b519-4ecd-80f5-c8c60472145b-client-ca-bundle\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:58.924608 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:58.924354    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2602823e-b519-4ecd-80f5-c8c60472145b-audit-log\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:58.924608 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:58.924376    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2602823e-b519-4ecd-80f5-c8c60472145b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:58.924608 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:58.924398    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2602823e-b519-4ecd-80f5-c8c60472145b-metrics-server-audit-profiles\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:58.924608 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:58.924429    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d87lp\" (UniqueName: \"kubernetes.io/projected/2602823e-b519-4ecd-80f5-c8c60472145b-kube-api-access-d87lp\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:58.924608 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:58.924469    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2602823e-b519-4ecd-80f5-c8c60472145b-secret-metrics-server-tls\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:59.025551 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.025514    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2602823e-b519-4ecd-80f5-c8c60472145b-secret-metrics-server-client-certs\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:59.025721 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.025578    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2602823e-b519-4ecd-80f5-c8c60472145b-client-ca-bundle\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:59.025721 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.025615    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2602823e-b519-4ecd-80f5-c8c60472145b-audit-log\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:59.025721 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.025647    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2602823e-b519-4ecd-80f5-c8c60472145b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:59.025721 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.025677    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2602823e-b519-4ecd-80f5-c8c60472145b-metrics-server-audit-profiles\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:59.025721 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.025711    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d87lp\" (UniqueName: \"kubernetes.io/projected/2602823e-b519-4ecd-80f5-c8c60472145b-kube-api-access-d87lp\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:59.026187 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.025772    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2602823e-b519-4ecd-80f5-c8c60472145b-secret-metrics-server-tls\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:59.026510 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.026483    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2602823e-b519-4ecd-80f5-c8c60472145b-audit-log\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:59.027103 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.027084    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2602823e-b519-4ecd-80f5-c8c60472145b-metrics-server-audit-profiles\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:59.027215 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.027139    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2602823e-b519-4ecd-80f5-c8c60472145b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:59.033880 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.033857    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2602823e-b519-4ecd-80f5-c8c60472145b-secret-metrics-server-tls\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:59.036484 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.036462    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2602823e-b519-4ecd-80f5-c8c60472145b-secret-metrics-server-client-certs\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:59.039866 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.039811    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2602823e-b519-4ecd-80f5-c8c60472145b-client-ca-bundle\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:59.048277 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.048127    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d87lp\" (UniqueName: \"kubernetes.io/projected/2602823e-b519-4ecd-80f5-c8c60472145b-kube-api-access-d87lp\") pod \"metrics-server-6fbd47b8fd-xdfgg\" (UID: \"2602823e-b519-4ecd-80f5-c8c60472145b\") " pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:59.104397 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.104372    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-r4jl7"]
Oct 30 00:41:59.106937 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.106914    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:41:59.109736 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.109454    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r4jl7"
Oct 30 00:41:59.138525 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.131615    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-r4jl7"]
Oct 30 00:41:59.138525 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.131145    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 00:41:59.138525 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.135568    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vc2d4\""
Oct 30 00:41:59.216762 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.214613    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qs6rx"
Oct 30 00:41:59.228432 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.228105    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdn4s\" (UniqueName: \"kubernetes.io/projected/b3d1f8e1-ca3d-4276-bf20-e499ab93eeee-kube-api-access-pdn4s\") pod \"monitoring-plugin-77f4f76594-r4jl7\" (UID: \"b3d1f8e1-ca3d-4276-bf20-e499ab93eeee\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r4jl7"
Oct 30 00:41:59.228432 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.228184    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b3d1f8e1-ca3d-4276-bf20-e499ab93eeee-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-r4jl7\" (UID: \"b3d1f8e1-ca3d-4276-bf20-e499ab93eeee\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r4jl7"
Oct 30 00:41:59.330400 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.329742    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdn4s\" (UniqueName: \"kubernetes.io/projected/b3d1f8e1-ca3d-4276-bf20-e499ab93eeee-kube-api-access-pdn4s\") pod \"monitoring-plugin-77f4f76594-r4jl7\" (UID: \"b3d1f8e1-ca3d-4276-bf20-e499ab93eeee\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r4jl7"
Oct 30 00:41:59.330517 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.330468    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b3d1f8e1-ca3d-4276-bf20-e499ab93eeee-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-r4jl7\" (UID: \"b3d1f8e1-ca3d-4276-bf20-e499ab93eeee\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r4jl7"
Oct 30 00:41:59.333115 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.333088    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b3d1f8e1-ca3d-4276-bf20-e499ab93eeee-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-r4jl7\" (UID: \"b3d1f8e1-ca3d-4276-bf20-e499ab93eeee\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r4jl7"
Oct 30 00:41:59.353589 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.353547    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdn4s\" (UniqueName: \"kubernetes.io/projected/b3d1f8e1-ca3d-4276-bf20-e499ab93eeee-kube-api-access-pdn4s\") pod \"monitoring-plugin-77f4f76594-r4jl7\" (UID: \"b3d1f8e1-ca3d-4276-bf20-e499ab93eeee\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r4jl7"
Oct 30 00:41:59.432941 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.432910    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"]
Oct 30 00:41:59.439333 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:59.439274    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2602823e_b519_4ecd_80f5_c8c60472145b.slice/crio-6f6b1507315c2d7131134d16ed8fd8fae9863c3d83771bfe0e99309a1a4547a3 WatchSource:0}: Error finding container 6f6b1507315c2d7131134d16ed8fd8fae9863c3d83771bfe0e99309a1a4547a3: Status 404 returned error can't find the container with id 6f6b1507315c2d7131134d16ed8fd8fae9863c3d83771bfe0e99309a1a4547a3
Oct 30 00:41:59.442664 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.442644    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r4jl7"
Oct 30 00:41:59.603740 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.603711    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg" event={"ID":"2602823e-b519-4ecd-80f5-c8c60472145b","Type":"ContainerStarted","Data":"6f6b1507315c2d7131134d16ed8fd8fae9863c3d83771bfe0e99309a1a4547a3"}
Oct 30 00:41:59.607700 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.607675    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4591819-3c83-4361-bbe4-977b5f4fd3c3","Type":"ContainerStarted","Data":"03bf3f9212a80e7e14376bdd274e0a90c87954d3fd05d581c725dcd06dfb771b"}
Oct 30 00:41:59.607792 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.607757    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4591819-3c83-4361-bbe4-977b5f4fd3c3","Type":"ContainerStarted","Data":"a40dfe4717b9cba42b6d8dad0b6aa0292fb28aa0d36abcbc12cfe90b59c9cdb1"}
Oct 30 00:41:59.607834 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.607799    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4591819-3c83-4361-bbe4-977b5f4fd3c3","Type":"ContainerStarted","Data":"febae4bd919ef1640ea62495d7c1d75ac30c4f85c088fda505e654b450c0a329"}
Oct 30 00:41:59.607879 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.607839    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4591819-3c83-4361-bbe4-977b5f4fd3c3","Type":"ContainerStarted","Data":"a99d2d345b7390a99e8cbe6517abf5e115618a17c26be1d98f9a6f5648d46a51"}
Oct 30 00:41:59.607930 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.607876    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4591819-3c83-4361-bbe4-977b5f4fd3c3","Type":"ContainerStarted","Data":"3faa7e31a4a92eaabb5debc15db55786a5fc6f816c6085099c5f94f15a2f6921"}
Oct 30 00:41:59.761736 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:41:59.761708    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-r4jl7"]
Oct 30 00:41:59.769929 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:41:59.769898    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d1f8e1_ca3d_4276_bf20_e499ab93eeee.slice/crio-84b85726a33be4db3e8e2d451f00190a14465653e301ba90fcf7954d8b7d30b8 WatchSource:0}: Error finding container 84b85726a33be4db3e8e2d451f00190a14465653e301ba90fcf7954d8b7d30b8: Status 404 returned error can't find the container with id 84b85726a33be4db3e8e2d451f00190a14465653e301ba90fcf7954d8b7d30b8
Oct 30 00:42:00.621043 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:00.621007    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4591819-3c83-4361-bbe4-977b5f4fd3c3","Type":"ContainerStarted","Data":"b2b5db804ae359a5953586c649c99e90d7f1aaf93225d4217f8606f2110b3618"}
Oct 30 00:42:00.625869 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:00.625825    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw" event={"ID":"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0","Type":"ContainerStarted","Data":"e4b90ff71e7038605f67552c0801823178ae215ddb0e7def125ac63da990bd88"}
Oct 30 00:42:00.626009 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:00.625901    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw" event={"ID":"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0","Type":"ContainerStarted","Data":"06cab795f4981beb2d18a7ae9cb4c1c69534776ae8f9e2f47a6cc9690b9d23d1"}
Oct 30 00:42:00.626009 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:00.625942    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw" event={"ID":"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0","Type":"ContainerStarted","Data":"9da07f505c95208bee66c3a4fca4d82ec9a5d4ef938ab8d56aebeced591dc7f6"}
Oct 30 00:42:00.626009 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:00.625979    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw" event={"ID":"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0","Type":"ContainerStarted","Data":"fe0287758acbb96d374e39b176cff3222d1768701eb500039c11eb5ca07e319a"}
Oct 30 00:42:00.626109 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:00.626019    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw" event={"ID":"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0","Type":"ContainerStarted","Data":"bfd596d4df38062b963c81a5908e1a904d2e695c1191e27d6a261c5470e93424"}
Oct 30 00:42:00.626109 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:00.626054    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw" event={"ID":"ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0","Type":"ContainerStarted","Data":"36569891f5c3af2907172e709ea511a106ef212f7fd7783a8288ae3861ad5dc1"}
Oct 30 00:42:00.626996 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:00.626910    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:42:00.628393 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:00.628371    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r4jl7" event={"ID":"b3d1f8e1-ca3d-4276-bf20-e499ab93eeee","Type":"ContainerStarted","Data":"84b85726a33be4db3e8e2d451f00190a14465653e301ba90fcf7954d8b7d30b8"}
Oct 30 00:42:00.682692 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:00.682636    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.39717383 podStartE2EDuration="5.682616788s" totalImagesPullingTime="3.270294303s" totalInitContainerRuntime="15.148655ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:00.682018684 +0000 UTC m=+29.431673996" watchObservedRunningTime="2026-10-30 00:42:00.682616788 +0000 UTC m=+29.432272098"
Oct 30 00:42:00.732103 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:00.731633    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw" podStartSLOduration=1.8229122850000001 podStartE2EDuration="4.731618679s" totalImagesPullingTime="2.908706394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:00.729770016 +0000 UTC m=+29.479425363" watchObservedRunningTime="2026-10-30 00:42:00.731618679 +0000 UTC m=+29.481273989"
Oct 30 00:42:01.407027 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.406994    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:42:01.421569 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.421533    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.491945 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.491914    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:42:01.497314 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.497281    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 00:42:01.497774 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.497757    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 00:42:01.497918 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.497906    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 00:42:01.498081 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.498067    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 00:42:01.500348 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.500323    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 00:42:01.500493 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.500469    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 00:42:01.500556 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.500549    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f7pl9\""
Oct 30 00:42:01.500611 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.500599    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 00:42:01.500669 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.500649    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5kuqvvfnaq0up\""
Oct 30 00:42:01.500737 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.500716    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 00:42:01.500815 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.500764    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 00:42:01.534874 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.534840    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 00:42:01.535036 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.535011    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 00:42:01.535144 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.535122    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 00:42:01.545866 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.545823    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 00:42:01.546021 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.545951    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 00:42:01.555620 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.555594    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.555757 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.555645    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.555757 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.555725    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.555868 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.555756    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.555868 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.555787    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.555992 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.555967    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.556044 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.556014    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.556098 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.556054    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.556098 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.556077    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-web-config\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.556222 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.556102    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnxdb\" (UniqueName: \"kubernetes.io/projected/8f66871c-bddb-4f93-8574-8706f525d49b-kube-api-access-jnxdb\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.556222 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.556123    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.556222 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.556181    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.556371 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.556243    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f66871c-bddb-4f93-8574-8706f525d49b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.556371 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.556269    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.556371 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.556286    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f66871c-bddb-4f93-8574-8706f525d49b-config-out\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.556371 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.556301    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.556535 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.556377    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-config\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.556535 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.556436    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.556535 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.556466    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.556535 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.556487    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.657487 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.657421    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.657487 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.657451    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.657487 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.657473    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f66871c-bddb-4f93-8574-8706f525d49b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.658059 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.657497    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.658059 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.657527    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f66871c-bddb-4f93-8574-8706f525d49b-config-out\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.658059 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.657555    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.658059 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.657579    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-config\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.658698 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.658672    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.658809 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.658739    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.658809 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.658770    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.658809 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.658797    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.658946 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.658840    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.658946 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.658902    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.658946 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.658928    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.659094 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.658952    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.659094 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.659022    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.659094 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.659069    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.659292 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.659101    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.659292 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.659133    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.659292 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.659184    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-web-config\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.659292 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.659212    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnxdb\" (UniqueName: \"kubernetes.io/projected/8f66871c-bddb-4f93-8574-8706f525d49b-kube-api-access-jnxdb\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.662262 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.659908    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f66871c-bddb-4f93-8574-8706f525d49b-config-out\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.662262 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.659930    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.662262 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.660390    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f66871c-bddb-4f93-8574-8706f525d49b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.662262 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.660452    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.662262 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.660507    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-config\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.662262 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.661042    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.662262 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.661420    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.662262 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.661836    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.662262 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.661935    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.662262 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.662182    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.662817 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.662459    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.663143 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.663125    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.663523 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.663504    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.663892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.663868    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.664198 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.664175    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.664645 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.664622    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.664757 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.664739    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-web-config\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.665084 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.665063    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.681285 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.681261    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnxdb\" (UniqueName: \"kubernetes.io/projected/8f66871c-bddb-4f93-8574-8706f525d49b-kube-api-access-jnxdb\") pod \"prometheus-k8s-0\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:01.856363 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:01.856332    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:02.242518 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:02.242488    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:42:02.245048 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:42:02.245011    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f66871c_bddb_4f93_8574_8706f525d49b.slice/crio-6650362f652c5f47bca73024c6aa1c33e07e2a66cb91c72df2d2a017232c1b0c WatchSource:0}: Error finding container 6650362f652c5f47bca73024c6aa1c33e07e2a66cb91c72df2d2a017232c1b0c: Status 404 returned error can't find the container with id 6650362f652c5f47bca73024c6aa1c33e07e2a66cb91c72df2d2a017232c1b0c
Oct 30 00:42:02.444961 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:02.444189    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:42:02.444961 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:02.444274    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:42:02.449728 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:02.449698    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:42:02.639678 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:02.639634    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg" event={"ID":"2602823e-b519-4ecd-80f5-c8c60472145b","Type":"ContainerStarted","Data":"7a17052e8ea7397b26ec7adfc118a33cd3a6f79d94cf61913e0c01cb660e5538"}
Oct 30 00:42:02.641669 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:02.641639    4425 generic.go:356] "Generic (PLEG): container finished" podID="8f66871c-bddb-4f93-8574-8706f525d49b" containerID="5b61de753851f99d5cc5565c84e0ad06e139e2e98797b5639396a55bbb9c877e" exitCode=0
Oct 30 00:42:02.641790 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:02.641752    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f66871c-bddb-4f93-8574-8706f525d49b","Type":"ContainerDied","Data":"5b61de753851f99d5cc5565c84e0ad06e139e2e98797b5639396a55bbb9c877e"}
Oct 30 00:42:02.641867 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:02.641819    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f66871c-bddb-4f93-8574-8706f525d49b","Type":"ContainerStarted","Data":"6650362f652c5f47bca73024c6aa1c33e07e2a66cb91c72df2d2a017232c1b0c"}
Oct 30 00:42:02.644097 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:02.643974    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r4jl7" event={"ID":"b3d1f8e1-ca3d-4276-bf20-e499ab93eeee","Type":"ContainerStarted","Data":"8be6bfb05d8b35ab0947e1a958bf69d41a70faaaca7db3ecf00ac4831cf5e2bf"}
Oct 30 00:42:02.644502 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:02.644477    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r4jl7"
Oct 30 00:42:02.649236 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:02.649218    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:42:02.650136 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:02.650116    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r4jl7"
Oct 30 00:42:02.677883 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:02.677833    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg" podStartSLOduration=2.272466218 podStartE2EDuration="4.67781989s" totalImagesPullingTime="2.405353672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:02.677300666 +0000 UTC m=+31.426955978" watchObservedRunningTime="2026-10-30 00:42:02.67781989 +0000 UTC m=+31.427475199"
Oct 30 00:42:02.712812 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:02.712760    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r4jl7" podStartSLOduration=1.634331975 podStartE2EDuration="3.712742927s" totalImagesPullingTime="2.078410952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:02.711958478 +0000 UTC m=+31.461613786" watchObservedRunningTime="2026-10-30 00:42:02.712742927 +0000 UTC m=+31.462398235"
Oct 30 00:42:05.661858 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:05.661816    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-fptfc" event={"ID":"75d61c2d-5a46-45a7-b9d7-fa072c685bde","Type":"ContainerStarted","Data":"4f1c634d32f4a4a5d9312108b0dcb00d90175b21162c19cbb3858da6737daa6e"}
Oct 30 00:42:05.663872 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:05.663806    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-fptfc"
Oct 30 00:42:05.665595 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:05.665570    4425 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-fptfc container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.10:8080/\": dial tcp 10.133.0.10:8080: connect: connection refused" start-of-body=
Oct 30 00:42:05.665695 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:05.665633    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-fptfc" podUID="75d61c2d-5a46-45a7-b9d7-fa072c685bde" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.10:8080/\": dial tcp 10.133.0.10:8080: connect: connection refused"
Oct 30 00:42:05.701020 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:05.700931    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-fptfc" podStartSLOduration=1.266677245 podStartE2EDuration="19.700916884s" totalImagesPullingTime="18.434239639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:05.700596027 +0000 UTC m=+34.450251342" watchObservedRunningTime="2026-10-30 00:42:05.700916884 +0000 UTC m=+34.450572184"
Oct 30 00:42:05.813241 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:05.813209    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-79ffc79964-cjk8r"]
Oct 30 00:42:05.828192 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:05.827727    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:05.850277 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:05.849133    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 00:42:05.867188 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:05.867134    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79ffc79964-cjk8r"]
Oct 30 00:42:05.900608 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:05.900575    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-oauth-serving-cert\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:05.900955 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:05.900656    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-service-ca\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:05.900955 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:05.900678    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-config\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:05.900955 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:05.900695    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-trusted-ca-bundle\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:05.900955 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:05.900728    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-oauth-config\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:05.900955 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:05.900750    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-tmp\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:05.900955 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:05.900776    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhcwh\" (UniqueName: \"kubernetes.io/projected/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-kube-api-access-rhcwh\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:05.900955 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:05.900802    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-serving-cert\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.002763 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.001321    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhcwh\" (UniqueName: \"kubernetes.io/projected/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-kube-api-access-rhcwh\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.002763 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.001389    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-serving-cert\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.002763 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.001463    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-oauth-serving-cert\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.002763 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.001510    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-service-ca\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.002763 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.001547    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-config\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.002763 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.001575    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-trusted-ca-bundle\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.002763 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.001621    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-oauth-config\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.002763 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.001637    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-tmp\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.002763 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.002081    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-tmp\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.004712 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.004664    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-oauth-serving-cert\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.004896 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.004847    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-service-ca\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.005017 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.004935    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-config\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.005360 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.005336    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-trusted-ca-bundle\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.010923 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.010897    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-oauth-config\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.012796 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.012775    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-serving-cert\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.023367 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.023343    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhcwh\" (UniqueName: \"kubernetes.io/projected/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-kube-api-access-rhcwh\") pod \"console-79ffc79964-cjk8r\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") " pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.170057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.170018    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:06.597596 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.591039    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79ffc79964-cjk8r"]
Oct 30 00:42:06.615736 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:42:06.614311    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ecabd9e_2157_4e6b_a21b_c93a7771cc46.slice/crio-b64d67bfc4aaa3e170add7c6b9a49c89bae4986ffca3ab43f4fc3c5d30d96c8d WatchSource:0}: Error finding container b64d67bfc4aaa3e170add7c6b9a49c89bae4986ffca3ab43f4fc3c5d30d96c8d: Status 404 returned error can't find the container with id b64d67bfc4aaa3e170add7c6b9a49c89bae4986ffca3ab43f4fc3c5d30d96c8d
Oct 30 00:42:06.672880 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.672849    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f66871c-bddb-4f93-8574-8706f525d49b","Type":"ContainerStarted","Data":"88dd5dbaf077c5e0cd902508bcf5fec34b53f567b262f48bc2a800b1818b2256"}
Oct 30 00:42:06.673190 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.672936    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f66871c-bddb-4f93-8574-8706f525d49b","Type":"ContainerStarted","Data":"8fa118ebffda26500fc23e49f0c183047642405072855954269fa368f69e508f"}
Oct 30 00:42:06.676559 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.676535    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79ffc79964-cjk8r" event={"ID":"3ecabd9e-2157-4e6b-a21b-c93a7771cc46","Type":"ContainerStarted","Data":"b64d67bfc4aaa3e170add7c6b9a49c89bae4986ffca3ab43f4fc3c5d30d96c8d"}
Oct 30 00:42:06.678716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:06.678616    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-fptfc"
Oct 30 00:42:07.558604 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:42:07.558566    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:42:07.656070 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:07.656035    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7bf64bb4d4-jhfpw"
Oct 30 00:42:07.681297 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:07.681258    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79ffc79964-cjk8r" event={"ID":"3ecabd9e-2157-4e6b-a21b-c93a7771cc46","Type":"ContainerStarted","Data":"ad52f963cb2b50eb1ed90304083daff2240cc2b42c59ee26b0278cbdca7a8baa"}
Oct 30 00:42:07.685711 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:07.685679    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f66871c-bddb-4f93-8574-8706f525d49b","Type":"ContainerStarted","Data":"00d20d1c54d9220de4df2104bdb3b2832f84d74460136e208216318ed801996b"}
Oct 30 00:42:07.685842 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:07.685740    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f66871c-bddb-4f93-8574-8706f525d49b","Type":"ContainerStarted","Data":"0120593312c9127d2a3c3223f90866e0bef965e1c98eb217970a01e69003b54e"}
Oct 30 00:42:07.685842 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:07.685773    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f66871c-bddb-4f93-8574-8706f525d49b","Type":"ContainerStarted","Data":"8f3f8cde70efd320c65b06781a9bf82d8d598bc66f0b0467ecf66283d784118e"}
Oct 30 00:42:07.685842 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:07.685805    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f66871c-bddb-4f93-8574-8706f525d49b","Type":"ContainerStarted","Data":"f3006f98457d180dbfd27731a8ee123d7fa4b4c6b030694c7038da9bc69f5380"}
Oct 30 00:42:07.754608 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:07.754559    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.077045929 podStartE2EDuration="6.75454552s" totalImagesPullingTime="3.653036045s" totalInitContainerRuntime="24.463546ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:07.753805543 +0000 UTC m=+36.503460858" watchObservedRunningTime="2026-10-30 00:42:07.75454552 +0000 UTC m=+36.504200830"
Oct 30 00:42:07.792466 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:07.792413    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-79ffc79964-cjk8r" podStartSLOduration=2.7923992 podStartE2EDuration="2.7923992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:07.791873015 +0000 UTC m=+36.541528336" watchObservedRunningTime="2026-10-30 00:42:07.7923992 +0000 UTC m=+36.542054508"
Oct 30 00:42:11.856610 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:11.856569    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:16.171038 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:16.170956    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:16.171612 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:16.171047    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:16.175365 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:16.175344    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:16.731089 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:16.731061    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:42:16.802992 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:16.802961    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85cd688b89-8tksl"]
Oct 30 00:42:19.108195 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:19.108151    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:42:19.108564 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:19.108224    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:42:37.553212 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:42:37.553171    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:42:39.112625 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:39.112594    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:42:39.116361 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:39.116341    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6fbd47b8fd-xdfgg"
Oct 30 00:42:41.826908 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:41.826853    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-85cd688b89-8tksl" podUID="d1d10b60-20ba-442d-815f-01653be20945" containerName="console" containerID="cri-o://cbd27d488d92814ca10860d928d67bf58d7a2d06c01dc181c90a00b3116d392f" gracePeriod=15
Oct 30 00:42:42.120886 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.120862    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:42:42.220208 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.220179    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1d10b60-20ba-442d-815f-01653be20945-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1d10b60-20ba-442d-815f-01653be20945-tmp\") pod \"d1d10b60-20ba-442d-815f-01653be20945\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") "
Oct 30 00:42:42.220349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.220224    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-oauth-serving-cert\") pod \"d1d10b60-20ba-442d-815f-01653be20945\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") "
Oct 30 00:42:42.220349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.220276    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1d10b60-20ba-442d-815f-01653be20945-kube-api-access-5plcv\" (UniqueName: \"kubernetes.io/projected/d1d10b60-20ba-442d-815f-01653be20945-kube-api-access-5plcv\") pod \"d1d10b60-20ba-442d-815f-01653be20945\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") "
Oct 30 00:42:42.220349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.220297    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-service-ca\") pod \"d1d10b60-20ba-442d-815f-01653be20945\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") "
Oct 30 00:42:42.220349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.220326    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1d10b60-20ba-442d-815f-01653be20945-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1d10b60-20ba-442d-815f-01653be20945-console-serving-cert\") pod \"d1d10b60-20ba-442d-815f-01653be20945\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") "
Oct 30 00:42:42.220349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.220349    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-console-config\" (UniqueName: \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-console-config\") pod \"d1d10b60-20ba-442d-815f-01653be20945\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") "
Oct 30 00:42:42.220582 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.220411    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1d10b60-20ba-442d-815f-01653be20945-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1d10b60-20ba-442d-815f-01653be20945-console-oauth-config\") pod \"d1d10b60-20ba-442d-815f-01653be20945\" (UID: \"d1d10b60-20ba-442d-815f-01653be20945\") "
Oct 30 00:42:42.220582 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.220455    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d10b60-20ba-442d-815f-01653be20945-tmp" pod "d1d10b60-20ba-442d-815f-01653be20945" (UID: "d1d10b60-20ba-442d-815f-01653be20945"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:42:42.220654 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.220633    4425 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1d10b60-20ba-442d-815f-01653be20945-tmp\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:42.220688 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.220652    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-oauth-serving-cert" pod "d1d10b60-20ba-442d-815f-01653be20945" (UID: "d1d10b60-20ba-442d-815f-01653be20945"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:42:42.220983 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.220936    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-service-ca" pod "d1d10b60-20ba-442d-815f-01653be20945" (UID: "d1d10b60-20ba-442d-815f-01653be20945"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:42:42.221099 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.220981    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-console-config" pod "d1d10b60-20ba-442d-815f-01653be20945" (UID: "d1d10b60-20ba-442d-815f-01653be20945"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:42:42.222519 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.222498    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d10b60-20ba-442d-815f-01653be20945-kube-api-access-5plcv" pod "d1d10b60-20ba-442d-815f-01653be20945" (UID: "d1d10b60-20ba-442d-815f-01653be20945"). InnerVolumeSpecName "kube-api-access-5plcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:42:42.222904 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.222883    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1d10b60-20ba-442d-815f-01653be20945-console-oauth-config" pod "d1d10b60-20ba-442d-815f-01653be20945" (UID: "d1d10b60-20ba-442d-815f-01653be20945"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:42:42.222994 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.222911    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1d10b60-20ba-442d-815f-01653be20945-console-serving-cert" pod "d1d10b60-20ba-442d-815f-01653be20945" (UID: "d1d10b60-20ba-442d-815f-01653be20945"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:42:42.321512 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.321485    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5plcv\" (UniqueName: \"kubernetes.io/projected/d1d10b60-20ba-442d-815f-01653be20945-kube-api-access-5plcv\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:42.321512 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.321508    4425 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:42.321646 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.321522    4425 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1d10b60-20ba-442d-815f-01653be20945-console-serving-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:42.321646 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.321534    4425 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-console-config\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:42.321646 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.321547    4425 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1d10b60-20ba-442d-815f-01653be20945-console-oauth-config\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:42.321646 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.321559    4425 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1d10b60-20ba-442d-815f-01653be20945-oauth-serving-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:42.827418 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.827380    4425 generic.go:356] "Generic (PLEG): container finished" podID="d1d10b60-20ba-442d-815f-01653be20945" containerID="cbd27d488d92814ca10860d928d67bf58d7a2d06c01dc181c90a00b3116d392f" exitCode=0
Oct 30 00:42:42.827799 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.827428    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85cd688b89-8tksl"
Oct 30 00:42:42.827799 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.827471    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85cd688b89-8tksl" event={"ID":"d1d10b60-20ba-442d-815f-01653be20945","Type":"ContainerDied","Data":"cbd27d488d92814ca10860d928d67bf58d7a2d06c01dc181c90a00b3116d392f"}
Oct 30 00:42:42.827799 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.827520    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85cd688b89-8tksl" event={"ID":"d1d10b60-20ba-442d-815f-01653be20945","Type":"ContainerDied","Data":"236a91cf73293893662e96421b503d2ffd6e19a8a1db92af788ec71613185073"}
Oct 30 00:42:42.827799 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.827540    4425 scope.go:122] "RemoveContainer" containerID="cbd27d488d92814ca10860d928d67bf58d7a2d06c01dc181c90a00b3116d392f"
Oct 30 00:42:42.841914 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.841892    4425 scope.go:122] "RemoveContainer" containerID="cbd27d488d92814ca10860d928d67bf58d7a2d06c01dc181c90a00b3116d392f"
Oct 30 00:42:42.842204 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:42:42.842180    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd27d488d92814ca10860d928d67bf58d7a2d06c01dc181c90a00b3116d392f\": container with ID starting with cbd27d488d92814ca10860d928d67bf58d7a2d06c01dc181c90a00b3116d392f not found: ID does not exist" containerID="cbd27d488d92814ca10860d928d67bf58d7a2d06c01dc181c90a00b3116d392f"
Oct 30 00:42:42.842259 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.842215    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd27d488d92814ca10860d928d67bf58d7a2d06c01dc181c90a00b3116d392f"} err="failed to get container status \"cbd27d488d92814ca10860d928d67bf58d7a2d06c01dc181c90a00b3116d392f\": rpc error: code = NotFound desc = could not find container \"cbd27d488d92814ca10860d928d67bf58d7a2d06c01dc181c90a00b3116d392f\": container with ID starting with cbd27d488d92814ca10860d928d67bf58d7a2d06c01dc181c90a00b3116d392f not found: ID does not exist"
Oct 30 00:42:42.870860 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.870839    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85cd688b89-8tksl"]
Oct 30 00:42:42.880539 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:42.880518    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85cd688b89-8tksl"]
Oct 30 00:42:43.419717 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:43.419690    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1d10b60-20ba-442d-815f-01653be20945" path="/var/lib/kubelet/pods/d1d10b60-20ba-442d-815f-01653be20945/volumes"
Oct 30 00:42:46.576788 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:46.576761    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:46.624051 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:46.624027    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:46.858054 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:46.857989    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:42:56.262087 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:56.262056    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:42:56.262582 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:56.262330    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="alertmanager" containerID="cri-o://3faa7e31a4a92eaabb5debc15db55786a5fc6f816c6085099c5f94f15a2f6921" gracePeriod=120
Oct 30 00:42:56.262582 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:56.262403    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="kube-rbac-proxy-metric" containerID="cri-o://03bf3f9212a80e7e14376bdd274e0a90c87954d3fd05d581c725dcd06dfb771b" gracePeriod=120
Oct 30 00:42:56.262582 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:56.262435    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="kube-rbac-proxy-web" containerID="cri-o://febae4bd919ef1640ea62495d7c1d75ac30c4f85c088fda505e654b450c0a329" gracePeriod=120
Oct 30 00:42:56.262582 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:56.262487    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="config-reloader" containerID="cri-o://a99d2d345b7390a99e8cbe6517abf5e115618a17c26be1d98f9a6f5648d46a51" gracePeriod=120
Oct 30 00:42:56.262582 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:56.262515    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="kube-rbac-proxy" containerID="cri-o://a40dfe4717b9cba42b6d8dad0b6aa0292fb28aa0d36abcbc12cfe90b59c9cdb1" gracePeriod=120
Oct 30 00:42:56.262582 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:56.262482    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="prom-label-proxy" containerID="cri-o://b2b5db804ae359a5953586c649c99e90d7f1aaf93225d4217f8606f2110b3618" gracePeriod=120
Oct 30 00:42:56.890784 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:56.890756    4425 generic.go:356] "Generic (PLEG): container finished" podID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerID="b2b5db804ae359a5953586c649c99e90d7f1aaf93225d4217f8606f2110b3618" exitCode=0
Oct 30 00:42:56.890784 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:56.890779    4425 generic.go:356] "Generic (PLEG): container finished" podID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerID="03bf3f9212a80e7e14376bdd274e0a90c87954d3fd05d581c725dcd06dfb771b" exitCode=0
Oct 30 00:42:56.890784 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:56.890788    4425 generic.go:356] "Generic (PLEG): container finished" podID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerID="a40dfe4717b9cba42b6d8dad0b6aa0292fb28aa0d36abcbc12cfe90b59c9cdb1" exitCode=0
Oct 30 00:42:56.891002 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:56.890796    4425 generic.go:356] "Generic (PLEG): container finished" podID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerID="a99d2d345b7390a99e8cbe6517abf5e115618a17c26be1d98f9a6f5648d46a51" exitCode=0
Oct 30 00:42:56.891002 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:56.890804    4425 generic.go:356] "Generic (PLEG): container finished" podID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerID="3faa7e31a4a92eaabb5debc15db55786a5fc6f816c6085099c5f94f15a2f6921" exitCode=0
Oct 30 00:42:56.891002 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:56.890824    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4591819-3c83-4361-bbe4-977b5f4fd3c3","Type":"ContainerDied","Data":"b2b5db804ae359a5953586c649c99e90d7f1aaf93225d4217f8606f2110b3618"}
Oct 30 00:42:56.891002 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:56.890885    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4591819-3c83-4361-bbe4-977b5f4fd3c3","Type":"ContainerDied","Data":"03bf3f9212a80e7e14376bdd274e0a90c87954d3fd05d581c725dcd06dfb771b"}
Oct 30 00:42:56.891002 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:56.890910    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4591819-3c83-4361-bbe4-977b5f4fd3c3","Type":"ContainerDied","Data":"a40dfe4717b9cba42b6d8dad0b6aa0292fb28aa0d36abcbc12cfe90b59c9cdb1"}
Oct 30 00:42:56.891002 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:56.890933    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4591819-3c83-4361-bbe4-977b5f4fd3c3","Type":"ContainerDied","Data":"a99d2d345b7390a99e8cbe6517abf5e115618a17c26be1d98f9a6f5648d46a51"}
Oct 30 00:42:56.891002 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:56.890956    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4591819-3c83-4361-bbe4-977b5f4fd3c3","Type":"ContainerDied","Data":"3faa7e31a4a92eaabb5debc15db55786a5fc6f816c6085099c5f94f15a2f6921"}
Oct 30 00:42:57.503219 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.503195    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:42:57.572395 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.572340    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4591819-3c83-4361-bbe4-977b5f4fd3c3-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4591819-3c83-4361-bbe4-977b5f4fd3c3-metrics-client-ca\") pod \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") "
Oct 30 00:42:57.572395 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.572373    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4591819-3c83-4361-bbe4-977b5f4fd3c3-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4591819-3c83-4361-bbe4-977b5f4fd3c3-config-out\") pod \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") "
Oct 30 00:42:57.572570 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.572399    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4591819-3c83-4361-bbe4-977b5f4fd3c3-tls-assets\" (UniqueName: \"kubernetes.io/projected/b4591819-3c83-4361-bbe4-977b5f4fd3c3-tls-assets\") pod \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") "
Oct 30 00:42:57.572570 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.572421    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4591819-3c83-4361-bbe4-977b5f4fd3c3-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b4591819-3c83-4361-bbe4-977b5f4fd3c3-alertmanager-main-db\") pod \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") "
Oct 30 00:42:57.572570 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.572446    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy\") pod \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") "
Oct 30 00:42:57.572570 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.572477    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4591819-3c83-4361-bbe4-977b5f4fd3c3-kube-api-access-pglgd\" (UniqueName: \"kubernetes.io/projected/b4591819-3c83-4361-bbe4-977b5f4fd3c3-kube-api-access-pglgd\") pod \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") "
Oct 30 00:42:57.572570 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.572495    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-main-tls\") pod \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") "
Oct 30 00:42:57.572570 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.572516    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-cluster-tls-config\") pod \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") "
Oct 30 00:42:57.572570 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.572545    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4591819-3c83-4361-bbe4-977b5f4fd3c3-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4591819-3c83-4361-bbe4-977b5f4fd3c3-alertmanager-trusted-ca-bundle\") pod \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") "
Oct 30 00:42:57.572902 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.572579    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-web-config\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-web-config\") pod \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") "
Oct 30 00:42:57.572902 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.572610    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") "
Oct 30 00:42:57.572902 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.572665    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy-web\") pod \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") "
Oct 30 00:42:57.572902 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.572697    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-config-volume\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-config-volume\") pod \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\" (UID: \"b4591819-3c83-4361-bbe4-977b5f4fd3c3\") "
Oct 30 00:42:57.573340 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.573255    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4591819-3c83-4361-bbe4-977b5f4fd3c3-alertmanager-main-db" pod "b4591819-3c83-4361-bbe4-977b5f4fd3c3" (UID: "b4591819-3c83-4361-bbe4-977b5f4fd3c3"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:42:57.573462 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.573438    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4591819-3c83-4361-bbe4-977b5f4fd3c3-metrics-client-ca" pod "b4591819-3c83-4361-bbe4-977b5f4fd3c3" (UID: "b4591819-3c83-4361-bbe4-977b5f4fd3c3"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:42:57.574322 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.574292    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4591819-3c83-4361-bbe4-977b5f4fd3c3-alertmanager-trusted-ca-bundle" pod "b4591819-3c83-4361-bbe4-977b5f4fd3c3" (UID: "b4591819-3c83-4361-bbe4-977b5f4fd3c3"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:42:57.575888 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.575862    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy-metric" pod "b4591819-3c83-4361-bbe4-977b5f4fd3c3" (UID: "b4591819-3c83-4361-bbe4-977b5f4fd3c3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:42:57.576046 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.575997    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy" pod "b4591819-3c83-4361-bbe4-977b5f4fd3c3" (UID: "b4591819-3c83-4361-bbe4-977b5f4fd3c3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:42:57.576046 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.576026    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4591819-3c83-4361-bbe4-977b5f4fd3c3-config-out" pod "b4591819-3c83-4361-bbe4-977b5f4fd3c3" (UID: "b4591819-3c83-4361-bbe4-977b5f4fd3c3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:42:57.576248 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.575999    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4591819-3c83-4361-bbe4-977b5f4fd3c3-tls-assets" pod "b4591819-3c83-4361-bbe4-977b5f4fd3c3" (UID: "b4591819-3c83-4361-bbe4-977b5f4fd3c3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:42:57.576369 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.576344    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4591819-3c83-4361-bbe4-977b5f4fd3c3-kube-api-access-pglgd" pod "b4591819-3c83-4361-bbe4-977b5f4fd3c3" (UID: "b4591819-3c83-4361-bbe4-977b5f4fd3c3"). InnerVolumeSpecName "kube-api-access-pglgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:42:57.576431 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.576377    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-config-volume" pod "b4591819-3c83-4361-bbe4-977b5f4fd3c3" (UID: "b4591819-3c83-4361-bbe4-977b5f4fd3c3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:42:57.577407 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.577383    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-main-tls" pod "b4591819-3c83-4361-bbe4-977b5f4fd3c3" (UID: "b4591819-3c83-4361-bbe4-977b5f4fd3c3"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:42:57.577617 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.577598    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy-web" pod "b4591819-3c83-4361-bbe4-977b5f4fd3c3" (UID: "b4591819-3c83-4361-bbe4-977b5f4fd3c3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:42:57.580788 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.580766    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-cluster-tls-config" pod "b4591819-3c83-4361-bbe4-977b5f4fd3c3" (UID: "b4591819-3c83-4361-bbe4-977b5f4fd3c3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:42:57.588555 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.588526    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-web-config" pod "b4591819-3c83-4361-bbe4-977b5f4fd3c3" (UID: "b4591819-3c83-4361-bbe4-977b5f4fd3c3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:42:57.673776 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.673738    4425 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:57.673776 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.673773    4425 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:57.673776 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.673783    4425 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4591819-3c83-4361-bbe4-977b5f4fd3c3-metrics-client-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:57.673776 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.673792    4425 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4591819-3c83-4361-bbe4-977b5f4fd3c3-config-out\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:57.674011 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.673800    4425 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4591819-3c83-4361-bbe4-977b5f4fd3c3-tls-assets\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:57.674011 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.673811    4425 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b4591819-3c83-4361-bbe4-977b5f4fd3c3-alertmanager-main-db\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:57.674011 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.673819    4425 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:57.674011 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.673828    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pglgd\" (UniqueName: \"kubernetes.io/projected/b4591819-3c83-4361-bbe4-977b5f4fd3c3-kube-api-access-pglgd\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:57.674011 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.673836    4425 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-main-tls\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:57.674011 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.673844    4425 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-cluster-tls-config\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:57.674011 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.673852    4425 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4591819-3c83-4361-bbe4-977b5f4fd3c3-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:57.674011 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.673861    4425 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-web-config\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:57.674011 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.673869    4425 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b4591819-3c83-4361-bbe4-977b5f4fd3c3-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:42:57.897899 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.897817    4425 generic.go:356] "Generic (PLEG): container finished" podID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerID="febae4bd919ef1640ea62495d7c1d75ac30c4f85c088fda505e654b450c0a329" exitCode=0
Oct 30 00:42:57.898048 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.897897    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4591819-3c83-4361-bbe4-977b5f4fd3c3","Type":"ContainerDied","Data":"febae4bd919ef1640ea62495d7c1d75ac30c4f85c088fda505e654b450c0a329"}
Oct 30 00:42:57.898048 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.897917    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:42:57.898048 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.897985    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4591819-3c83-4361-bbe4-977b5f4fd3c3","Type":"ContainerDied","Data":"ab9035eae016d96cf49f6c509c13215331675895ccba6cabc641d6a68cefd58c"}
Oct 30 00:42:57.898048 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.898034    4425 scope.go:122] "RemoveContainer" containerID="b2b5db804ae359a5953586c649c99e90d7f1aaf93225d4217f8606f2110b3618"
Oct 30 00:42:57.905999 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.905897    4425 scope.go:122] "RemoveContainer" containerID="03bf3f9212a80e7e14376bdd274e0a90c87954d3fd05d581c725dcd06dfb771b"
Oct 30 00:42:57.912624 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.912610    4425 scope.go:122] "RemoveContainer" containerID="a40dfe4717b9cba42b6d8dad0b6aa0292fb28aa0d36abcbc12cfe90b59c9cdb1"
Oct 30 00:42:57.919449 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.919428    4425 scope.go:122] "RemoveContainer" containerID="febae4bd919ef1640ea62495d7c1d75ac30c4f85c088fda505e654b450c0a329"
Oct 30 00:42:57.925746 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.925730    4425 scope.go:122] "RemoveContainer" containerID="a99d2d345b7390a99e8cbe6517abf5e115618a17c26be1d98f9a6f5648d46a51"
Oct 30 00:42:57.932409 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.932394    4425 scope.go:122] "RemoveContainer" containerID="3faa7e31a4a92eaabb5debc15db55786a5fc6f816c6085099c5f94f15a2f6921"
Oct 30 00:42:57.939004 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.938987    4425 scope.go:122] "RemoveContainer" containerID="f0be405cd48ea0923fb509b440facaa3fe8cb52081d666d0390800a3a2bbb6dd"
Oct 30 00:42:57.944639 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.944620    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:42:57.945600 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.945586    4425 scope.go:122] "RemoveContainer" containerID="b2b5db804ae359a5953586c649c99e90d7f1aaf93225d4217f8606f2110b3618"
Oct 30 00:42:57.945844 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:42:57.945826    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b5db804ae359a5953586c649c99e90d7f1aaf93225d4217f8606f2110b3618\": container with ID starting with b2b5db804ae359a5953586c649c99e90d7f1aaf93225d4217f8606f2110b3618 not found: ID does not exist" containerID="b2b5db804ae359a5953586c649c99e90d7f1aaf93225d4217f8606f2110b3618"
Oct 30 00:42:57.945895 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.945852    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b5db804ae359a5953586c649c99e90d7f1aaf93225d4217f8606f2110b3618"} err="failed to get container status \"b2b5db804ae359a5953586c649c99e90d7f1aaf93225d4217f8606f2110b3618\": rpc error: code = NotFound desc = could not find container \"b2b5db804ae359a5953586c649c99e90d7f1aaf93225d4217f8606f2110b3618\": container with ID starting with b2b5db804ae359a5953586c649c99e90d7f1aaf93225d4217f8606f2110b3618 not found: ID does not exist"
Oct 30 00:42:57.945895 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.945869    4425 scope.go:122] "RemoveContainer" containerID="03bf3f9212a80e7e14376bdd274e0a90c87954d3fd05d581c725dcd06dfb771b"
Oct 30 00:42:57.946094 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:42:57.946078    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03bf3f9212a80e7e14376bdd274e0a90c87954d3fd05d581c725dcd06dfb771b\": container with ID starting with 03bf3f9212a80e7e14376bdd274e0a90c87954d3fd05d581c725dcd06dfb771b not found: ID does not exist" containerID="03bf3f9212a80e7e14376bdd274e0a90c87954d3fd05d581c725dcd06dfb771b"
Oct 30 00:42:57.946131 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.946098    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03bf3f9212a80e7e14376bdd274e0a90c87954d3fd05d581c725dcd06dfb771b"} err="failed to get container status \"03bf3f9212a80e7e14376bdd274e0a90c87954d3fd05d581c725dcd06dfb771b\": rpc error: code = NotFound desc = could not find container \"03bf3f9212a80e7e14376bdd274e0a90c87954d3fd05d581c725dcd06dfb771b\": container with ID starting with 03bf3f9212a80e7e14376bdd274e0a90c87954d3fd05d581c725dcd06dfb771b not found: ID does not exist"
Oct 30 00:42:57.946131 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.946111    4425 scope.go:122] "RemoveContainer" containerID="a40dfe4717b9cba42b6d8dad0b6aa0292fb28aa0d36abcbc12cfe90b59c9cdb1"
Oct 30 00:42:57.946353 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:42:57.946337    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a40dfe4717b9cba42b6d8dad0b6aa0292fb28aa0d36abcbc12cfe90b59c9cdb1\": container with ID starting with a40dfe4717b9cba42b6d8dad0b6aa0292fb28aa0d36abcbc12cfe90b59c9cdb1 not found: ID does not exist" containerID="a40dfe4717b9cba42b6d8dad0b6aa0292fb28aa0d36abcbc12cfe90b59c9cdb1"
Oct 30 00:42:57.946390 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.946360    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40dfe4717b9cba42b6d8dad0b6aa0292fb28aa0d36abcbc12cfe90b59c9cdb1"} err="failed to get container status \"a40dfe4717b9cba42b6d8dad0b6aa0292fb28aa0d36abcbc12cfe90b59c9cdb1\": rpc error: code = NotFound desc = could not find container \"a40dfe4717b9cba42b6d8dad0b6aa0292fb28aa0d36abcbc12cfe90b59c9cdb1\": container with ID starting with a40dfe4717b9cba42b6d8dad0b6aa0292fb28aa0d36abcbc12cfe90b59c9cdb1 not found: ID does not exist"
Oct 30 00:42:57.946390 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.946374    4425 scope.go:122] "RemoveContainer" containerID="febae4bd919ef1640ea62495d7c1d75ac30c4f85c088fda505e654b450c0a329"
Oct 30 00:42:57.946621 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:42:57.946605    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"febae4bd919ef1640ea62495d7c1d75ac30c4f85c088fda505e654b450c0a329\": container with ID starting with febae4bd919ef1640ea62495d7c1d75ac30c4f85c088fda505e654b450c0a329 not found: ID does not exist" containerID="febae4bd919ef1640ea62495d7c1d75ac30c4f85c088fda505e654b450c0a329"
Oct 30 00:42:57.946677 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.946625    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"febae4bd919ef1640ea62495d7c1d75ac30c4f85c088fda505e654b450c0a329"} err="failed to get container status \"febae4bd919ef1640ea62495d7c1d75ac30c4f85c088fda505e654b450c0a329\": rpc error: code = NotFound desc = could not find container \"febae4bd919ef1640ea62495d7c1d75ac30c4f85c088fda505e654b450c0a329\": container with ID starting with febae4bd919ef1640ea62495d7c1d75ac30c4f85c088fda505e654b450c0a329 not found: ID does not exist"
Oct 30 00:42:57.946677 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.946644    4425 scope.go:122] "RemoveContainer" containerID="a99d2d345b7390a99e8cbe6517abf5e115618a17c26be1d98f9a6f5648d46a51"
Oct 30 00:42:57.946830 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:42:57.946814    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a99d2d345b7390a99e8cbe6517abf5e115618a17c26be1d98f9a6f5648d46a51\": container with ID starting with a99d2d345b7390a99e8cbe6517abf5e115618a17c26be1d98f9a6f5648d46a51 not found: ID does not exist" containerID="a99d2d345b7390a99e8cbe6517abf5e115618a17c26be1d98f9a6f5648d46a51"
Oct 30 00:42:57.946869 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.946836    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99d2d345b7390a99e8cbe6517abf5e115618a17c26be1d98f9a6f5648d46a51"} err="failed to get container status \"a99d2d345b7390a99e8cbe6517abf5e115618a17c26be1d98f9a6f5648d46a51\": rpc error: code = NotFound desc = could not find container \"a99d2d345b7390a99e8cbe6517abf5e115618a17c26be1d98f9a6f5648d46a51\": container with ID starting with a99d2d345b7390a99e8cbe6517abf5e115618a17c26be1d98f9a6f5648d46a51 not found: ID does not exist"
Oct 30 00:42:57.946869 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.946851    4425 scope.go:122] "RemoveContainer" containerID="3faa7e31a4a92eaabb5debc15db55786a5fc6f816c6085099c5f94f15a2f6921"
Oct 30 00:42:57.947107 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:42:57.947086    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3faa7e31a4a92eaabb5debc15db55786a5fc6f816c6085099c5f94f15a2f6921\": container with ID starting with 3faa7e31a4a92eaabb5debc15db55786a5fc6f816c6085099c5f94f15a2f6921 not found: ID does not exist" containerID="3faa7e31a4a92eaabb5debc15db55786a5fc6f816c6085099c5f94f15a2f6921"
Oct 30 00:42:57.947149 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.947112    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3faa7e31a4a92eaabb5debc15db55786a5fc6f816c6085099c5f94f15a2f6921"} err="failed to get container status \"3faa7e31a4a92eaabb5debc15db55786a5fc6f816c6085099c5f94f15a2f6921\": rpc error: code = NotFound desc = could not find container \"3faa7e31a4a92eaabb5debc15db55786a5fc6f816c6085099c5f94f15a2f6921\": container with ID starting with 3faa7e31a4a92eaabb5debc15db55786a5fc6f816c6085099c5f94f15a2f6921 not found: ID does not exist"
Oct 30 00:42:57.947149 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.947126    4425 scope.go:122] "RemoveContainer" containerID="f0be405cd48ea0923fb509b440facaa3fe8cb52081d666d0390800a3a2bbb6dd"
Oct 30 00:42:57.947392 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:42:57.947373    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0be405cd48ea0923fb509b440facaa3fe8cb52081d666d0390800a3a2bbb6dd\": container with ID starting with f0be405cd48ea0923fb509b440facaa3fe8cb52081d666d0390800a3a2bbb6dd not found: ID does not exist" containerID="f0be405cd48ea0923fb509b440facaa3fe8cb52081d666d0390800a3a2bbb6dd"
Oct 30 00:42:57.947442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.947398    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0be405cd48ea0923fb509b440facaa3fe8cb52081d666d0390800a3a2bbb6dd"} err="failed to get container status \"f0be405cd48ea0923fb509b440facaa3fe8cb52081d666d0390800a3a2bbb6dd\": rpc error: code = NotFound desc = could not find container \"f0be405cd48ea0923fb509b440facaa3fe8cb52081d666d0390800a3a2bbb6dd\": container with ID starting with f0be405cd48ea0923fb509b440facaa3fe8cb52081d666d0390800a3a2bbb6dd not found: ID does not exist"
Oct 30 00:42:57.954369 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:57.954351    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:42:59.419784 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:42:59.419749    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" path="/var/lib/kubelet/pods/b4591819-3c83-4361-bbe4-977b5f4fd3c3/volumes"
Oct 30 00:43:00.903532 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:00.903497    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:43:00.903924 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:00.903784    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="prometheus" containerID="cri-o://8fa118ebffda26500fc23e49f0c183047642405072855954269fa368f69e508f" gracePeriod=600
Oct 30 00:43:00.903924 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:00.903816    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="kube-rbac-proxy" containerID="cri-o://0120593312c9127d2a3c3223f90866e0bef965e1c98eb217970a01e69003b54e" gracePeriod=600
Oct 30 00:43:00.903924 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:00.903878    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://00d20d1c54d9220de4df2104bdb3b2832f84d74460136e208216318ed801996b" gracePeriod=600
Oct 30 00:43:00.903924 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:00.903805    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="thanos-sidecar" containerID="cri-o://f3006f98457d180dbfd27731a8ee123d7fa4b4c6b030694c7038da9bc69f5380" gracePeriod=600
Oct 30 00:43:00.904126 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:00.903837    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="kube-rbac-proxy-web" containerID="cri-o://8f3f8cde70efd320c65b06781a9bf82d8d598bc66f0b0467ecf66283d784118e" gracePeriod=600
Oct 30 00:43:00.904126 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:00.903860    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="config-reloader" containerID="cri-o://88dd5dbaf077c5e0cd902508bcf5fec34b53f567b262f48bc2a800b1818b2256" gracePeriod=600
Oct 30 00:43:01.843715 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:43:01.843667    4425 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="8fa118ebffda26500fc23e49f0c183047642405072855954269fa368f69e508f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 00:43:01.843715 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:43:01.843725    4425 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="8fa118ebffda26500fc23e49f0c183047642405072855954269fa368f69e508f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 00:43:01.844008 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:43:01.843742    4425 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="8fa118ebffda26500fc23e49f0c183047642405072855954269fa368f69e508f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 00:43:01.844008 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:01.843756    4425 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Oct 30 00:43:01.938868 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:01.938839    4425 generic.go:356] "Generic (PLEG): container finished" podID="8f66871c-bddb-4f93-8574-8706f525d49b" containerID="00d20d1c54d9220de4df2104bdb3b2832f84d74460136e208216318ed801996b" exitCode=0
Oct 30 00:43:01.938868 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:01.938862    4425 generic.go:356] "Generic (PLEG): container finished" podID="8f66871c-bddb-4f93-8574-8706f525d49b" containerID="0120593312c9127d2a3c3223f90866e0bef965e1c98eb217970a01e69003b54e" exitCode=0
Oct 30 00:43:01.938868 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:01.938870    4425 generic.go:356] "Generic (PLEG): container finished" podID="8f66871c-bddb-4f93-8574-8706f525d49b" containerID="f3006f98457d180dbfd27731a8ee123d7fa4b4c6b030694c7038da9bc69f5380" exitCode=0
Oct 30 00:43:01.938868 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:01.938876    4425 generic.go:356] "Generic (PLEG): container finished" podID="8f66871c-bddb-4f93-8574-8706f525d49b" containerID="88dd5dbaf077c5e0cd902508bcf5fec34b53f567b262f48bc2a800b1818b2256" exitCode=0
Oct 30 00:43:01.939402 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:01.938882    4425 generic.go:356] "Generic (PLEG): container finished" podID="8f66871c-bddb-4f93-8574-8706f525d49b" containerID="8fa118ebffda26500fc23e49f0c183047642405072855954269fa368f69e508f" exitCode=0
Oct 30 00:43:01.939402 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:01.938910    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f66871c-bddb-4f93-8574-8706f525d49b","Type":"ContainerDied","Data":"00d20d1c54d9220de4df2104bdb3b2832f84d74460136e208216318ed801996b"}
Oct 30 00:43:01.939402 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:01.939030    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f66871c-bddb-4f93-8574-8706f525d49b","Type":"ContainerDied","Data":"0120593312c9127d2a3c3223f90866e0bef965e1c98eb217970a01e69003b54e"}
Oct 30 00:43:01.939402 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:01.939073    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f66871c-bddb-4f93-8574-8706f525d49b","Type":"ContainerDied","Data":"f3006f98457d180dbfd27731a8ee123d7fa4b4c6b030694c7038da9bc69f5380"}
Oct 30 00:43:01.939402 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:01.939114    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f66871c-bddb-4f93-8574-8706f525d49b","Type":"ContainerDied","Data":"88dd5dbaf077c5e0cd902508bcf5fec34b53f567b262f48bc2a800b1818b2256"}
Oct 30 00:43:01.939402 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:01.939171    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f66871c-bddb-4f93-8574-8706f525d49b","Type":"ContainerDied","Data":"8fa118ebffda26500fc23e49f0c183047642405072855954269fa368f69e508f"}
Oct 30 00:43:02.146992 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.146969    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:02.211686 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.211661    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f66871c-bddb-4f93-8574-8706f525d49b-tls-assets\" (UniqueName: \"kubernetes.io/projected/8f66871c-bddb-4f93-8574-8706f525d49b-tls-assets\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.211822 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.211708    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-metrics-client-ca\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.211822 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.211726    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-kubelet-serving-ca-bundle\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.211822 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.211753    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-serving-certs-ca-bundle\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.211822 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.211783    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-config\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-config\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.211822 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.211812    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.212091 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.211852    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-tls\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.212091 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.211907    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f66871c-bddb-4f93-8574-8706f525d49b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f66871c-bddb-4f93-8574-8706f525d49b-config-out\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.212091 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.211935    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-grpc-tls\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.212091 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.211961    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f66871c-bddb-4f93-8574-8706f525d49b-kube-api-access-jnxdb\" (UniqueName: \"kubernetes.io/projected/8f66871c-bddb-4f93-8574-8706f525d49b-kube-api-access-jnxdb\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.212091 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.211997    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-0\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.212091 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.212029    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-1\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.212091 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.212057    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-db\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.214010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.212097    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-kube-rbac-proxy\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.214010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.212105    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-metrics-client-ca" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:43:02.214010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.212126    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-web-config\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-web-config\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.214010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.212135    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-kubelet-serving-ca-bundle" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:43:02.214010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.212182    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.214010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.212227    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-2\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.214010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.212254    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-metrics-client-certs\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.214010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.212248    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-serving-certs-ca-bundle" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:43:02.214010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.212289    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-thanos-prometheus-http-client-file\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.214010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.212322    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-trusted-ca-bundle\") pod \"8f66871c-bddb-4f93-8574-8706f525d49b\" (UID: \"8f66871c-bddb-4f93-8574-8706f525d49b\") "
Oct 30 00:43:02.214010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.212516    4425 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-metrics-client-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.214010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.212540    4425 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.214010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.212558    4425 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.214010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.212976    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-trusted-ca-bundle" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:43:02.214010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.213282    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-2" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:43:02.214010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.213289    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-1" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:43:02.214010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.213424    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-db" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:43:02.215136 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.214547    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-0" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:43:02.215136 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.214734    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-kube-rbac-proxy-web" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:43:02.215136 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.214843    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-config" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:43:02.215449 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.215291    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f66871c-bddb-4f93-8574-8706f525d49b-kube-api-access-jnxdb" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "kube-api-access-jnxdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:43:02.215704 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.215668    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f66871c-bddb-4f93-8574-8706f525d49b-tls-assets" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:43:02.215960 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.215910    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f66871c-bddb-4f93-8574-8706f525d49b-config-out" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:43:02.216562 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.216495    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-thanos-sidecar-tls" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:43:02.216562 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.216526    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-metrics-client-certs" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:43:02.216973 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.216937    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-kube-rbac-proxy" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:43:02.217071 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.217049    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-tls" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:43:02.217626 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.217603    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-grpc-tls" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:43:02.217707 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.217684    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-thanos-prometheus-http-client-file" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:43:02.227073 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.227053    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-web-config" pod "8f66871c-bddb-4f93-8574-8706f525d49b" (UID: "8f66871c-bddb-4f93-8574-8706f525d49b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:43:02.313838 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.313808    4425 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.313838 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.313835    4425 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.314029 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.313849    4425 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-metrics-client-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.314029 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.313862    4425 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.314029 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.313875    4425 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.314029 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.313889    4425 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f66871c-bddb-4f93-8574-8706f525d49b-tls-assets\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.314029 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.313901    4425 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-config\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.314029 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.313913    4425 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.314029 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.313926    4425 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.314029 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.313937    4425 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f66871c-bddb-4f93-8574-8706f525d49b-config-out\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.314029 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.313949    4425 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-grpc-tls\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.314029 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.313961    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnxdb\" (UniqueName: \"kubernetes.io/projected/8f66871c-bddb-4f93-8574-8706f525d49b-kube-api-access-jnxdb\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.314029 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.313973    4425 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.314029 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.313987    4425 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.314029 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.314000    4425 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f66871c-bddb-4f93-8574-8706f525d49b-prometheus-k8s-db\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.314029 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.314014    4425 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-secret-kube-rbac-proxy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.314029 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.314026    4425 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f66871c-bddb-4f93-8574-8706f525d49b-web-config\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:02.945907 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.945871    4425 generic.go:356] "Generic (PLEG): container finished" podID="8f66871c-bddb-4f93-8574-8706f525d49b" containerID="8f3f8cde70efd320c65b06781a9bf82d8d598bc66f0b0467ecf66283d784118e" exitCode=0
Oct 30 00:43:02.946318 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.945951    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:02.946318 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.945960    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f66871c-bddb-4f93-8574-8706f525d49b","Type":"ContainerDied","Data":"8f3f8cde70efd320c65b06781a9bf82d8d598bc66f0b0467ecf66283d784118e"}
Oct 30 00:43:02.946318 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.946030    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f66871c-bddb-4f93-8574-8706f525d49b","Type":"ContainerDied","Data":"6650362f652c5f47bca73024c6aa1c33e07e2a66cb91c72df2d2a017232c1b0c"}
Oct 30 00:43:02.946318 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.946063    4425 scope.go:122] "RemoveContainer" containerID="00d20d1c54d9220de4df2104bdb3b2832f84d74460136e208216318ed801996b"
Oct 30 00:43:02.953877 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.953792    4425 scope.go:122] "RemoveContainer" containerID="0120593312c9127d2a3c3223f90866e0bef965e1c98eb217970a01e69003b54e"
Oct 30 00:43:02.960268 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.960249    4425 scope.go:122] "RemoveContainer" containerID="8f3f8cde70efd320c65b06781a9bf82d8d598bc66f0b0467ecf66283d784118e"
Oct 30 00:43:02.966377 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.966361    4425 scope.go:122] "RemoveContainer" containerID="f3006f98457d180dbfd27731a8ee123d7fa4b4c6b030694c7038da9bc69f5380"
Oct 30 00:43:02.972433 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.972417    4425 scope.go:122] "RemoveContainer" containerID="88dd5dbaf077c5e0cd902508bcf5fec34b53f567b262f48bc2a800b1818b2256"
Oct 30 00:43:02.979138 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.979122    4425 scope.go:122] "RemoveContainer" containerID="8fa118ebffda26500fc23e49f0c183047642405072855954269fa368f69e508f"
Oct 30 00:43:02.985709 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.985692    4425 scope.go:122] "RemoveContainer" containerID="5b61de753851f99d5cc5565c84e0ad06e139e2e98797b5639396a55bbb9c877e"
Oct 30 00:43:02.991756 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.991740    4425 scope.go:122] "RemoveContainer" containerID="00d20d1c54d9220de4df2104bdb3b2832f84d74460136e208216318ed801996b"
Oct 30 00:43:02.991990 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:43:02.991971    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d20d1c54d9220de4df2104bdb3b2832f84d74460136e208216318ed801996b\": container with ID starting with 00d20d1c54d9220de4df2104bdb3b2832f84d74460136e208216318ed801996b not found: ID does not exist" containerID="00d20d1c54d9220de4df2104bdb3b2832f84d74460136e208216318ed801996b"
Oct 30 00:43:02.992060 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.992002    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d20d1c54d9220de4df2104bdb3b2832f84d74460136e208216318ed801996b"} err="failed to get container status \"00d20d1c54d9220de4df2104bdb3b2832f84d74460136e208216318ed801996b\": rpc error: code = NotFound desc = could not find container \"00d20d1c54d9220de4df2104bdb3b2832f84d74460136e208216318ed801996b\": container with ID starting with 00d20d1c54d9220de4df2104bdb3b2832f84d74460136e208216318ed801996b not found: ID does not exist"
Oct 30 00:43:02.992060 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.992023    4425 scope.go:122] "RemoveContainer" containerID="0120593312c9127d2a3c3223f90866e0bef965e1c98eb217970a01e69003b54e"
Oct 30 00:43:02.992275 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:43:02.992256    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0120593312c9127d2a3c3223f90866e0bef965e1c98eb217970a01e69003b54e\": container with ID starting with 0120593312c9127d2a3c3223f90866e0bef965e1c98eb217970a01e69003b54e not found: ID does not exist" containerID="0120593312c9127d2a3c3223f90866e0bef965e1c98eb217970a01e69003b54e"
Oct 30 00:43:02.992329 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.992281    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0120593312c9127d2a3c3223f90866e0bef965e1c98eb217970a01e69003b54e"} err="failed to get container status \"0120593312c9127d2a3c3223f90866e0bef965e1c98eb217970a01e69003b54e\": rpc error: code = NotFound desc = could not find container \"0120593312c9127d2a3c3223f90866e0bef965e1c98eb217970a01e69003b54e\": container with ID starting with 0120593312c9127d2a3c3223f90866e0bef965e1c98eb217970a01e69003b54e not found: ID does not exist"
Oct 30 00:43:02.992329 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.992301    4425 scope.go:122] "RemoveContainer" containerID="8f3f8cde70efd320c65b06781a9bf82d8d598bc66f0b0467ecf66283d784118e"
Oct 30 00:43:02.992512 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:43:02.992498    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3f8cde70efd320c65b06781a9bf82d8d598bc66f0b0467ecf66283d784118e\": container with ID starting with 8f3f8cde70efd320c65b06781a9bf82d8d598bc66f0b0467ecf66283d784118e not found: ID does not exist" containerID="8f3f8cde70efd320c65b06781a9bf82d8d598bc66f0b0467ecf66283d784118e"
Oct 30 00:43:02.992553 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.992516    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3f8cde70efd320c65b06781a9bf82d8d598bc66f0b0467ecf66283d784118e"} err="failed to get container status \"8f3f8cde70efd320c65b06781a9bf82d8d598bc66f0b0467ecf66283d784118e\": rpc error: code = NotFound desc = could not find container \"8f3f8cde70efd320c65b06781a9bf82d8d598bc66f0b0467ecf66283d784118e\": container with ID starting with 8f3f8cde70efd320c65b06781a9bf82d8d598bc66f0b0467ecf66283d784118e not found: ID does not exist"
Oct 30 00:43:02.992553 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.992528    4425 scope.go:122] "RemoveContainer" containerID="f3006f98457d180dbfd27731a8ee123d7fa4b4c6b030694c7038da9bc69f5380"
Oct 30 00:43:02.992746 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:43:02.992730    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3006f98457d180dbfd27731a8ee123d7fa4b4c6b030694c7038da9bc69f5380\": container with ID starting with f3006f98457d180dbfd27731a8ee123d7fa4b4c6b030694c7038da9bc69f5380 not found: ID does not exist" containerID="f3006f98457d180dbfd27731a8ee123d7fa4b4c6b030694c7038da9bc69f5380"
Oct 30 00:43:02.992806 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.992754    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3006f98457d180dbfd27731a8ee123d7fa4b4c6b030694c7038da9bc69f5380"} err="failed to get container status \"f3006f98457d180dbfd27731a8ee123d7fa4b4c6b030694c7038da9bc69f5380\": rpc error: code = NotFound desc = could not find container \"f3006f98457d180dbfd27731a8ee123d7fa4b4c6b030694c7038da9bc69f5380\": container with ID starting with f3006f98457d180dbfd27731a8ee123d7fa4b4c6b030694c7038da9bc69f5380 not found: ID does not exist"
Oct 30 00:43:02.992806 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.992774    4425 scope.go:122] "RemoveContainer" containerID="88dd5dbaf077c5e0cd902508bcf5fec34b53f567b262f48bc2a800b1818b2256"
Oct 30 00:43:02.992984 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:43:02.992970    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88dd5dbaf077c5e0cd902508bcf5fec34b53f567b262f48bc2a800b1818b2256\": container with ID starting with 88dd5dbaf077c5e0cd902508bcf5fec34b53f567b262f48bc2a800b1818b2256 not found: ID does not exist" containerID="88dd5dbaf077c5e0cd902508bcf5fec34b53f567b262f48bc2a800b1818b2256"
Oct 30 00:43:02.993028 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.992987    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88dd5dbaf077c5e0cd902508bcf5fec34b53f567b262f48bc2a800b1818b2256"} err="failed to get container status \"88dd5dbaf077c5e0cd902508bcf5fec34b53f567b262f48bc2a800b1818b2256\": rpc error: code = NotFound desc = could not find container \"88dd5dbaf077c5e0cd902508bcf5fec34b53f567b262f48bc2a800b1818b2256\": container with ID starting with 88dd5dbaf077c5e0cd902508bcf5fec34b53f567b262f48bc2a800b1818b2256 not found: ID does not exist"
Oct 30 00:43:02.993028 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.992999    4425 scope.go:122] "RemoveContainer" containerID="8fa118ebffda26500fc23e49f0c183047642405072855954269fa368f69e508f"
Oct 30 00:43:02.993274 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:43:02.993253    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fa118ebffda26500fc23e49f0c183047642405072855954269fa368f69e508f\": container with ID starting with 8fa118ebffda26500fc23e49f0c183047642405072855954269fa368f69e508f not found: ID does not exist" containerID="8fa118ebffda26500fc23e49f0c183047642405072855954269fa368f69e508f"
Oct 30 00:43:02.993321 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.993281    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa118ebffda26500fc23e49f0c183047642405072855954269fa368f69e508f"} err="failed to get container status \"8fa118ebffda26500fc23e49f0c183047642405072855954269fa368f69e508f\": rpc error: code = NotFound desc = could not find container \"8fa118ebffda26500fc23e49f0c183047642405072855954269fa368f69e508f\": container with ID starting with 8fa118ebffda26500fc23e49f0c183047642405072855954269fa368f69e508f not found: ID does not exist"
Oct 30 00:43:02.993321 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.993299    4425 scope.go:122] "RemoveContainer" containerID="5b61de753851f99d5cc5565c84e0ad06e139e2e98797b5639396a55bbb9c877e"
Oct 30 00:43:02.993515 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:43:02.993501    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b61de753851f99d5cc5565c84e0ad06e139e2e98797b5639396a55bbb9c877e\": container with ID starting with 5b61de753851f99d5cc5565c84e0ad06e139e2e98797b5639396a55bbb9c877e not found: ID does not exist" containerID="5b61de753851f99d5cc5565c84e0ad06e139e2e98797b5639396a55bbb9c877e"
Oct 30 00:43:02.993565 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:02.993520    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b61de753851f99d5cc5565c84e0ad06e139e2e98797b5639396a55bbb9c877e"} err="failed to get container status \"5b61de753851f99d5cc5565c84e0ad06e139e2e98797b5639396a55bbb9c877e\": rpc error: code = NotFound desc = could not find container \"5b61de753851f99d5cc5565c84e0ad06e139e2e98797b5639396a55bbb9c877e\": container with ID starting with 5b61de753851f99d5cc5565c84e0ad06e139e2e98797b5639396a55bbb9c877e not found: ID does not exist"
Oct 30 00:43:03.032544 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.032523    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:43:03.051444 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.051422    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:43:03.092397 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092374    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:43:03.092630 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092620    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="kube-rbac-proxy-web"
Oct 30 00:43:03.092675 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092632    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="kube-rbac-proxy-web"
Oct 30 00:43:03.092675 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092645    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="init-config-reloader"
Oct 30 00:43:03.092675 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092650    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="init-config-reloader"
Oct 30 00:43:03.092675 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092656    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="prometheus"
Oct 30 00:43:03.092675 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092661    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="prometheus"
Oct 30 00:43:03.092675 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092670    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="prom-label-proxy"
Oct 30 00:43:03.092675 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092675    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="prom-label-proxy"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092682    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="thanos-sidecar"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092686    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="thanos-sidecar"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092694    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="alertmanager"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092698    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="alertmanager"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092704    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="init-config-reloader"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092708    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="init-config-reloader"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092715    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="config-reloader"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092720    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="config-reloader"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092729    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="kube-rbac-proxy"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092734    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="kube-rbac-proxy"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092742    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="kube-rbac-proxy"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092747    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="kube-rbac-proxy"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092753    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="kube-rbac-proxy-metric"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092757    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="kube-rbac-proxy-metric"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092764    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="kube-rbac-proxy-web"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092769    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="kube-rbac-proxy-web"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092778    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="kube-rbac-proxy-thanos"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092784    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="kube-rbac-proxy-thanos"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092803    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="config-reloader"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092809    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="config-reloader"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092822    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d1d10b60-20ba-442d-815f-01653be20945" containerName="console"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092828    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1d10b60-20ba-442d-815f-01653be20945" containerName="console"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092895    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1d10b60-20ba-442d-815f-01653be20945" containerName="console"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092908    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="kube-rbac-proxy-web"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092918    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="prometheus"
Oct 30 00:43:03.092917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092929    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="prom-label-proxy"
Oct 30 00:43:03.093725 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092939    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="thanos-sidecar"
Oct 30 00:43:03.093725 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092945    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="alertmanager"
Oct 30 00:43:03.093725 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092951    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="config-reloader"
Oct 30 00:43:03.093725 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092957    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="kube-rbac-proxy"
Oct 30 00:43:03.093725 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092963    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="kube-rbac-proxy"
Oct 30 00:43:03.093725 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092968    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="kube-rbac-proxy-metric"
Oct 30 00:43:03.093725 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092973    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4591819-3c83-4361-bbe4-977b5f4fd3c3" containerName="kube-rbac-proxy-web"
Oct 30 00:43:03.093725 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092979    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="kube-rbac-proxy-thanos"
Oct 30 00:43:03.093725 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.092985    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" containerName="config-reloader"
Oct 30 00:43:03.096864 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.096848    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.110804 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.110784    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 00:43:03.111109 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.111085    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 00:43:03.111310 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.111294    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 00:43:03.121361 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.121341    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 00:43:03.121451 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.121387    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 00:43:03.121451 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.121402    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 00:43:03.121451 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.121415    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 00:43:03.121557 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.121473    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 00:43:03.121690 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.121672    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f7pl9\""
Oct 30 00:43:03.121991 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.121972    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 00:43:03.122106 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.122040    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 00:43:03.122106 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.122060    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 00:43:03.122106 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.121980    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 00:43:03.122261 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.122140    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 00:43:03.122307 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.122267    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5kuqvvfnaq0up\""
Oct 30 00:43:03.122588 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.122575    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 00:43:03.134267 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.133540    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:43:03.220176 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220104    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220176 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220131    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220330 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220176    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a1a742db-26fb-4195-9676-364b7d9487c4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220330 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220249    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220330 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220279    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220330 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220310    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-web-config\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220503 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220357    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220503 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220403    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1a742db-26fb-4195-9676-364b7d9487c4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220503 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220423    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j54p\" (UniqueName: \"kubernetes.io/projected/a1a742db-26fb-4195-9676-364b7d9487c4-kube-api-access-6j54p\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220503 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220437    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220503 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220477    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220709 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220505    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1a742db-26fb-4195-9676-364b7d9487c4-config-out\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220709 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220547    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220709 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220575    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220709 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220591    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220709 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220621    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220709 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220659    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220709 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220675    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-config\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220709 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220689    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.220709 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.220704    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.321429 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321400    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a1a742db-26fb-4195-9676-364b7d9487c4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.321429 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321431    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.321617 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321446    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.321617 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321465    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-web-config\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.321617 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321483    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.321617 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321503    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1a742db-26fb-4195-9676-364b7d9487c4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.321617 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321530    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j54p\" (UniqueName: \"kubernetes.io/projected/a1a742db-26fb-4195-9676-364b7d9487c4-kube-api-access-6j54p\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.321617 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321553    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.321617 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321582    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.321617 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321606    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1a742db-26fb-4195-9676-364b7d9487c4-config-out\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.322012 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321634    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.322012 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321653    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.322012 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321668    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.322012 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321690    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.322012 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321720    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.322012 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321744    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-config\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.322012 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321765    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.322012 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321789    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.322012 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321830    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.322012 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.321855    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.322535 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.322128    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.322535 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.322258    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.322535 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.322341    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.323187 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.322869    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.323187 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.323149    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.323849 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.323473    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.323849 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.323571    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a1a742db-26fb-4195-9676-364b7d9487c4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.324640 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.324612    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.324736 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.324644    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-web-config\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.324736 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.324699    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a1a742db-26fb-4195-9676-364b7d9487c4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.326242 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.326203    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-config\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.326404 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.326360    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.326514 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.326484    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.326992 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.326967    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.327081 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.327058    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.327193 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.327171    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a1a742db-26fb-4195-9676-364b7d9487c4-config-out\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.327337 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.327204    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a1a742db-26fb-4195-9676-364b7d9487c4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.327574 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.327556    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.327635 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.327588    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a1a742db-26fb-4195-9676-364b7d9487c4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.342806 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.342783    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j54p\" (UniqueName: \"kubernetes.io/projected/a1a742db-26fb-4195-9676-364b7d9487c4-kube-api-access-6j54p\") pod \"prometheus-k8s-0\" (UID: \"a1a742db-26fb-4195-9676-364b7d9487c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.414926 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.414903    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:03.419558 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.419529    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f66871c-bddb-4f93-8574-8706f525d49b" path="/var/lib/kubelet/pods/8f66871c-bddb-4f93-8574-8706f525d49b/volumes"
Oct 30 00:43:03.741931 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.741897    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 00:43:03.744459 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:43:03.744432    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a742db_26fb_4195_9676_364b7d9487c4.slice/crio-c9a379c15bd45be8d62a3014072a06c481ecd2c195c0252a2245f4289520fdf9 WatchSource:0}: Error finding container c9a379c15bd45be8d62a3014072a06c481ecd2c195c0252a2245f4289520fdf9: Status 404 returned error can't find the container with id c9a379c15bd45be8d62a3014072a06c481ecd2c195c0252a2245f4289520fdf9
Oct 30 00:43:03.952411 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.952379    4425 generic.go:356] "Generic (PLEG): container finished" podID="a1a742db-26fb-4195-9676-364b7d9487c4" containerID="6ef46e7980d3c9addebd7d0f5fdeb1a0c699795c04548b1450be781866f41706" exitCode=0
Oct 30 00:43:03.952755 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.952435    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a742db-26fb-4195-9676-364b7d9487c4","Type":"ContainerDied","Data":"6ef46e7980d3c9addebd7d0f5fdeb1a0c699795c04548b1450be781866f41706"}
Oct 30 00:43:03.952755 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:03.952499    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a742db-26fb-4195-9676-364b7d9487c4","Type":"ContainerStarted","Data":"c9a379c15bd45be8d62a3014072a06c481ecd2c195c0252a2245f4289520fdf9"}
Oct 30 00:43:04.959597 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:04.959558    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a742db-26fb-4195-9676-364b7d9487c4","Type":"ContainerStarted","Data":"24725fedc8ba640041708aeaad6e22c17b83aff23d68e36f5211d0bd51818b5b"}
Oct 30 00:43:04.960055 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:04.959621    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a742db-26fb-4195-9676-364b7d9487c4","Type":"ContainerStarted","Data":"01004779be3487916aab4047d2e7a278be96de36ff0ab25c68bbdb07c2903bf8"}
Oct 30 00:43:04.960055 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:04.959647    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a742db-26fb-4195-9676-364b7d9487c4","Type":"ContainerStarted","Data":"e0dab1817c0307099aa6109fba40540c352702ad7dfad1766281efab65f1e7ac"}
Oct 30 00:43:04.960055 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:04.959670    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a742db-26fb-4195-9676-364b7d9487c4","Type":"ContainerStarted","Data":"89a0505eda52e6a69597a32594f0dcdf99ce2e2b3c179fb9d31fe264904a9054"}
Oct 30 00:43:04.960055 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:04.959692    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a742db-26fb-4195-9676-364b7d9487c4","Type":"ContainerStarted","Data":"24254d12a1a39cd367266ebe8b4f0a96e2f6a37bc519add9ebd387f1dde84cc6"}
Oct 30 00:43:04.960055 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:04.959756    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a1a742db-26fb-4195-9676-364b7d9487c4","Type":"ContainerStarted","Data":"7bc596bd002e93b71cdb4be66986188ba96acf288f0a0612bd7e55307049002e"}
Oct 30 00:43:05.020513 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:05.020466    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.003489788 podStartE2EDuration="2.020452475s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.962687ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:05.019472041 +0000 UTC m=+93.769127349" watchObservedRunningTime="2026-10-30 00:43:05.020452475 +0000 UTC m=+93.770107783"
Oct 30 00:43:07.552522 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:43:07.552481    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:43:08.415009 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:08.414971    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:43:14.637714 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:14.637680    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79ffc79964-cjk8r"]
Oct 30 00:43:37.553635 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:43:37.553598    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:43:39.657210 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:39.657139    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-79ffc79964-cjk8r" podUID="3ecabd9e-2157-4e6b-a21b-c93a7771cc46" containerName="console" containerID="cri-o://ad52f963cb2b50eb1ed90304083daff2240cc2b42c59ee26b0278cbdca7a8baa" gracePeriod=15
Oct 30 00:43:39.899320 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:39.899295    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:43:39.999125 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:39.999067    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-serving-cert\") pod \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") "
Oct 30 00:43:39.999125 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:39.999119    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-kube-api-access-rhcwh\" (UniqueName: \"kubernetes.io/projected/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-kube-api-access-rhcwh\") pod \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") "
Oct 30 00:43:39.999325 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:39.999147    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-tmp\") pod \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") "
Oct 30 00:43:39.999325 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:39.999197    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-trusted-ca-bundle\") pod \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") "
Oct 30 00:43:39.999325 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:39.999220    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-oauth-serving-cert\") pod \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") "
Oct 30 00:43:39.999325 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:39.999256    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-service-ca\") pod \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") "
Oct 30 00:43:39.999325 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:39.999280    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-config\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-config\") pod \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") "
Oct 30 00:43:39.999325 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:39.999297    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-oauth-config\") pod \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\" (UID: \"3ecabd9e-2157-4e6b-a21b-c93a7771cc46\") "
Oct 30 00:43:39.999635 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:39.999493    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-tmp" pod "3ecabd9e-2157-4e6b-a21b-c93a7771cc46" (UID: "3ecabd9e-2157-4e6b-a21b-c93a7771cc46"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:43:39.999692 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:39.999673    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-config" pod "3ecabd9e-2157-4e6b-a21b-c93a7771cc46" (UID: "3ecabd9e-2157-4e6b-a21b-c93a7771cc46"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:43:39.999746 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:39.999680    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-trusted-ca-bundle" pod "3ecabd9e-2157-4e6b-a21b-c93a7771cc46" (UID: "3ecabd9e-2157-4e6b-a21b-c93a7771cc46"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:43:39.999799 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:39.999761    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-service-ca" pod "3ecabd9e-2157-4e6b-a21b-c93a7771cc46" (UID: "3ecabd9e-2157-4e6b-a21b-c93a7771cc46"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:43:39.999850 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:39.999793    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-oauth-serving-cert" pod "3ecabd9e-2157-4e6b-a21b-c93a7771cc46" (UID: "3ecabd9e-2157-4e6b-a21b-c93a7771cc46"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:43:40.001435 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.001411    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-oauth-config" pod "3ecabd9e-2157-4e6b-a21b-c93a7771cc46" (UID: "3ecabd9e-2157-4e6b-a21b-c93a7771cc46"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:43:40.001689 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.001670    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-kube-api-access-rhcwh" pod "3ecabd9e-2157-4e6b-a21b-c93a7771cc46" (UID: "3ecabd9e-2157-4e6b-a21b-c93a7771cc46"). InnerVolumeSpecName "kube-api-access-rhcwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:43:40.001689 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.001675    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-serving-cert" pod "3ecabd9e-2157-4e6b-a21b-c93a7771cc46" (UID: "3ecabd9e-2157-4e6b-a21b-c93a7771cc46"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:43:40.085914 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.085885    4425 generic.go:356] "Generic (PLEG): container finished" podID="3ecabd9e-2157-4e6b-a21b-c93a7771cc46" containerID="ad52f963cb2b50eb1ed90304083daff2240cc2b42c59ee26b0278cbdca7a8baa" exitCode=0
Oct 30 00:43:40.086016 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.085937    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79ffc79964-cjk8r"
Oct 30 00:43:40.086016 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.085966    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79ffc79964-cjk8r" event={"ID":"3ecabd9e-2157-4e6b-a21b-c93a7771cc46","Type":"ContainerDied","Data":"ad52f963cb2b50eb1ed90304083daff2240cc2b42c59ee26b0278cbdca7a8baa"}
Oct 30 00:43:40.086088 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.086018    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79ffc79964-cjk8r" event={"ID":"3ecabd9e-2157-4e6b-a21b-c93a7771cc46","Type":"ContainerDied","Data":"b64d67bfc4aaa3e170add7c6b9a49c89bae4986ffca3ab43f4fc3c5d30d96c8d"}
Oct 30 00:43:40.086088 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.086041    4425 scope.go:122] "RemoveContainer" containerID="ad52f963cb2b50eb1ed90304083daff2240cc2b42c59ee26b0278cbdca7a8baa"
Oct 30 00:43:40.094381 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.094364    4425 scope.go:122] "RemoveContainer" containerID="ad52f963cb2b50eb1ed90304083daff2240cc2b42c59ee26b0278cbdca7a8baa"
Oct 30 00:43:40.094658 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:43:40.094639    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad52f963cb2b50eb1ed90304083daff2240cc2b42c59ee26b0278cbdca7a8baa\": container with ID starting with ad52f963cb2b50eb1ed90304083daff2240cc2b42c59ee26b0278cbdca7a8baa not found: ID does not exist" containerID="ad52f963cb2b50eb1ed90304083daff2240cc2b42c59ee26b0278cbdca7a8baa"
Oct 30 00:43:40.094704 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.094667    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad52f963cb2b50eb1ed90304083daff2240cc2b42c59ee26b0278cbdca7a8baa"} err="failed to get container status \"ad52f963cb2b50eb1ed90304083daff2240cc2b42c59ee26b0278cbdca7a8baa\": rpc error: code = NotFound desc = could not find container \"ad52f963cb2b50eb1ed90304083daff2240cc2b42c59ee26b0278cbdca7a8baa\": container with ID starting with ad52f963cb2b50eb1ed90304083daff2240cc2b42c59ee26b0278cbdca7a8baa not found: ID does not exist"
Oct 30 00:43:40.100055 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.100036    4425 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-trusted-ca-bundle\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:40.100055 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.100056    4425 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-oauth-serving-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:40.100187 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.100065    4425 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:40.100187 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.100074    4425 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-config\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:40.100187 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.100083    4425 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-oauth-config\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:40.100187 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.100091    4425 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-console-serving-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:40.100187 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.100099    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhcwh\" (UniqueName: \"kubernetes.io/projected/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-kube-api-access-rhcwh\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:40.100187 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.100108    4425 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ecabd9e-2157-4e6b-a21b-c93a7771cc46-tmp\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:43:40.129704 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.129679    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79ffc79964-cjk8r"]
Oct 30 00:43:40.133576 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:40.133559    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79ffc79964-cjk8r"]
Oct 30 00:43:41.419540 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:43:41.419504    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ecabd9e-2157-4e6b-a21b-c93a7771cc46" path="/var/lib/kubelet/pods/3ecabd9e-2157-4e6b-a21b-c93a7771cc46/volumes"
Oct 30 00:44:03.415323 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:44:03.415262    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:44:03.430649 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:44:03.430627    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:44:04.183941 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:44:04.183916    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 00:44:07.553120 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:44:07.553080    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:44:37.552360 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:44:37.552324    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:45:07.552993 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:45:07.552954    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:45:37.552644 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:45:37.552604    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:46:07.551932 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:46:07.551893    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:46:17.371762 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.371726    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"]
Oct 30 00:46:17.372281 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.372060    4425 cpu_manager.go:414] "containerMap: removing container" podUID="3ecabd9e-2157-4e6b-a21b-c93a7771cc46" containerName="console"
Oct 30 00:46:17.372281 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.372073    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ecabd9e-2157-4e6b-a21b-c93a7771cc46" containerName="console"
Oct 30 00:46:17.372281 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.372127    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ecabd9e-2157-4e6b-a21b-c93a7771cc46" containerName="console"
Oct 30 00:46:17.375167 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.375135    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.388087 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.388066    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 00:46:17.389526 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.389498    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 00:46:17.389658 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.389606    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 00:46:17.398963 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.398944    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-lkrjc\""
Oct 30 00:46:17.399061 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.398983    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 00:46:17.399168 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.399144    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 00:46:17.462486 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.462464    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"]
Oct 30 00:46:17.506994 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.506971    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6891e83-9345-443d-8539-7f9174c3e648-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.507122 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.507020    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5hkm\" (UniqueName: \"kubernetes.io/projected/b6891e83-9345-443d-8539-7f9174c3e648-kube-api-access-l5hkm\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.507235 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.507212    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6891e83-9345-443d-8539-7f9174c3e648-tmp\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.507301 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.507259    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6891e83-9345-443d-8539-7f9174c3e648-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.507301 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.507294    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6891e83-9345-443d-8539-7f9174c3e648-manifests\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.507387 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.507366    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6891e83-9345-443d-8539-7f9174c3e648-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.607699 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.607673    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6891e83-9345-443d-8539-7f9174c3e648-tmp\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.607801 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.607704    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6891e83-9345-443d-8539-7f9174c3e648-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.607801 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.607723    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6891e83-9345-443d-8539-7f9174c3e648-manifests\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.607888 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.607846    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6891e83-9345-443d-8539-7f9174c3e648-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.607940 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.607889    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6891e83-9345-443d-8539-7f9174c3e648-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.607993 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.607970    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5hkm\" (UniqueName: \"kubernetes.io/projected/b6891e83-9345-443d-8539-7f9174c3e648-kube-api-access-l5hkm\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.608047 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.608025    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6891e83-9345-443d-8539-7f9174c3e648-manifests\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.610225 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.610198    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6891e83-9345-443d-8539-7f9174c3e648-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.610384 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.610360    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6891e83-9345-443d-8539-7f9174c3e648-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.610454 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.610385    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6891e83-9345-443d-8539-7f9174c3e648-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.610454 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.610384    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6891e83-9345-443d-8539-7f9174c3e648-tmp\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.629292 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.629235    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5hkm\" (UniqueName: \"kubernetes.io/projected/b6891e83-9345-443d-8539-7f9174c3e648-kube-api-access-l5hkm\") pod \"opendatahub-operator-controller-manager-f7f9fcc75-cskfq\" (UID: \"b6891e83-9345-443d-8539-7f9174c3e648\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:17.689311 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:17.689287    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:18.028385 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:18.028354    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"]
Oct 30 00:46:18.030781 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:46:18.030758    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6891e83_9345_443d_8539_7f9174c3e648.slice/crio-ec24cd90039511dbb3de3593409ed5a7c80454eab6d7df2821b177781018eff6 WatchSource:0}: Error finding container ec24cd90039511dbb3de3593409ed5a7c80454eab6d7df2821b177781018eff6: Status 404 returned error can't find the container with id ec24cd90039511dbb3de3593409ed5a7c80454eab6d7df2821b177781018eff6
Oct 30 00:46:18.634909 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:18.634755    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq" event={"ID":"b6891e83-9345-443d-8539-7f9174c3e648","Type":"ContainerStarted","Data":"ec24cd90039511dbb3de3593409ed5a7c80454eab6d7df2821b177781018eff6"}
Oct 30 00:46:21.647990 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:21.647959    4425 generic.go:356] "Generic (PLEG): container finished" podID="b6891e83-9345-443d-8539-7f9174c3e648" containerID="071639725e7860342fa7d6b01ca9d26d47f6c36a19713a843fe4e8c4f738e593" exitCode=0
Oct 30 00:46:21.648370 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:21.648040    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq" event={"ID":"b6891e83-9345-443d-8539-7f9174c3e648","Type":"ContainerDied","Data":"071639725e7860342fa7d6b01ca9d26d47f6c36a19713a843fe4e8c4f738e593"}
Oct 30 00:46:22.654437 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:22.654398    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq" event={"ID":"b6891e83-9345-443d-8539-7f9174c3e648","Type":"ContainerStarted","Data":"94422d7982c3888acf691a3c93a0885d0af790de401125c793ca5d70b47ebe55"}
Oct 30 00:46:22.654933 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:22.654580    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:22.750246 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:22.750190    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq" podStartSLOduration=2.293951971 podStartE2EDuration="5.750176486s" totalImagesPullingTime="3.048845494s" totalInitContainerRuntime="407.379021ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:22.739800746 +0000 UTC m=+291.489456055" watchObservedRunningTime="2026-10-30 00:46:22.750176486 +0000 UTC m=+291.499831789"
Oct 30 00:46:27.906578 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:27.906543    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-56pr2"]
Oct 30 00:46:27.914595 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:27.914569    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-56pr2"
Oct 30 00:46:27.939559 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:27.939533    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 00:46:27.939559 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:27.939564    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:46:27.939759 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:27.939664    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-8sxgv\""
Oct 30 00:46:27.940638 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:27.940615    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-56pr2"]
Oct 30 00:46:27.990033 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:27.990002    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71e604b6-8548-4bab-823d-b80c3f51c0da-tmp\") pod \"jobset-operator-674cd59dc5-56pr2\" (UID: \"71e604b6-8548-4bab-823d-b80c3f51c0da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-56pr2"
Oct 30 00:46:27.990185 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:27.990073    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgmhb\" (UniqueName: \"kubernetes.io/projected/71e604b6-8548-4bab-823d-b80c3f51c0da-kube-api-access-pgmhb\") pod \"jobset-operator-674cd59dc5-56pr2\" (UID: \"71e604b6-8548-4bab-823d-b80c3f51c0da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-56pr2"
Oct 30 00:46:28.090818 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:28.090783    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgmhb\" (UniqueName: \"kubernetes.io/projected/71e604b6-8548-4bab-823d-b80c3f51c0da-kube-api-access-pgmhb\") pod \"jobset-operator-674cd59dc5-56pr2\" (UID: \"71e604b6-8548-4bab-823d-b80c3f51c0da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-56pr2"
Oct 30 00:46:28.090978 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:28.090849    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71e604b6-8548-4bab-823d-b80c3f51c0da-tmp\") pod \"jobset-operator-674cd59dc5-56pr2\" (UID: \"71e604b6-8548-4bab-823d-b80c3f51c0da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-56pr2"
Oct 30 00:46:28.091261 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:28.091240    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71e604b6-8548-4bab-823d-b80c3f51c0da-tmp\") pod \"jobset-operator-674cd59dc5-56pr2\" (UID: \"71e604b6-8548-4bab-823d-b80c3f51c0da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-56pr2"
Oct 30 00:46:28.111506 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:28.111475    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgmhb\" (UniqueName: \"kubernetes.io/projected/71e604b6-8548-4bab-823d-b80c3f51c0da-kube-api-access-pgmhb\") pod \"jobset-operator-674cd59dc5-56pr2\" (UID: \"71e604b6-8548-4bab-823d-b80c3f51c0da\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-56pr2"
Oct 30 00:46:28.235492 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:28.235434    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-56pr2"
Oct 30 00:46:28.535311 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:28.535245    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-56pr2"]
Oct 30 00:46:28.538268 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:46:28.538243    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71e604b6_8548_4bab_823d_b80c3f51c0da.slice/crio-3d2ecea62e5920b37745814efbe2809886cbf2b76f655e59eb094e5f9add55dc WatchSource:0}: Error finding container 3d2ecea62e5920b37745814efbe2809886cbf2b76f655e59eb094e5f9add55dc: Status 404 returned error can't find the container with id 3d2ecea62e5920b37745814efbe2809886cbf2b76f655e59eb094e5f9add55dc
Oct 30 00:46:28.676832 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:28.676802    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-56pr2" event={"ID":"71e604b6-8548-4bab-823d-b80c3f51c0da","Type":"ContainerStarted","Data":"3d2ecea62e5920b37745814efbe2809886cbf2b76f655e59eb094e5f9add55dc"}
Oct 30 00:46:31.692300 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:31.692225    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-56pr2" event={"ID":"71e604b6-8548-4bab-823d-b80c3f51c0da","Type":"ContainerStarted","Data":"069181246f255aeffa41da77593d16c2ae823a6ef645d512d5921f1f91e68cd0"}
Oct 30 00:46:31.733785 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:31.733740    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-56pr2" podStartSLOduration=1.863964952 podStartE2EDuration="4.733728595s" totalImagesPullingTime="2.869763643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:31.73332708 +0000 UTC m=+300.482982385" watchObservedRunningTime="2026-10-30 00:46:31.733728595 +0000 UTC m=+300.483383898"
Oct 30 00:46:33.659614 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:33.659583    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f7f9fcc75-cskfq"
Oct 30 00:46:34.955776 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:34.955746    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-dbkj2"]
Oct 30 00:46:34.961076 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:34.961051    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dbkj2"
Oct 30 00:46:34.978847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:34.978826    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 00:46:34.988680 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:34.988649    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 00:46:34.988821 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:34.988710    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 00:46:34.988821 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:34.988716    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 00:46:34.988821 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:34.988738    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5d5sd\""
Oct 30 00:46:34.989984 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:34.989963    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-dbkj2"]
Oct 30 00:46:35.050304 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:35.050274    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/95a0d310-2023-40af-bf5e-d3e30e739590-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-dbkj2\" (UID: \"95a0d310-2023-40af-bf5e-d3e30e739590\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dbkj2"
Oct 30 00:46:35.050405 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:35.050323    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvvzj\" (UniqueName: \"kubernetes.io/projected/95a0d310-2023-40af-bf5e-d3e30e739590-kube-api-access-qvvzj\") pod \"kuadrant-console-plugin-66c4fb575d-dbkj2\" (UID: \"95a0d310-2023-40af-bf5e-d3e30e739590\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dbkj2"
Oct 30 00:46:35.050405 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:35.050345    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/95a0d310-2023-40af-bf5e-d3e30e739590-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-dbkj2\" (UID: \"95a0d310-2023-40af-bf5e-d3e30e739590\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dbkj2"
Oct 30 00:46:35.151025 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:35.150996    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/95a0d310-2023-40af-bf5e-d3e30e739590-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-dbkj2\" (UID: \"95a0d310-2023-40af-bf5e-d3e30e739590\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dbkj2"
Oct 30 00:46:35.151187 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:35.151057    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvzj\" (UniqueName: \"kubernetes.io/projected/95a0d310-2023-40af-bf5e-d3e30e739590-kube-api-access-qvvzj\") pod \"kuadrant-console-plugin-66c4fb575d-dbkj2\" (UID: \"95a0d310-2023-40af-bf5e-d3e30e739590\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dbkj2"
Oct 30 00:46:35.151187 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:35.151091    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/95a0d310-2023-40af-bf5e-d3e30e739590-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-dbkj2\" (UID: \"95a0d310-2023-40af-bf5e-d3e30e739590\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dbkj2"
Oct 30 00:46:35.151668 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:35.151644    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/95a0d310-2023-40af-bf5e-d3e30e739590-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-dbkj2\" (UID: \"95a0d310-2023-40af-bf5e-d3e30e739590\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dbkj2"
Oct 30 00:46:35.153483 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:35.153462    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/95a0d310-2023-40af-bf5e-d3e30e739590-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-dbkj2\" (UID: \"95a0d310-2023-40af-bf5e-d3e30e739590\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dbkj2"
Oct 30 00:46:35.175467 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:35.175440    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvvzj\" (UniqueName: \"kubernetes.io/projected/95a0d310-2023-40af-bf5e-d3e30e739590-kube-api-access-qvvzj\") pod \"kuadrant-console-plugin-66c4fb575d-dbkj2\" (UID: \"95a0d310-2023-40af-bf5e-d3e30e739590\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dbkj2"
Oct 30 00:46:35.271209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:35.271185    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dbkj2"
Oct 30 00:46:35.563758 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:35.563695    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-dbkj2"]
Oct 30 00:46:35.566847 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:46:35.566824    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95a0d310_2023_40af_bf5e_d3e30e739590.slice/crio-28ac30e699e93a168923ae95f9e380e6449f844b361481a02d753b3cb0b64a6b WatchSource:0}: Error finding container 28ac30e699e93a168923ae95f9e380e6449f844b361481a02d753b3cb0b64a6b: Status 404 returned error can't find the container with id 28ac30e699e93a168923ae95f9e380e6449f844b361481a02d753b3cb0b64a6b
Oct 30 00:46:35.568151 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:35.568137    4425 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:46:35.708720 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:35.708698    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dbkj2" event={"ID":"95a0d310-2023-40af-bf5e-d3e30e739590","Type":"ContainerStarted","Data":"28ac30e699e93a168923ae95f9e380e6449f844b361481a02d753b3cb0b64a6b"}
Oct 30 00:46:37.556645 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:46:37.556602    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:46:39.644853 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:39.644815    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-lzvsj"]
Oct 30 00:46:39.653148 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:39.653125    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lzvsj"
Oct 30 00:46:39.679049 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:39.679009    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 00:46:39.679049 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:39.679028    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 00:46:39.680316 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:39.680294    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-lzvsj"]
Oct 30 00:46:39.749530 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:39.749500    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-bkpv2\""
Oct 30 00:46:39.793619 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:39.793582    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22950bbb-30b5-46a1-955e-d8a087806fbd-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-lzvsj\" (UID: \"22950bbb-30b5-46a1-955e-d8a087806fbd\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lzvsj"
Oct 30 00:46:39.793813 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:39.793643    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2wkq\" (UniqueName: \"kubernetes.io/projected/22950bbb-30b5-46a1-955e-d8a087806fbd-kube-api-access-c2wkq\") pod \"cert-manager-webhook-5dbbbf75cc-lzvsj\" (UID: \"22950bbb-30b5-46a1-955e-d8a087806fbd\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lzvsj"
Oct 30 00:46:39.895185 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:39.895097    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22950bbb-30b5-46a1-955e-d8a087806fbd-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-lzvsj\" (UID: \"22950bbb-30b5-46a1-955e-d8a087806fbd\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lzvsj"
Oct 30 00:46:39.895185 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:39.895144    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wkq\" (UniqueName: \"kubernetes.io/projected/22950bbb-30b5-46a1-955e-d8a087806fbd-kube-api-access-c2wkq\") pod \"cert-manager-webhook-5dbbbf75cc-lzvsj\" (UID: \"22950bbb-30b5-46a1-955e-d8a087806fbd\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lzvsj"
Oct 30 00:46:39.917519 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:39.917495    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22950bbb-30b5-46a1-955e-d8a087806fbd-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-lzvsj\" (UID: \"22950bbb-30b5-46a1-955e-d8a087806fbd\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lzvsj"
Oct 30 00:46:39.917729 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:39.917709    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2wkq\" (UniqueName: \"kubernetes.io/projected/22950bbb-30b5-46a1-955e-d8a087806fbd-kube-api-access-c2wkq\") pod \"cert-manager-webhook-5dbbbf75cc-lzvsj\" (UID: \"22950bbb-30b5-46a1-955e-d8a087806fbd\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lzvsj"
Oct 30 00:46:39.968437 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:39.968413    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lzvsj"
Oct 30 00:46:40.265142 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:40.265114    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-lzvsj"]
Oct 30 00:46:40.268456 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:46:40.268430    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22950bbb_30b5_46a1_955e_d8a087806fbd.slice/crio-b73d929a85f3959b70c70730662274937b953ed7747b10ef18258a2b2ed99298 WatchSource:0}: Error finding container b73d929a85f3959b70c70730662274937b953ed7747b10ef18258a2b2ed99298: Status 404 returned error can't find the container with id b73d929a85f3959b70c70730662274937b953ed7747b10ef18258a2b2ed99298
Oct 30 00:46:40.735378 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:40.735292    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dbkj2" event={"ID":"95a0d310-2023-40af-bf5e-d3e30e739590","Type":"ContainerStarted","Data":"d24abab0c4dfbb93cbb8e27db43ab6d8d13b94a19a5fcf6a2578c850e84e62c4"}
Oct 30 00:46:40.736561 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:40.736536    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lzvsj" event={"ID":"22950bbb-30b5-46a1-955e-d8a087806fbd","Type":"ContainerStarted","Data":"b73d929a85f3959b70c70730662274937b953ed7747b10ef18258a2b2ed99298"}
Oct 30 00:46:40.797956 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:40.797905    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-dbkj2" podStartSLOduration=2.403369683 podStartE2EDuration="6.797888512s" totalImagesPullingTime="4.394518829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:40.797068775 +0000 UTC m=+309.546724086" watchObservedRunningTime="2026-10-30 00:46:40.797888512 +0000 UTC m=+309.547543821"
Oct 30 00:46:43.752405 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:43.752374    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lzvsj" event={"ID":"22950bbb-30b5-46a1-955e-d8a087806fbd","Type":"ContainerStarted","Data":"d5c232121f4eae4c9912c5157c4fb206b434e68a9de7be98a1205718acea3b79"}
Oct 30 00:46:43.752773 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:43.752505    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lzvsj"
Oct 30 00:46:43.789172 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:43.789110    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lzvsj" podStartSLOduration=1.863139875 podStartE2EDuration="4.789098057s" totalImagesPullingTime="2.925958182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:43.788932903 +0000 UTC m=+312.538588217" watchObservedRunningTime="2026-10-30 00:46:43.789098057 +0000 UTC m=+312.538753360"
Oct 30 00:46:46.509477 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:46.509398    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-l9glj"]
Oct 30 00:46:46.513016 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:46.512996    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l9glj"
Oct 30 00:46:46.529866 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:46.529846    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-jblbn\""
Oct 30 00:46:46.540918 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:46.540893    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-l9glj"]
Oct 30 00:46:46.654110 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:46.654086    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svkhj\" (UniqueName: \"kubernetes.io/projected/1dc9357b-4f64-4b1d-b12d-11ad40cd9384-kube-api-access-svkhj\") pod \"cert-manager-cainjector-5c57bbd4fc-l9glj\" (UID: \"1dc9357b-4f64-4b1d-b12d-11ad40cd9384\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l9glj"
Oct 30 00:46:46.654235 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:46.654131    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1dc9357b-4f64-4b1d-b12d-11ad40cd9384-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-l9glj\" (UID: \"1dc9357b-4f64-4b1d-b12d-11ad40cd9384\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l9glj"
Oct 30 00:46:46.755478 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:46.755445    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svkhj\" (UniqueName: \"kubernetes.io/projected/1dc9357b-4f64-4b1d-b12d-11ad40cd9384-kube-api-access-svkhj\") pod \"cert-manager-cainjector-5c57bbd4fc-l9glj\" (UID: \"1dc9357b-4f64-4b1d-b12d-11ad40cd9384\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l9glj"
Oct 30 00:46:46.755588 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:46.755521    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1dc9357b-4f64-4b1d-b12d-11ad40cd9384-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-l9glj\" (UID: \"1dc9357b-4f64-4b1d-b12d-11ad40cd9384\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l9glj"
Oct 30 00:46:46.784724 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:46.784664    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svkhj\" (UniqueName: \"kubernetes.io/projected/1dc9357b-4f64-4b1d-b12d-11ad40cd9384-kube-api-access-svkhj\") pod \"cert-manager-cainjector-5c57bbd4fc-l9glj\" (UID: \"1dc9357b-4f64-4b1d-b12d-11ad40cd9384\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l9glj"
Oct 30 00:46:46.785042 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:46.785025    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1dc9357b-4f64-4b1d-b12d-11ad40cd9384-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-l9glj\" (UID: \"1dc9357b-4f64-4b1d-b12d-11ad40cd9384\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l9glj"
Oct 30 00:46:46.823537 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:46.823516    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l9glj"
Oct 30 00:46:47.147822 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.147794    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-l9glj"]
Oct 30 00:46:47.149615 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:46:47.149588    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dc9357b_4f64_4b1d_b12d_11ad40cd9384.slice/crio-6e10f9d5c4d6d0fd41fccc55b92cb0b73dc3b2511a059b195b027ab19085fcc5 WatchSource:0}: Error finding container 6e10f9d5c4d6d0fd41fccc55b92cb0b73dc3b2511a059b195b027ab19085fcc5: Status 404 returned error can't find the container with id 6e10f9d5c4d6d0fd41fccc55b92cb0b73dc3b2511a059b195b027ab19085fcc5
Oct 30 00:46:47.597811 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.597560    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-57cd69bbcd-k5fn7"]
Oct 30 00:46:47.605409 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.605377    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-k5fn7"
Oct 30 00:46:47.624932 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.624906    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57cd69bbcd-k5fn7"]
Oct 30 00:46:47.632911 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.632885    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 00:46:47.634083 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.634062    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 00:46:47.702936 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.702874    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 00:46:47.703131 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.703043    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 00:46:47.703303 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.703283    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-t6svh\""
Oct 30 00:46:47.766685 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.766650    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae116534-9fdc-4ce1-8385-a2c7b0738d88-tls-certs\") pod \"kube-auth-proxy-57cd69bbcd-k5fn7\" (UID: \"ae116534-9fdc-4ce1-8385-a2c7b0738d88\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-k5fn7"
Oct 30 00:46:47.766849 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.766709    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9zcm\" (UniqueName: \"kubernetes.io/projected/ae116534-9fdc-4ce1-8385-a2c7b0738d88-kube-api-access-j9zcm\") pod \"kube-auth-proxy-57cd69bbcd-k5fn7\" (UID: \"ae116534-9fdc-4ce1-8385-a2c7b0738d88\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-k5fn7"
Oct 30 00:46:47.766849 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.766834    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae116534-9fdc-4ce1-8385-a2c7b0738d88-tmp\") pod \"kube-auth-proxy-57cd69bbcd-k5fn7\" (UID: \"ae116534-9fdc-4ce1-8385-a2c7b0738d88\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-k5fn7"
Oct 30 00:46:47.773651 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.773623    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l9glj" event={"ID":"1dc9357b-4f64-4b1d-b12d-11ad40cd9384","Type":"ContainerStarted","Data":"34224d654d525a4a364a23f772b8a2730c477a1329806c3f7e6702fc89cfefdd"}
Oct 30 00:46:47.773804 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.773663    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l9glj" event={"ID":"1dc9357b-4f64-4b1d-b12d-11ad40cd9384","Type":"ContainerStarted","Data":"6e10f9d5c4d6d0fd41fccc55b92cb0b73dc3b2511a059b195b027ab19085fcc5"}
Oct 30 00:46:47.812839 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.812786    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-l9glj" podStartSLOduration=1.812769882 podStartE2EDuration="1.812769882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:47.812063776 +0000 UTC m=+316.561719087" watchObservedRunningTime="2026-10-30 00:46:47.812769882 +0000 UTC m=+316.562425194"
Oct 30 00:46:47.867741 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.867662    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae116534-9fdc-4ce1-8385-a2c7b0738d88-tmp\") pod \"kube-auth-proxy-57cd69bbcd-k5fn7\" (UID: \"ae116534-9fdc-4ce1-8385-a2c7b0738d88\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-k5fn7"
Oct 30 00:46:47.867932 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.867913    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae116534-9fdc-4ce1-8385-a2c7b0738d88-tls-certs\") pod \"kube-auth-proxy-57cd69bbcd-k5fn7\" (UID: \"ae116534-9fdc-4ce1-8385-a2c7b0738d88\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-k5fn7"
Oct 30 00:46:47.868046 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.868029    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9zcm\" (UniqueName: \"kubernetes.io/projected/ae116534-9fdc-4ce1-8385-a2c7b0738d88-kube-api-access-j9zcm\") pod \"kube-auth-proxy-57cd69bbcd-k5fn7\" (UID: \"ae116534-9fdc-4ce1-8385-a2c7b0738d88\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-k5fn7"
Oct 30 00:46:47.869980 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.869955    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae116534-9fdc-4ce1-8385-a2c7b0738d88-tmp\") pod \"kube-auth-proxy-57cd69bbcd-k5fn7\" (UID: \"ae116534-9fdc-4ce1-8385-a2c7b0738d88\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-k5fn7"
Oct 30 00:46:47.870672 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.870652    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae116534-9fdc-4ce1-8385-a2c7b0738d88-tls-certs\") pod \"kube-auth-proxy-57cd69bbcd-k5fn7\" (UID: \"ae116534-9fdc-4ce1-8385-a2c7b0738d88\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-k5fn7"
Oct 30 00:46:47.891783 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.891744    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9zcm\" (UniqueName: \"kubernetes.io/projected/ae116534-9fdc-4ce1-8385-a2c7b0738d88-kube-api-access-j9zcm\") pod \"kube-auth-proxy-57cd69bbcd-k5fn7\" (UID: \"ae116534-9fdc-4ce1-8385-a2c7b0738d88\") " pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-k5fn7"
Oct 30 00:46:47.922443 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:47.922419    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-k5fn7"
Oct 30 00:46:48.217098 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:48.217058    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57cd69bbcd-k5fn7"]
Oct 30 00:46:48.219848 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:46:48.219823    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae116534_9fdc_4ce1_8385_a2c7b0738d88.slice/crio-57935ddcf0730ea3474168ff012e9c52b3ae300a0c06795b517e39eaef26f14b WatchSource:0}: Error finding container 57935ddcf0730ea3474168ff012e9c52b3ae300a0c06795b517e39eaef26f14b: Status 404 returned error can't find the container with id 57935ddcf0730ea3474168ff012e9c52b3ae300a0c06795b517e39eaef26f14b
Oct 30 00:46:48.785498 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:48.785430    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-k5fn7" event={"ID":"ae116534-9fdc-4ce1-8385-a2c7b0738d88","Type":"ContainerStarted","Data":"57935ddcf0730ea3474168ff012e9c52b3ae300a0c06795b517e39eaef26f14b"}
Oct 30 00:46:48.972372 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:48.972338    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-5vf87"]
Oct 30 00:46:48.979182 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:48.975980    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-5vf87"
Oct 30 00:46:49.000642 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:49.000576    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gmqk6\""
Oct 30 00:46:49.002716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:49.002690    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-5vf87"]
Oct 30 00:46:49.080302 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:49.080225    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flxzg\" (UniqueName: \"kubernetes.io/projected/2be25bb0-9095-4664-9ff2-c4a1f1daa5b7-kube-api-access-flxzg\") pod \"cert-manager-597b95cb84-5vf87\" (UID: \"2be25bb0-9095-4664-9ff2-c4a1f1daa5b7\") " pod="cert-manager/cert-manager-597b95cb84-5vf87"
Oct 30 00:46:49.080452 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:49.080340    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2be25bb0-9095-4664-9ff2-c4a1f1daa5b7-bound-sa-token\") pod \"cert-manager-597b95cb84-5vf87\" (UID: \"2be25bb0-9095-4664-9ff2-c4a1f1daa5b7\") " pod="cert-manager/cert-manager-597b95cb84-5vf87"
Oct 30 00:46:49.181854 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:49.181821    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flxzg\" (UniqueName: \"kubernetes.io/projected/2be25bb0-9095-4664-9ff2-c4a1f1daa5b7-kube-api-access-flxzg\") pod \"cert-manager-597b95cb84-5vf87\" (UID: \"2be25bb0-9095-4664-9ff2-c4a1f1daa5b7\") " pod="cert-manager/cert-manager-597b95cb84-5vf87"
Oct 30 00:46:49.182017 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:49.181927    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2be25bb0-9095-4664-9ff2-c4a1f1daa5b7-bound-sa-token\") pod \"cert-manager-597b95cb84-5vf87\" (UID: \"2be25bb0-9095-4664-9ff2-c4a1f1daa5b7\") " pod="cert-manager/cert-manager-597b95cb84-5vf87"
Oct 30 00:46:49.205849 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:49.205822    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flxzg\" (UniqueName: \"kubernetes.io/projected/2be25bb0-9095-4664-9ff2-c4a1f1daa5b7-kube-api-access-flxzg\") pod \"cert-manager-597b95cb84-5vf87\" (UID: \"2be25bb0-9095-4664-9ff2-c4a1f1daa5b7\") " pod="cert-manager/cert-manager-597b95cb84-5vf87"
Oct 30 00:46:49.206876 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:49.206851    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2be25bb0-9095-4664-9ff2-c4a1f1daa5b7-bound-sa-token\") pod \"cert-manager-597b95cb84-5vf87\" (UID: \"2be25bb0-9095-4664-9ff2-c4a1f1daa5b7\") " pod="cert-manager/cert-manager-597b95cb84-5vf87"
Oct 30 00:46:49.293308 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:49.293276    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-5vf87"
Oct 30 00:46:49.610109 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:49.610065    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-5vf87"]
Oct 30 00:46:49.613880 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:46:49.613808    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2be25bb0_9095_4664_9ff2_c4a1f1daa5b7.slice/crio-9643d907e8f1f51512bfcc59e2389f4aa5ffe93cacf7284e4479389b399bcd32 WatchSource:0}: Error finding container 9643d907e8f1f51512bfcc59e2389f4aa5ffe93cacf7284e4479389b399bcd32: Status 404 returned error can't find the container with id 9643d907e8f1f51512bfcc59e2389f4aa5ffe93cacf7284e4479389b399bcd32
Oct 30 00:46:49.759610 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:49.758967    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-lzvsj"
Oct 30 00:46:49.793763 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:49.793718    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-5vf87" event={"ID":"2be25bb0-9095-4664-9ff2-c4a1f1daa5b7","Type":"ContainerStarted","Data":"98f53b90c4974bff53f16ccd19f5ed97d435fbc8aaa35d9a352fdcf64621400a"}
Oct 30 00:46:49.794126 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:49.793769    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-5vf87" event={"ID":"2be25bb0-9095-4664-9ff2-c4a1f1daa5b7","Type":"ContainerStarted","Data":"9643d907e8f1f51512bfcc59e2389f4aa5ffe93cacf7284e4479389b399bcd32"}
Oct 30 00:46:50.800959 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:50.800872    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-k5fn7" event={"ID":"ae116534-9fdc-4ce1-8385-a2c7b0738d88","Type":"ContainerStarted","Data":"eb2634e9e58210cc03e8fc75fd51b4da0b9d6aed20344e9dacf67e311694108f"}
Oct 30 00:46:50.839093 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:50.839045    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-5vf87" podStartSLOduration=2.839031795 podStartE2EDuration="2.839031795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:49.837402336 +0000 UTC m=+318.587057647" watchObservedRunningTime="2026-10-30 00:46:50.839031795 +0000 UTC m=+319.588687104"
Oct 30 00:46:50.839413 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:46:50.839377    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-57cd69bbcd-k5fn7" podStartSLOduration=1.512527218 podStartE2EDuration="3.839364323s" totalImagesPullingTime="2.326837105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:50.837696961 +0000 UTC m=+319.587352270" watchObservedRunningTime="2026-10-30 00:46:50.839364323 +0000 UTC m=+319.589019633"
Oct 30 00:47:07.553130 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:47:07.553095    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:47:18.261330 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.261297    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-l5n4t"]
Oct 30 00:47:18.264294 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.264273    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:47:18.264428 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.264391    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-l5n4t"
Oct 30 00:47:18.267131 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.267113    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"
Oct 30 00:47:18.284323 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.284305    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-l5n4t"]
Oct 30 00:47:18.292919 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.292899    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2xt6n\""
Oct 30 00:47:18.293015 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.292945    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 00:47:18.295052 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.295030    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:47:18.318568 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.318545    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4trg\" (UniqueName: \"kubernetes.io/projected/290587aa-9503-4d06-9bad-33d5d7771c55-kube-api-access-z4trg\") pod \"authorino-56c6c9ddd5-l5n4t\" (UID: \"290587aa-9503-4d06-9bad-33d5d7771c55\") " pod="kuadrant-system/authorino-56c6c9ddd5-l5n4t"
Oct 30 00:47:18.356396 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.356373    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:47:18.419475 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.419452    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7377e3da-0408-4a67-bfe1-f4e481cb09a8-config-file\") pod \"limitador-limitador-7c9b87d574-gkbtv\" (UID: \"7377e3da-0408-4a67-bfe1-f4e481cb09a8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"
Oct 30 00:47:18.419566 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.419497    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws4bt\" (UniqueName: \"kubernetes.io/projected/7377e3da-0408-4a67-bfe1-f4e481cb09a8-kube-api-access-ws4bt\") pod \"limitador-limitador-7c9b87d574-gkbtv\" (UID: \"7377e3da-0408-4a67-bfe1-f4e481cb09a8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"
Oct 30 00:47:18.419633 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.419619    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4trg\" (UniqueName: \"kubernetes.io/projected/290587aa-9503-4d06-9bad-33d5d7771c55-kube-api-access-z4trg\") pod \"authorino-56c6c9ddd5-l5n4t\" (UID: \"290587aa-9503-4d06-9bad-33d5d7771c55\") " pod="kuadrant-system/authorino-56c6c9ddd5-l5n4t"
Oct 30 00:47:18.440021 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.439998    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4trg\" (UniqueName: \"kubernetes.io/projected/290587aa-9503-4d06-9bad-33d5d7771c55-kube-api-access-z4trg\") pod \"authorino-56c6c9ddd5-l5n4t\" (UID: \"290587aa-9503-4d06-9bad-33d5d7771c55\") " pod="kuadrant-system/authorino-56c6c9ddd5-l5n4t"
Oct 30 00:47:18.520397 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.520350    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7377e3da-0408-4a67-bfe1-f4e481cb09a8-config-file\") pod \"limitador-limitador-7c9b87d574-gkbtv\" (UID: \"7377e3da-0408-4a67-bfe1-f4e481cb09a8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"
Oct 30 00:47:18.520397 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.520386    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws4bt\" (UniqueName: \"kubernetes.io/projected/7377e3da-0408-4a67-bfe1-f4e481cb09a8-kube-api-access-ws4bt\") pod \"limitador-limitador-7c9b87d574-gkbtv\" (UID: \"7377e3da-0408-4a67-bfe1-f4e481cb09a8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"
Oct 30 00:47:18.520962 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.520942    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7377e3da-0408-4a67-bfe1-f4e481cb09a8-config-file\") pod \"limitador-limitador-7c9b87d574-gkbtv\" (UID: \"7377e3da-0408-4a67-bfe1-f4e481cb09a8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"
Oct 30 00:47:18.540818 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.540797    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws4bt\" (UniqueName: \"kubernetes.io/projected/7377e3da-0408-4a67-bfe1-f4e481cb09a8-kube-api-access-ws4bt\") pod \"limitador-limitador-7c9b87d574-gkbtv\" (UID: \"7377e3da-0408-4a67-bfe1-f4e481cb09a8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"
Oct 30 00:47:18.577373 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.577354    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-l5n4t"
Oct 30 00:47:18.583013 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.582996    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"
Oct 30 00:47:18.872828 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.872754    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-l5n4t"]
Oct 30 00:47:18.880670 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:47:18.880645    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod290587aa_9503_4d06_9bad_33d5d7771c55.slice/crio-ad84343f67aa177105374be6266fd0b16a5eee56aac635c1cd564bb2363ba7c2 WatchSource:0}: Error finding container ad84343f67aa177105374be6266fd0b16a5eee56aac635c1cd564bb2363ba7c2: Status 404 returned error can't find the container with id ad84343f67aa177105374be6266fd0b16a5eee56aac635c1cd564bb2363ba7c2
Oct 30 00:47:18.885878 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.885858    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:47:18.888792 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:47:18.888771    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7377e3da_0408_4a67_bfe1_f4e481cb09a8.slice/crio-78cc43be637ecf4607fdc7f7af31db9ca86717ef1c9920eb7f220694e8aa5487 WatchSource:0}: Error finding container 78cc43be637ecf4607fdc7f7af31db9ca86717ef1c9920eb7f220694e8aa5487: Status 404 returned error can't find the container with id 78cc43be637ecf4607fdc7f7af31db9ca86717ef1c9920eb7f220694e8aa5487
Oct 30 00:47:18.912708 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.912686    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkbtv" event={"ID":"7377e3da-0408-4a67-bfe1-f4e481cb09a8","Type":"ContainerStarted","Data":"78cc43be637ecf4607fdc7f7af31db9ca86717ef1c9920eb7f220694e8aa5487"}
Oct 30 00:47:18.913842 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:18.913820    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-l5n4t" event={"ID":"290587aa-9503-4d06-9bad-33d5d7771c55","Type":"ContainerStarted","Data":"ad84343f67aa177105374be6266fd0b16a5eee56aac635c1cd564bb2363ba7c2"}
Oct 30 00:47:21.931347 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:21.931312    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkbtv" event={"ID":"7377e3da-0408-4a67-bfe1-f4e481cb09a8","Type":"ContainerStarted","Data":"3e07b97a152077dc9e449ec1f51300a1ad0b8bcb9b7f0b8cf36b3da231915800"}
Oct 30 00:47:21.931727 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:21.931438    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"
Oct 30 00:47:21.932953 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:21.932930    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-l5n4t" event={"ID":"290587aa-9503-4d06-9bad-33d5d7771c55","Type":"ContainerStarted","Data":"718ad7374fa368baf361c4803452adc8f6c39e2d466ab0b9a95d40ec0313a39a"}
Oct 30 00:47:21.972211 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:21.972135    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkbtv" podStartSLOduration=1.033801635 podStartE2EDuration="3.972123553s" totalImagesPullingTime="2.938321918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:21.971957228 +0000 UTC m=+350.721612561" watchObservedRunningTime="2026-10-30 00:47:21.972123553 +0000 UTC m=+350.721778862"
Oct 30 00:47:22.006541 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:22.006500    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-l5n4t" podStartSLOduration=1.368982677 podStartE2EDuration="4.006488727s" totalImagesPullingTime="2.63750605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:22.00544891 +0000 UTC m=+350.755104219" watchObservedRunningTime="2026-10-30 00:47:22.006488727 +0000 UTC m=+350.756144035"
Oct 30 00:47:32.938820 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:32.938792    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"
Oct 30 00:47:37.553299 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:47:37.553262    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:47:51.888074 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:51.888028    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9zbsm"]
Oct 30 00:47:51.929738 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:51.929711    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9zbsm"]
Oct 30 00:47:51.929900 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:51.929817    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-9zbsm"
Oct 30 00:47:51.937824 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:51.937800    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"]
Oct 30 00:47:51.944730 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:51.944707    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 00:47:51.944830 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:51.944753    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 00:47:51.961611 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:51.961593    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"
Oct 30 00:47:51.967896 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:51.967876    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"]
Oct 30 00:47:51.969125 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:51.969105    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 00:47:51.969291 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:51.969106    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-4pzw7\""
Oct 30 00:47:51.979180 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:51.979146    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-5mcgv"]
Oct 30 00:47:51.989468 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:51.989448    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s5tl\" (UniqueName: \"kubernetes.io/projected/26afb620-1e3d-4a2a-8c18-03eab11bd24f-kube-api-access-2s5tl\") pod \"llmisvc-controller-manager-869d5dcc5d-gjn8l\" (UID: \"26afb620-1e3d-4a2a-8c18-03eab11bd24f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"
Oct 30 00:47:51.989561 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:51.989501    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3009c800-6df6-4c1e-935f-3a7e5be62572-cert\") pod \"kserve-controller-manager-776cc66d47-9zbsm\" (UID: \"3009c800-6df6-4c1e-935f-3a7e5be62572\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9zbsm"
Oct 30 00:47:51.989611 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:51.989597    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26afb620-1e3d-4a2a-8c18-03eab11bd24f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-gjn8l\" (UID: \"26afb620-1e3d-4a2a-8c18-03eab11bd24f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"
Oct 30 00:47:51.989647 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:51.989633    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svq8w\" (UniqueName: \"kubernetes.io/projected/3009c800-6df6-4c1e-935f-3a7e5be62572-kube-api-access-svq8w\") pod \"kserve-controller-manager-776cc66d47-9zbsm\" (UID: \"3009c800-6df6-4c1e-935f-3a7e5be62572\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9zbsm"
Oct 30 00:47:51.997483 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:51.997466    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-5mcgv"
Oct 30 00:47:52.006056 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.005315    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-5mcgv"]
Oct 30 00:47:52.022039 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.022017    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-9xl4p"]
Oct 30 00:47:52.038020 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.038002    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-9xl4p"
Oct 30 00:47:52.048144 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.048123    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-9xl4p"]
Oct 30 00:47:52.068951 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.068927    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 00:47:52.069091 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.069017    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-ckx9h\""
Oct 30 00:47:52.069145 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.069135    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-f4tln\""
Oct 30 00:47:52.069214 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.069135    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 00:47:52.069214 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.069183    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-86wrh\""
Oct 30 00:47:52.069292 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.069214    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 00:47:52.069292 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.069251    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 00:47:52.090175 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.090133    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s5tl\" (UniqueName: \"kubernetes.io/projected/26afb620-1e3d-4a2a-8c18-03eab11bd24f-kube-api-access-2s5tl\") pod \"llmisvc-controller-manager-869d5dcc5d-gjn8l\" (UID: \"26afb620-1e3d-4a2a-8c18-03eab11bd24f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"
Oct 30 00:47:52.090311 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.090212    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3009c800-6df6-4c1e-935f-3a7e5be62572-cert\") pod \"kserve-controller-manager-776cc66d47-9zbsm\" (UID: \"3009c800-6df6-4c1e-935f-3a7e5be62572\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9zbsm"
Oct 30 00:47:52.090311 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.090250    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26afb620-1e3d-4a2a-8c18-03eab11bd24f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-gjn8l\" (UID: \"26afb620-1e3d-4a2a-8c18-03eab11bd24f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"
Oct 30 00:47:52.090516 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.090492    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svq8w\" (UniqueName: \"kubernetes.io/projected/3009c800-6df6-4c1e-935f-3a7e5be62572-kube-api-access-svq8w\") pod \"kserve-controller-manager-776cc66d47-9zbsm\" (UID: \"3009c800-6df6-4c1e-935f-3a7e5be62572\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9zbsm"
Oct 30 00:47:52.092784 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.092756    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26afb620-1e3d-4a2a-8c18-03eab11bd24f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-gjn8l\" (UID: \"26afb620-1e3d-4a2a-8c18-03eab11bd24f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"
Oct 30 00:47:52.093015 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.092999    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3009c800-6df6-4c1e-935f-3a7e5be62572-cert\") pod \"kserve-controller-manager-776cc66d47-9zbsm\" (UID: \"3009c800-6df6-4c1e-935f-3a7e5be62572\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9zbsm"
Oct 30 00:47:52.110855 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.110833    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s5tl\" (UniqueName: \"kubernetes.io/projected/26afb620-1e3d-4a2a-8c18-03eab11bd24f-kube-api-access-2s5tl\") pod \"llmisvc-controller-manager-869d5dcc5d-gjn8l\" (UID: \"26afb620-1e3d-4a2a-8c18-03eab11bd24f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"
Oct 30 00:47:52.110933 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.110857    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svq8w\" (UniqueName: \"kubernetes.io/projected/3009c800-6df6-4c1e-935f-3a7e5be62572-kube-api-access-svq8w\") pod \"kserve-controller-manager-776cc66d47-9zbsm\" (UID: \"3009c800-6df6-4c1e-935f-3a7e5be62572\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9zbsm"
Oct 30 00:47:52.191597 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.191524    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b42c9fca-e232-4900-adca-ddd04928e24c-tls-certs\") pod \"model-serving-api-75bd89fdfc-5mcgv\" (UID: \"b42c9fca-e232-4900-adca-ddd04928e24c\") " pod="opendatahub/model-serving-api-75bd89fdfc-5mcgv"
Oct 30 00:47:52.191597 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.191565    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm88s\" (UniqueName: \"kubernetes.io/projected/b42c9fca-e232-4900-adca-ddd04928e24c-kube-api-access-bm88s\") pod \"model-serving-api-75bd89fdfc-5mcgv\" (UID: \"b42c9fca-e232-4900-adca-ddd04928e24c\") " pod="opendatahub/model-serving-api-75bd89fdfc-5mcgv"
Oct 30 00:47:52.191597 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.191581    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33acc2a2-ca7f-4702-be45-c4f12d99d4b7-cert\") pod \"odh-model-controller-769875db56-9xl4p\" (UID: \"33acc2a2-ca7f-4702-be45-c4f12d99d4b7\") " pod="opendatahub/odh-model-controller-769875db56-9xl4p"
Oct 30 00:47:52.191846 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.191630    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2tgq\" (UniqueName: \"kubernetes.io/projected/33acc2a2-ca7f-4702-be45-c4f12d99d4b7-kube-api-access-d2tgq\") pod \"odh-model-controller-769875db56-9xl4p\" (UID: \"33acc2a2-ca7f-4702-be45-c4f12d99d4b7\") " pod="opendatahub/odh-model-controller-769875db56-9xl4p"
Oct 30 00:47:52.191846 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.191737    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33acc2a2-ca7f-4702-be45-c4f12d99d4b7-metrics-certs\") pod \"odh-model-controller-769875db56-9xl4p\" (UID: \"33acc2a2-ca7f-4702-be45-c4f12d99d4b7\") " pod="opendatahub/odh-model-controller-769875db56-9xl4p"
Oct 30 00:47:52.272550 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.272514    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-9zbsm"
Oct 30 00:47:52.292178 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.292133    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"
Oct 30 00:47:52.292378 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.292359    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b42c9fca-e232-4900-adca-ddd04928e24c-tls-certs\") pod \"model-serving-api-75bd89fdfc-5mcgv\" (UID: \"b42c9fca-e232-4900-adca-ddd04928e24c\") " pod="opendatahub/model-serving-api-75bd89fdfc-5mcgv"
Oct 30 00:47:52.292435 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.292404    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm88s\" (UniqueName: \"kubernetes.io/projected/b42c9fca-e232-4900-adca-ddd04928e24c-kube-api-access-bm88s\") pod \"model-serving-api-75bd89fdfc-5mcgv\" (UID: \"b42c9fca-e232-4900-adca-ddd04928e24c\") " pod="opendatahub/model-serving-api-75bd89fdfc-5mcgv"
Oct 30 00:47:52.292435 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.292422    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33acc2a2-ca7f-4702-be45-c4f12d99d4b7-cert\") pod \"odh-model-controller-769875db56-9xl4p\" (UID: \"33acc2a2-ca7f-4702-be45-c4f12d99d4b7\") " pod="opendatahub/odh-model-controller-769875db56-9xl4p"
Oct 30 00:47:52.292537 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.292437    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2tgq\" (UniqueName: \"kubernetes.io/projected/33acc2a2-ca7f-4702-be45-c4f12d99d4b7-kube-api-access-d2tgq\") pod \"odh-model-controller-769875db56-9xl4p\" (UID: \"33acc2a2-ca7f-4702-be45-c4f12d99d4b7\") " pod="opendatahub/odh-model-controller-769875db56-9xl4p"
Oct 30 00:47:52.292537 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.292461    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33acc2a2-ca7f-4702-be45-c4f12d99d4b7-metrics-certs\") pod \"odh-model-controller-769875db56-9xl4p\" (UID: \"33acc2a2-ca7f-4702-be45-c4f12d99d4b7\") " pod="opendatahub/odh-model-controller-769875db56-9xl4p"
Oct 30 00:47:52.295419 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.295396    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b42c9fca-e232-4900-adca-ddd04928e24c-tls-certs\") pod \"model-serving-api-75bd89fdfc-5mcgv\" (UID: \"b42c9fca-e232-4900-adca-ddd04928e24c\") " pod="opendatahub/model-serving-api-75bd89fdfc-5mcgv"
Oct 30 00:47:52.295536 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.295520    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33acc2a2-ca7f-4702-be45-c4f12d99d4b7-cert\") pod \"odh-model-controller-769875db56-9xl4p\" (UID: \"33acc2a2-ca7f-4702-be45-c4f12d99d4b7\") " pod="opendatahub/odh-model-controller-769875db56-9xl4p"
Oct 30 00:47:52.295681 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.295659    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33acc2a2-ca7f-4702-be45-c4f12d99d4b7-metrics-certs\") pod \"odh-model-controller-769875db56-9xl4p\" (UID: \"33acc2a2-ca7f-4702-be45-c4f12d99d4b7\") " pod="opendatahub/odh-model-controller-769875db56-9xl4p"
Oct 30 00:47:52.315673 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.315114    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm88s\" (UniqueName: \"kubernetes.io/projected/b42c9fca-e232-4900-adca-ddd04928e24c-kube-api-access-bm88s\") pod \"model-serving-api-75bd89fdfc-5mcgv\" (UID: \"b42c9fca-e232-4900-adca-ddd04928e24c\") " pod="opendatahub/model-serving-api-75bd89fdfc-5mcgv"
Oct 30 00:47:52.315673 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.315605    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2tgq\" (UniqueName: \"kubernetes.io/projected/33acc2a2-ca7f-4702-be45-c4f12d99d4b7-kube-api-access-d2tgq\") pod \"odh-model-controller-769875db56-9xl4p\" (UID: \"33acc2a2-ca7f-4702-be45-c4f12d99d4b7\") " pod="opendatahub/odh-model-controller-769875db56-9xl4p"
Oct 30 00:47:52.332601 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.332354    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-5mcgv"
Oct 30 00:47:52.375170 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.371463    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-9xl4p"
Oct 30 00:47:52.577866 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.577843    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9zbsm"]
Oct 30 00:47:52.580442 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:47:52.580415    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3009c800_6df6_4c1e_935f_3a7e5be62572.slice/crio-0093908e8ade6c2ad3a203c441389dced0348fae11aad0e43a897b2a725f1021 WatchSource:0}: Error finding container 0093908e8ade6c2ad3a203c441389dced0348fae11aad0e43a897b2a725f1021: Status 404 returned error can't find the container with id 0093908e8ade6c2ad3a203c441389dced0348fae11aad0e43a897b2a725f1021
Oct 30 00:47:52.607723 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.607697    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"]
Oct 30 00:47:52.610368 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:47:52.610344    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod26afb620_1e3d_4a2a_8c18_03eab11bd24f.slice/crio-751e7620c196ad8d2b32b40e773e1c59b473dd2a66201642005721d67129464c WatchSource:0}: Error finding container 751e7620c196ad8d2b32b40e773e1c59b473dd2a66201642005721d67129464c: Status 404 returned error can't find the container with id 751e7620c196ad8d2b32b40e773e1c59b473dd2a66201642005721d67129464c
Oct 30 00:47:52.662076 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.662053    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-5mcgv"]
Oct 30 00:47:52.664019 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:47:52.663990    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42c9fca_e232_4900_adca_ddd04928e24c.slice/crio-35627298034f91947a8a5f1a5765cced93dc76469be7c4b20b6f04e0ea84c674 WatchSource:0}: Error finding container 35627298034f91947a8a5f1a5765cced93dc76469be7c4b20b6f04e0ea84c674: Status 404 returned error can't find the container with id 35627298034f91947a8a5f1a5765cced93dc76469be7c4b20b6f04e0ea84c674
Oct 30 00:47:52.686705 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:52.686684    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-9xl4p"]
Oct 30 00:47:52.688679 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:47:52.688654    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33acc2a2_ca7f_4702_be45_c4f12d99d4b7.slice/crio-1fe115512f6f7ac42ad79253a3c142775064c8f784e46e643b890a1405380b1c WatchSource:0}: Error finding container 1fe115512f6f7ac42ad79253a3c142775064c8f784e46e643b890a1405380b1c: Status 404 returned error can't find the container with id 1fe115512f6f7ac42ad79253a3c142775064c8f784e46e643b890a1405380b1c
Oct 30 00:47:53.065478 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:53.065441    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-9xl4p" event={"ID":"33acc2a2-ca7f-4702-be45-c4f12d99d4b7","Type":"ContainerStarted","Data":"1fe115512f6f7ac42ad79253a3c142775064c8f784e46e643b890a1405380b1c"}
Oct 30 00:47:53.066945 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:53.066915    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-9zbsm" event={"ID":"3009c800-6df6-4c1e-935f-3a7e5be62572","Type":"ContainerStarted","Data":"0093908e8ade6c2ad3a203c441389dced0348fae11aad0e43a897b2a725f1021"}
Oct 30 00:47:53.068243 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:53.068203    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-5mcgv" event={"ID":"b42c9fca-e232-4900-adca-ddd04928e24c","Type":"ContainerStarted","Data":"35627298034f91947a8a5f1a5765cced93dc76469be7c4b20b6f04e0ea84c674"}
Oct 30 00:47:53.069536 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:53.069510    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l" event={"ID":"26afb620-1e3d-4a2a-8c18-03eab11bd24f","Type":"ContainerStarted","Data":"751e7620c196ad8d2b32b40e773e1c59b473dd2a66201642005721d67129464c"}
Oct 30 00:47:56.093100 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.093050    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-9zbsm" event={"ID":"3009c800-6df6-4c1e-935f-3a7e5be62572","Type":"ContainerStarted","Data":"612bb2b2c993703bc7fd8d595495744817f1b3b544b13383f75242a8709073a0"}
Oct 30 00:47:56.093612 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.093182    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-9zbsm"
Oct 30 00:47:56.094919 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.094887    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-5mcgv" event={"ID":"b42c9fca-e232-4900-adca-ddd04928e24c","Type":"ContainerStarted","Data":"9391158999b44c70c1ec3bb59357aaf164bf2107e30da03490cd9f71dedfe490"}
Oct 30 00:47:56.095059 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.094944    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-5mcgv"
Oct 30 00:47:56.096703 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.096680    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l" event={"ID":"26afb620-1e3d-4a2a-8c18-03eab11bd24f","Type":"ContainerStarted","Data":"f20472235a877539c112e1a480d3ac5ee9b4aa3ed7a7aebe13d078062a07d40a"}
Oct 30 00:47:56.096824 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.096727    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"
Oct 30 00:47:56.116800 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.116778    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6496db66cc-nzwpd"]
Oct 30 00:47:56.121395 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.121373    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6496db66cc-nzwpd"
Oct 30 00:47:56.126030 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.124916    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/858deb1c-4c93-42c2-9fc9-8a53b27fa19b-webhook-cert\") pod \"maas-controller-6496db66cc-nzwpd\" (UID: \"858deb1c-4c93-42c2-9fc9-8a53b27fa19b\") " pod="opendatahub/maas-controller-6496db66cc-nzwpd"
Oct 30 00:47:56.126030 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.124967    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/858deb1c-4c93-42c2-9fc9-8a53b27fa19b-metrics-certs\") pod \"maas-controller-6496db66cc-nzwpd\" (UID: \"858deb1c-4c93-42c2-9fc9-8a53b27fa19b\") " pod="opendatahub/maas-controller-6496db66cc-nzwpd"
Oct 30 00:47:56.126030 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.125067    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj9lt\" (UniqueName: \"kubernetes.io/projected/858deb1c-4c93-42c2-9fc9-8a53b27fa19b-kube-api-access-bj9lt\") pod \"maas-controller-6496db66cc-nzwpd\" (UID: \"858deb1c-4c93-42c2-9fc9-8a53b27fa19b\") " pod="opendatahub/maas-controller-6496db66cc-nzwpd"
Oct 30 00:47:56.130460 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.129218    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-9zbsm" podStartSLOduration=2.107283144 podStartE2EDuration="5.129205078s" totalImagesPullingTime="3.021921934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:56.12837613 +0000 UTC m=+384.878031440" watchObservedRunningTime="2026-10-30 00:47:56.129205078 +0000 UTC m=+384.878860390"
Oct 30 00:47:56.135997 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.135975    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 00:47:56.138057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.138035    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gzqd4\""
Oct 30 00:47:56.138186 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.138101    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 00:47:56.138261 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.138250    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 00:47:56.142392 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.142309    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6496db66cc-nzwpd"]
Oct 30 00:47:56.200052 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.199992    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l" podStartSLOduration=2.198403982 podStartE2EDuration="5.199978089s" totalImagesPullingTime="3.001574107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:56.198802186 +0000 UTC m=+384.948457496" watchObservedRunningTime="2026-10-30 00:47:56.199978089 +0000 UTC m=+384.949633398"
Oct 30 00:47:56.226027 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.225995    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj9lt\" (UniqueName: \"kubernetes.io/projected/858deb1c-4c93-42c2-9fc9-8a53b27fa19b-kube-api-access-bj9lt\") pod \"maas-controller-6496db66cc-nzwpd\" (UID: \"858deb1c-4c93-42c2-9fc9-8a53b27fa19b\") " pod="opendatahub/maas-controller-6496db66cc-nzwpd"
Oct 30 00:47:56.226222 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.226070    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/858deb1c-4c93-42c2-9fc9-8a53b27fa19b-webhook-cert\") pod \"maas-controller-6496db66cc-nzwpd\" (UID: \"858deb1c-4c93-42c2-9fc9-8a53b27fa19b\") " pod="opendatahub/maas-controller-6496db66cc-nzwpd"
Oct 30 00:47:56.226222 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.226100    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/858deb1c-4c93-42c2-9fc9-8a53b27fa19b-metrics-certs\") pod \"maas-controller-6496db66cc-nzwpd\" (UID: \"858deb1c-4c93-42c2-9fc9-8a53b27fa19b\") " pod="opendatahub/maas-controller-6496db66cc-nzwpd"
Oct 30 00:47:56.228502 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.228473    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/858deb1c-4c93-42c2-9fc9-8a53b27fa19b-webhook-cert\") pod \"maas-controller-6496db66cc-nzwpd\" (UID: \"858deb1c-4c93-42c2-9fc9-8a53b27fa19b\") " pod="opendatahub/maas-controller-6496db66cc-nzwpd"
Oct 30 00:47:56.228608 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.228513    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/858deb1c-4c93-42c2-9fc9-8a53b27fa19b-metrics-certs\") pod \"maas-controller-6496db66cc-nzwpd\" (UID: \"858deb1c-4c93-42c2-9fc9-8a53b27fa19b\") " pod="opendatahub/maas-controller-6496db66cc-nzwpd"
Oct 30 00:47:56.239749 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.239704    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-5mcgv" podStartSLOduration=2.690363823 podStartE2EDuration="5.239691209s" totalImagesPullingTime="2.549327386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:56.238343123 +0000 UTC m=+384.987998432" watchObservedRunningTime="2026-10-30 00:47:56.239691209 +0000 UTC m=+384.989346517"
Oct 30 00:47:56.248017 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.247995    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj9lt\" (UniqueName: \"kubernetes.io/projected/858deb1c-4c93-42c2-9fc9-8a53b27fa19b-kube-api-access-bj9lt\") pod \"maas-controller-6496db66cc-nzwpd\" (UID: \"858deb1c-4c93-42c2-9fc9-8a53b27fa19b\") " pod="opendatahub/maas-controller-6496db66cc-nzwpd"
Oct 30 00:47:56.437983 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.437903    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6496db66cc-nzwpd"
Oct 30 00:47:56.749426 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:56.749335    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6496db66cc-nzwpd"]
Oct 30 00:47:56.752490 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:47:56.752459    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858deb1c_4c93_42c2_9fc9_8a53b27fa19b.slice/crio-ef3abbbf248d1b2c85aa692bf89f48e890c0e9a519110ffc32ecc50c19dbb68f WatchSource:0}: Error finding container ef3abbbf248d1b2c85aa692bf89f48e890c0e9a519110ffc32ecc50c19dbb68f: Status 404 returned error can't find the container with id ef3abbbf248d1b2c85aa692bf89f48e890c0e9a519110ffc32ecc50c19dbb68f
Oct 30 00:47:57.103678 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:57.103639    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-9xl4p" event={"ID":"33acc2a2-ca7f-4702-be45-c4f12d99d4b7","Type":"ContainerStarted","Data":"c703e3aaff91f9f29039575e3383f2f1eab43696463dd269a3669c63d526e1b7"}
Oct 30 00:47:57.104122 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:57.103708    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-9xl4p"
Oct 30 00:47:57.104898 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:57.104875    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6496db66cc-nzwpd" event={"ID":"858deb1c-4c93-42c2-9fc9-8a53b27fa19b","Type":"ContainerStarted","Data":"ef3abbbf248d1b2c85aa692bf89f48e890c0e9a519110ffc32ecc50c19dbb68f"}
Oct 30 00:47:57.141653 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:47:57.141597    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-9xl4p" podStartSLOduration=1.543910029 podStartE2EDuration="5.141572231s" totalImagesPullingTime="3.597662202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:57.139114852 +0000 UTC m=+385.888770161" watchObservedRunningTime="2026-10-30 00:47:57.141572231 +0000 UTC m=+385.891227541"
Oct 30 00:48:00.125129 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:00.125096    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6496db66cc-nzwpd" event={"ID":"858deb1c-4c93-42c2-9fc9-8a53b27fa19b","Type":"ContainerStarted","Data":"1d3fdbf3176b6f01b6711ca147221aac319db9124a8ffec740a8ccd6ed01d2c7"}
Oct 30 00:48:00.125516 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:00.125240    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6496db66cc-nzwpd"
Oct 30 00:48:00.201407 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:00.201359    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6496db66cc-nzwpd" podStartSLOduration=1.8599371919999999 podStartE2EDuration="4.201347873s" totalImagesPullingTime="2.341410681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:00.200069322 +0000 UTC m=+388.949724630" watchObservedRunningTime="2026-10-30 00:48:00.201347873 +0000 UTC m=+388.951003182"
Oct 30 00:48:07.109511 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:07.109479    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-5mcgv"
Oct 30 00:48:07.553115 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:48:07.553072    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:48:08.115853 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:08.115824    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-9xl4p"
Oct 30 00:48:11.131868 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:11.131839    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6496db66cc-nzwpd"
Oct 30 00:48:27.107587 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:27.107556    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"
Oct 30 00:48:27.110725 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:27.110704    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-9zbsm"
Oct 30 00:48:33.950713 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:33.950627    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:48:37.552893 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:48:37.552856    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:48:42.338961 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:42.338930    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7b8nh"]
Oct 30 00:48:42.346412 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:42.346390    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7b8nh"
Oct 30 00:48:42.370581 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:42.365640    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7b8nh"]
Oct 30 00:48:42.515294 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:42.515254    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f28b4\" (UniqueName: \"kubernetes.io/projected/10d7885b-ffc3-42b7-a277-cece076c1fd0-kube-api-access-f28b4\") pod \"authorino-6f6944bcdd-7b8nh\" (UID: \"10d7885b-ffc3-42b7-a277-cece076c1fd0\") " pod="kuadrant-system/authorino-6f6944bcdd-7b8nh"
Oct 30 00:48:42.556525 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:42.556490    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7b8nh"]
Oct 30 00:48:42.556715 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:48:42.556696    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-f28b4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-7b8nh" podUID="10d7885b-ffc3-42b7-a277-cece076c1fd0"
Oct 30 00:48:42.616565 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:42.616496    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28b4\" (UniqueName: \"kubernetes.io/projected/10d7885b-ffc3-42b7-a277-cece076c1fd0-kube-api-access-f28b4\") pod \"authorino-6f6944bcdd-7b8nh\" (UID: \"10d7885b-ffc3-42b7-a277-cece076c1fd0\") " pod="kuadrant-system/authorino-6f6944bcdd-7b8nh"
Oct 30 00:48:42.637726 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:42.637697    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28b4\" (UniqueName: \"kubernetes.io/projected/10d7885b-ffc3-42b7-a277-cece076c1fd0-kube-api-access-f28b4\") pod \"authorino-6f6944bcdd-7b8nh\" (UID: \"10d7885b-ffc3-42b7-a277-cece076c1fd0\") " pod="kuadrant-system/authorino-6f6944bcdd-7b8nh"
Oct 30 00:48:42.740840 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:42.740808    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c45845467-796bs"]
Oct 30 00:48:42.743207 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:42.743187    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c45845467-796bs"
Oct 30 00:48:42.758700 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:42.758681    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 00:48:42.767990 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:42.767968    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c45845467-796bs"]
Oct 30 00:48:42.919239 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:42.919163    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l49h8\" (UniqueName: \"kubernetes.io/projected/38a2d7f1-ae7c-4e84-9e84-8029b40613eb-kube-api-access-l49h8\") pod \"authorino-5c45845467-796bs\" (UID: \"38a2d7f1-ae7c-4e84-9e84-8029b40613eb\") " pod="kuadrant-system/authorino-5c45845467-796bs"
Oct 30 00:48:42.919370 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:42.919257    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/38a2d7f1-ae7c-4e84-9e84-8029b40613eb-tls-cert\") pod \"authorino-5c45845467-796bs\" (UID: \"38a2d7f1-ae7c-4e84-9e84-8029b40613eb\") " pod="kuadrant-system/authorino-5c45845467-796bs"
Oct 30 00:48:43.019724 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:43.019696    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l49h8\" (UniqueName: \"kubernetes.io/projected/38a2d7f1-ae7c-4e84-9e84-8029b40613eb-kube-api-access-l49h8\") pod \"authorino-5c45845467-796bs\" (UID: \"38a2d7f1-ae7c-4e84-9e84-8029b40613eb\") " pod="kuadrant-system/authorino-5c45845467-796bs"
Oct 30 00:48:43.019867 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:43.019736    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/38a2d7f1-ae7c-4e84-9e84-8029b40613eb-tls-cert\") pod \"authorino-5c45845467-796bs\" (UID: \"38a2d7f1-ae7c-4e84-9e84-8029b40613eb\") " pod="kuadrant-system/authorino-5c45845467-796bs"
Oct 30 00:48:43.021880 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:43.021862    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/38a2d7f1-ae7c-4e84-9e84-8029b40613eb-tls-cert\") pod \"authorino-5c45845467-796bs\" (UID: \"38a2d7f1-ae7c-4e84-9e84-8029b40613eb\") " pod="kuadrant-system/authorino-5c45845467-796bs"
Oct 30 00:48:43.041269 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:43.041248    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l49h8\" (UniqueName: \"kubernetes.io/projected/38a2d7f1-ae7c-4e84-9e84-8029b40613eb-kube-api-access-l49h8\") pod \"authorino-5c45845467-796bs\" (UID: \"38a2d7f1-ae7c-4e84-9e84-8029b40613eb\") " pod="kuadrant-system/authorino-5c45845467-796bs"
Oct 30 00:48:43.054055 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:43.054036    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c45845467-796bs"
Oct 30 00:48:43.317692 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:43.317662    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7b8nh"
Oct 30 00:48:43.323801 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:43.323777    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7b8nh"
Oct 30 00:48:43.345014 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:43.344846    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c45845467-796bs"]
Oct 30 00:48:43.347175 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:48:43.347136    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38a2d7f1_ae7c_4e84_9e84_8029b40613eb.slice/crio-2471ca121d32b26b32b851746af7ec6582cf41cffe55bb55ec3074d19ac0f805 WatchSource:0}: Error finding container 2471ca121d32b26b32b851746af7ec6582cf41cffe55bb55ec3074d19ac0f805: Status 404 returned error can't find the container with id 2471ca121d32b26b32b851746af7ec6582cf41cffe55bb55ec3074d19ac0f805
Oct 30 00:48:43.422411 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:43.422387    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10d7885b-ffc3-42b7-a277-cece076c1fd0-kube-api-access-f28b4\" (UniqueName: \"kubernetes.io/projected/10d7885b-ffc3-42b7-a277-cece076c1fd0-kube-api-access-f28b4\") pod \"10d7885b-ffc3-42b7-a277-cece076c1fd0\" (UID: \"10d7885b-ffc3-42b7-a277-cece076c1fd0\") "
Oct 30 00:48:43.424314 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:43.424285    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d7885b-ffc3-42b7-a277-cece076c1fd0-kube-api-access-f28b4" pod "10d7885b-ffc3-42b7-a277-cece076c1fd0" (UID: "10d7885b-ffc3-42b7-a277-cece076c1fd0"). InnerVolumeSpecName "kube-api-access-f28b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:43.523761 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:43.523735    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f28b4\" (UniqueName: \"kubernetes.io/projected/10d7885b-ffc3-42b7-a277-cece076c1fd0-kube-api-access-f28b4\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:48:44.323480 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:44.323447    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c45845467-796bs" event={"ID":"38a2d7f1-ae7c-4e84-9e84-8029b40613eb","Type":"ContainerStarted","Data":"2471ca121d32b26b32b851746af7ec6582cf41cffe55bb55ec3074d19ac0f805"}
Oct 30 00:48:44.323480 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:44.323466    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-7b8nh"
Oct 30 00:48:44.390300 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:44.390265    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7b8nh"]
Oct 30 00:48:44.395784 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:44.395749    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-7b8nh"]
Oct 30 00:48:45.330778 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:45.330743    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c45845467-796bs" event={"ID":"38a2d7f1-ae7c-4e84-9e84-8029b40613eb","Type":"ContainerStarted","Data":"44346b8bd6cd2d79d5e2515a54c1c7b8c2b2320e3cf6a92f93059db33083b369"}
Oct 30 00:48:45.365619 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:45.365556    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5c45845467-796bs" podStartSLOduration=1.542660452 podStartE2EDuration="3.365541832s" totalImagesPullingTime="1.82288138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:45.3646168 +0000 UTC m=+434.114272111" watchObservedRunningTime="2026-10-30 00:48:45.365541832 +0000 UTC m=+434.115197147"
Oct 30 00:48:45.391856 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:45.391825    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-l5n4t"]
Oct 30 00:48:45.392305 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:45.392035    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-l5n4t" podUID="290587aa-9503-4d06-9bad-33d5d7771c55" containerName="authorino" containerID="cri-o://718ad7374fa368baf361c4803452adc8f6c39e2d466ab0b9a95d40ec0313a39a" gracePeriod=30
Oct 30 00:48:45.429684 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:45.429614    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10d7885b-ffc3-42b7-a277-cece076c1fd0" path="/var/lib/kubelet/pods/10d7885b-ffc3-42b7-a277-cece076c1fd0/volumes"
Oct 30 00:48:45.639783 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:45.639753    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-l5n4t"
Oct 30 00:48:45.741442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:45.741354    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290587aa-9503-4d06-9bad-33d5d7771c55-kube-api-access-z4trg\" (UniqueName: \"kubernetes.io/projected/290587aa-9503-4d06-9bad-33d5d7771c55-kube-api-access-z4trg\") pod \"290587aa-9503-4d06-9bad-33d5d7771c55\" (UID: \"290587aa-9503-4d06-9bad-33d5d7771c55\") "
Oct 30 00:48:45.743526 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:45.743497    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290587aa-9503-4d06-9bad-33d5d7771c55-kube-api-access-z4trg" pod "290587aa-9503-4d06-9bad-33d5d7771c55" (UID: "290587aa-9503-4d06-9bad-33d5d7771c55"). InnerVolumeSpecName "kube-api-access-z4trg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:48:45.842546 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:45.842507    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4trg\" (UniqueName: \"kubernetes.io/projected/290587aa-9503-4d06-9bad-33d5d7771c55-kube-api-access-z4trg\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:48:46.276254 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.276216    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-n47qc"]
Oct 30 00:48:46.276685 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.276668    4425 cpu_manager.go:414] "containerMap: removing container" podUID="290587aa-9503-4d06-9bad-33d5d7771c55" containerName="authorino"
Oct 30 00:48:46.276685 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.276686    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290587aa-9503-4d06-9bad-33d5d7771c55" containerName="authorino"
Oct 30 00:48:46.276813 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.276779    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="290587aa-9503-4d06-9bad-33d5d7771c55" containerName="authorino"
Oct 30 00:48:46.278835 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.278817    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-n47qc"
Oct 30 00:48:46.302703 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.302681    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-n47qc"]
Oct 30 00:48:46.337403 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.337368    4425 generic.go:356] "Generic (PLEG): container finished" podID="290587aa-9503-4d06-9bad-33d5d7771c55" containerID="718ad7374fa368baf361c4803452adc8f6c39e2d466ab0b9a95d40ec0313a39a" exitCode=0
Oct 30 00:48:46.337584 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.337418    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-l5n4t"
Oct 30 00:48:46.337584 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.337512    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-l5n4t" event={"ID":"290587aa-9503-4d06-9bad-33d5d7771c55","Type":"ContainerDied","Data":"718ad7374fa368baf361c4803452adc8f6c39e2d466ab0b9a95d40ec0313a39a"}
Oct 30 00:48:46.337584 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.337565    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-l5n4t" event={"ID":"290587aa-9503-4d06-9bad-33d5d7771c55","Type":"ContainerDied","Data":"ad84343f67aa177105374be6266fd0b16a5eee56aac635c1cd564bb2363ba7c2"}
Oct 30 00:48:46.337753 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.337598    4425 scope.go:122] "RemoveContainer" containerID="718ad7374fa368baf361c4803452adc8f6c39e2d466ab0b9a95d40ec0313a39a"
Oct 30 00:48:46.346654 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.346638    4425 scope.go:122] "RemoveContainer" containerID="718ad7374fa368baf361c4803452adc8f6c39e2d466ab0b9a95d40ec0313a39a"
Oct 30 00:48:46.346914 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:48:46.346896    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"718ad7374fa368baf361c4803452adc8f6c39e2d466ab0b9a95d40ec0313a39a\": container with ID starting with 718ad7374fa368baf361c4803452adc8f6c39e2d466ab0b9a95d40ec0313a39a not found: ID does not exist" containerID="718ad7374fa368baf361c4803452adc8f6c39e2d466ab0b9a95d40ec0313a39a"
Oct 30 00:48:46.346974 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.346926    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"718ad7374fa368baf361c4803452adc8f6c39e2d466ab0b9a95d40ec0313a39a"} err="failed to get container status \"718ad7374fa368baf361c4803452adc8f6c39e2d466ab0b9a95d40ec0313a39a\": rpc error: code = NotFound desc = could not find container \"718ad7374fa368baf361c4803452adc8f6c39e2d466ab0b9a95d40ec0313a39a\": container with ID starting with 718ad7374fa368baf361c4803452adc8f6c39e2d466ab0b9a95d40ec0313a39a not found: ID does not exist"
Oct 30 00:48:46.382906 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.382874    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-l5n4t"]
Oct 30 00:48:46.386658 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.386633    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-l5n4t"]
Oct 30 00:48:46.447859 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.447786    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb6jt\" (UniqueName: \"kubernetes.io/projected/80392122-eba8-4040-a019-e44488201532-kube-api-access-fb6jt\") pod \"llmisvc-controller-manager-6466c554d8-n47qc\" (UID: \"80392122-eba8-4040-a019-e44488201532\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-n47qc"
Oct 30 00:48:46.448238 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.447881    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80392122-eba8-4040-a019-e44488201532-cert\") pod \"llmisvc-controller-manager-6466c554d8-n47qc\" (UID: \"80392122-eba8-4040-a019-e44488201532\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-n47qc"
Oct 30 00:48:46.549326 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.549289    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb6jt\" (UniqueName: \"kubernetes.io/projected/80392122-eba8-4040-a019-e44488201532-kube-api-access-fb6jt\") pod \"llmisvc-controller-manager-6466c554d8-n47qc\" (UID: \"80392122-eba8-4040-a019-e44488201532\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-n47qc"
Oct 30 00:48:46.549482 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.549390    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80392122-eba8-4040-a019-e44488201532-cert\") pod \"llmisvc-controller-manager-6466c554d8-n47qc\" (UID: \"80392122-eba8-4040-a019-e44488201532\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-n47qc"
Oct 30 00:48:46.551740 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.551720    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80392122-eba8-4040-a019-e44488201532-cert\") pod \"llmisvc-controller-manager-6466c554d8-n47qc\" (UID: \"80392122-eba8-4040-a019-e44488201532\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-n47qc"
Oct 30 00:48:46.570172 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.570127    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb6jt\" (UniqueName: \"kubernetes.io/projected/80392122-eba8-4040-a019-e44488201532-kube-api-access-fb6jt\") pod \"llmisvc-controller-manager-6466c554d8-n47qc\" (UID: \"80392122-eba8-4040-a019-e44488201532\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-n47qc"
Oct 30 00:48:46.591844 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.591819    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-n47qc"
Oct 30 00:48:46.883582 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:46.883549    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-n47qc"]
Oct 30 00:48:46.886622 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:48:46.886589    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80392122_eba8_4040_a019_e44488201532.slice/crio-bc08497006cb310feb1cead72f7a71e4913bd5f14063863c65026eb98564337a WatchSource:0}: Error finding container bc08497006cb310feb1cead72f7a71e4913bd5f14063863c65026eb98564337a: Status 404 returned error can't find the container with id bc08497006cb310feb1cead72f7a71e4913bd5f14063863c65026eb98564337a
Oct 30 00:48:47.344968 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:47.344934    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-n47qc" event={"ID":"80392122-eba8-4040-a019-e44488201532","Type":"ContainerStarted","Data":"6be1b78d8c59237174743f1070905b6dc5859f8ba46bfbf5d7b26b5a53280468"}
Oct 30 00:48:47.345178 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:47.344994    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-n47qc" event={"ID":"80392122-eba8-4040-a019-e44488201532","Type":"ContainerStarted","Data":"bc08497006cb310feb1cead72f7a71e4913bd5f14063863c65026eb98564337a"}
Oct 30 00:48:47.345178 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:47.345141    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-n47qc"
Oct 30 00:48:47.381223 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:47.381169    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-n47qc" podStartSLOduration=1.09738167 podStartE2EDuration="1.381144218s" totalImagesPullingTime="283.762548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:47.380121166 +0000 UTC m=+436.129776462" watchObservedRunningTime="2026-10-30 00:48:47.381144218 +0000 UTC m=+436.130799564"
Oct 30 00:48:47.420378 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:48:47.420352    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="290587aa-9503-4d06-9bad-33d5d7771c55" path="/var/lib/kubelet/pods/290587aa-9503-4d06-9bad-33d5d7771c55/volumes"
Oct 30 00:49:07.553293 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:49:07.553258    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:49:18.352329 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:18.352299    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-n47qc"
Oct 30 00:49:18.419177 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:18.419134    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"]
Oct 30 00:49:18.419454 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:18.419421    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l" podUID="26afb620-1e3d-4a2a-8c18-03eab11bd24f" containerName="manager" containerID="cri-o://f20472235a877539c112e1a480d3ac5ee9b4aa3ed7a7aebe13d078062a07d40a" gracePeriod=30
Oct 30 00:49:18.664582 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:18.664560    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"
Oct 30 00:49:18.808209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:18.808178    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26afb620-1e3d-4a2a-8c18-03eab11bd24f-kube-api-access-2s5tl\" (UniqueName: \"kubernetes.io/projected/26afb620-1e3d-4a2a-8c18-03eab11bd24f-kube-api-access-2s5tl\") pod \"26afb620-1e3d-4a2a-8c18-03eab11bd24f\" (UID: \"26afb620-1e3d-4a2a-8c18-03eab11bd24f\") "
Oct 30 00:49:18.808344 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:18.808270    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26afb620-1e3d-4a2a-8c18-03eab11bd24f-cert\" (UniqueName: \"kubernetes.io/secret/26afb620-1e3d-4a2a-8c18-03eab11bd24f-cert\") pod \"26afb620-1e3d-4a2a-8c18-03eab11bd24f\" (UID: \"26afb620-1e3d-4a2a-8c18-03eab11bd24f\") "
Oct 30 00:49:18.810376 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:18.810342    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26afb620-1e3d-4a2a-8c18-03eab11bd24f-cert" pod "26afb620-1e3d-4a2a-8c18-03eab11bd24f" (UID: "26afb620-1e3d-4a2a-8c18-03eab11bd24f"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:49:18.810486 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:18.810426    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26afb620-1e3d-4a2a-8c18-03eab11bd24f-kube-api-access-2s5tl" pod "26afb620-1e3d-4a2a-8c18-03eab11bd24f" (UID: "26afb620-1e3d-4a2a-8c18-03eab11bd24f"). InnerVolumeSpecName "kube-api-access-2s5tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:49:18.908926 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:18.908873    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s5tl\" (UniqueName: \"kubernetes.io/projected/26afb620-1e3d-4a2a-8c18-03eab11bd24f-kube-api-access-2s5tl\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:49:18.908926 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:18.908895    4425 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26afb620-1e3d-4a2a-8c18-03eab11bd24f-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:49:19.504323 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:19.504293    4425 generic.go:356] "Generic (PLEG): container finished" podID="26afb620-1e3d-4a2a-8c18-03eab11bd24f" containerID="f20472235a877539c112e1a480d3ac5ee9b4aa3ed7a7aebe13d078062a07d40a" exitCode=0
Oct 30 00:49:19.504716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:19.504332    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"
Oct 30 00:49:19.504716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:19.504366    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l" event={"ID":"26afb620-1e3d-4a2a-8c18-03eab11bd24f","Type":"ContainerDied","Data":"f20472235a877539c112e1a480d3ac5ee9b4aa3ed7a7aebe13d078062a07d40a"}
Oct 30 00:49:19.504716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:19.504404    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l" event={"ID":"26afb620-1e3d-4a2a-8c18-03eab11bd24f","Type":"ContainerDied","Data":"751e7620c196ad8d2b32b40e773e1c59b473dd2a66201642005721d67129464c"}
Oct 30 00:49:19.504716 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:19.504425    4425 scope.go:122] "RemoveContainer" containerID="f20472235a877539c112e1a480d3ac5ee9b4aa3ed7a7aebe13d078062a07d40a"
Oct 30 00:49:19.512400 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:19.512384    4425 scope.go:122] "RemoveContainer" containerID="f20472235a877539c112e1a480d3ac5ee9b4aa3ed7a7aebe13d078062a07d40a"
Oct 30 00:49:19.512676 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:49:19.512655    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20472235a877539c112e1a480d3ac5ee9b4aa3ed7a7aebe13d078062a07d40a\": container with ID starting with f20472235a877539c112e1a480d3ac5ee9b4aa3ed7a7aebe13d078062a07d40a not found: ID does not exist" containerID="f20472235a877539c112e1a480d3ac5ee9b4aa3ed7a7aebe13d078062a07d40a"
Oct 30 00:49:19.512722 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:19.512685    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20472235a877539c112e1a480d3ac5ee9b4aa3ed7a7aebe13d078062a07d40a"} err="failed to get container status \"f20472235a877539c112e1a480d3ac5ee9b4aa3ed7a7aebe13d078062a07d40a\": rpc error: code = NotFound desc = could not find container \"f20472235a877539c112e1a480d3ac5ee9b4aa3ed7a7aebe13d078062a07d40a\": container with ID starting with f20472235a877539c112e1a480d3ac5ee9b4aa3ed7a7aebe13d078062a07d40a not found: ID does not exist"
Oct 30 00:49:19.551613 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:19.551582    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"]
Oct 30 00:49:19.561303 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:19.561279    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-gjn8l"]
Oct 30 00:49:21.420721 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:21.420691    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26afb620-1e3d-4a2a-8c18-03eab11bd24f" path="/var/lib/kubelet/pods/26afb620-1e3d-4a2a-8c18-03eab11bd24f/volumes"
Oct 30 00:49:37.553910 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:49:37.553871    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:49:55.920426 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:55.920335    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:49:59.908413 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:49:59.908367    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:50:05.707711 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:05.707674    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:50:07.558202 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:50:07.558145    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:50:14.716791 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:14.716751    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:50:27.475318 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.475254    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"]
Oct 30 00:50:27.475706 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.475629    4425 cpu_manager.go:414] "containerMap: removing container" podUID="26afb620-1e3d-4a2a-8c18-03eab11bd24f" containerName="manager"
Oct 30 00:50:27.475706 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.475641    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26afb620-1e3d-4a2a-8c18-03eab11bd24f" containerName="manager"
Oct 30 00:50:27.475706 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.475699    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="26afb620-1e3d-4a2a-8c18-03eab11bd24f" containerName="manager"
Oct 30 00:50:27.477800 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.477774    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.506334 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.506310    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"]
Oct 30 00:50:27.565539 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.565510    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.565675 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.565557    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.565675 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.565580    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfee5645-34a5-4aec-a646-1742da2db219-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.565675 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.565630    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfee5645-34a5-4aec-a646-1742da2db219-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.565784 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.565698    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfee5645-34a5-4aec-a646-1742da2db219-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.565784 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.565720    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.565846 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.565803    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.565846 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.565830    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.565909 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.565849    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwpgg\" (UniqueName: \"kubernetes.io/projected/bfee5645-34a5-4aec-a646-1742da2db219-kube-api-access-pwpgg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.589473 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.589451    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 00:50:27.589578 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.589494    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-spn7j\""
Oct 30 00:50:27.666914 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.666891    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.666914 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.666916    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.667124 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.666940    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwpgg\" (UniqueName: \"kubernetes.io/projected/bfee5645-34a5-4aec-a646-1742da2db219-kube-api-access-pwpgg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.667124 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.666992    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.667124 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.667039    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.667124 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.667054    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfee5645-34a5-4aec-a646-1742da2db219-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.667124 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.667071    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfee5645-34a5-4aec-a646-1742da2db219-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.667124 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.667108    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfee5645-34a5-4aec-a646-1742da2db219-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.667552 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.667129    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.667552 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.667358    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.667552 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.667392    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.667552 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.667404    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.667813 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.667791    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.668036 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.668009    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfee5645-34a5-4aec-a646-1742da2db219-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.669567 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.669548    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.670085 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.670067    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfee5645-34a5-4aec-a646-1742da2db219-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.687874 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.687854    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfee5645-34a5-4aec-a646-1742da2db219-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.688023 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.688003    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwpgg\" (UniqueName: \"kubernetes.io/projected/bfee5645-34a5-4aec-a646-1742da2db219-kube-api-access-pwpgg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8z8gn\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:27.790264 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:27.790234    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:28.097149 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:28.097121    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"]
Oct 30 00:50:28.100119 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:50:28.100089    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfee5645_34a5_4aec_a646_1742da2db219.slice/crio-dd53e649c690e53111d458452efcade54273561d76d4fb2abb1eb9a954621b09 WatchSource:0}: Error finding container dd53e649c690e53111d458452efcade54273561d76d4fb2abb1eb9a954621b09: Status 404 returned error can't find the container with id dd53e649c690e53111d458452efcade54273561d76d4fb2abb1eb9a954621b09
Oct 30 00:50:28.833104 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:28.833068    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn" event={"ID":"bfee5645-34a5-4aec-a646-1742da2db219","Type":"ContainerStarted","Data":"dd53e649c690e53111d458452efcade54273561d76d4fb2abb1eb9a954621b09"}
Oct 30 00:50:30.696296 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:30.696265    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:30.696551 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:30.696331    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:30.696551 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:30.696358    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:50:30.846590 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:30.846555    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn" event={"ID":"bfee5645-34a5-4aec-a646-1742da2db219","Type":"ContainerStarted","Data":"4d12da9a8d2fbf2aa0ee18105f287f3b094b01431211ba204519c50bd5249380"}
Oct 30 00:50:30.911866 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:30.911817    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn" podStartSLOduration=1.318284676 podStartE2EDuration="3.911803193s" totalImagesPullingTime="2.593518517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:30.900746781 +0000 UTC m=+539.650402091" watchObservedRunningTime="2026-10-30 00:50:30.911803193 +0000 UTC m=+539.661458550"
Oct 30 00:50:31.790466 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:31.790426    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:31.792044 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:31.792009    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8z8gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 00:50:31.792181 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:31.792064    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn" podUID="bfee5645-34a5-4aec-a646-1742da2db219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 00:50:32.790859 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:32.790822    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8z8gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 00:50:32.791269 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:32.790875    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn" podUID="bfee5645-34a5-4aec-a646-1742da2db219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 00:50:33.790757 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:33.790720    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8z8gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 00:50:33.790936 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:33.790781    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn" podUID="bfee5645-34a5-4aec-a646-1742da2db219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 00:50:34.790977 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:34.790940    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8z8gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 00:50:34.791391 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:34.790992    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn" podUID="bfee5645-34a5-4aec-a646-1742da2db219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 00:50:35.791025 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:35.790988    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8z8gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 00:50:35.791459 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:35.791048    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn" podUID="bfee5645-34a5-4aec-a646-1742da2db219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 00:50:36.791194 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:36.791137    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8z8gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 00:50:36.791703 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:36.791220    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn" podUID="bfee5645-34a5-4aec-a646-1742da2db219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 00:50:37.553686 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:50:37.553649    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:50:37.791245 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:37.791212    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8z8gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 00:50:37.791666 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:37.791275    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn" podUID="bfee5645-34a5-4aec-a646-1742da2db219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 00:50:38.877516 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:38.877485    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:38.878043 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:38.877829    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:38.878765 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:38.878745    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 00:50:51.159958 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:51.159923    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c45845467-796bs"]
Oct 30 00:50:51.160530 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:51.160109    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5c45845467-796bs" podUID="38a2d7f1-ae7c-4e84-9e84-8029b40613eb" containerName="authorino" containerID="cri-o://44346b8bd6cd2d79d5e2515a54c1c7b8c2b2320e3cf6a92f93059db33083b369" gracePeriod=30
Oct 30 00:50:51.405253 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:51.405226    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c45845467-796bs"
Oct 30 00:50:51.571281 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:51.571249    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a2d7f1-ae7c-4e84-9e84-8029b40613eb-kube-api-access-l49h8\" (UniqueName: \"kubernetes.io/projected/38a2d7f1-ae7c-4e84-9e84-8029b40613eb-kube-api-access-l49h8\") pod \"38a2d7f1-ae7c-4e84-9e84-8029b40613eb\" (UID: \"38a2d7f1-ae7c-4e84-9e84-8029b40613eb\") "
Oct 30 00:50:51.571435 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:51.571304    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38a2d7f1-ae7c-4e84-9e84-8029b40613eb-tls-cert\" (UniqueName: \"kubernetes.io/secret/38a2d7f1-ae7c-4e84-9e84-8029b40613eb-tls-cert\") pod \"38a2d7f1-ae7c-4e84-9e84-8029b40613eb\" (UID: \"38a2d7f1-ae7c-4e84-9e84-8029b40613eb\") "
Oct 30 00:50:51.573340 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:51.573315    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a2d7f1-ae7c-4e84-9e84-8029b40613eb-kube-api-access-l49h8" pod "38a2d7f1-ae7c-4e84-9e84-8029b40613eb" (UID: "38a2d7f1-ae7c-4e84-9e84-8029b40613eb"). InnerVolumeSpecName "kube-api-access-l49h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:51.581323 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:51.581302    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38a2d7f1-ae7c-4e84-9e84-8029b40613eb-tls-cert" pod "38a2d7f1-ae7c-4e84-9e84-8029b40613eb" (UID: "38a2d7f1-ae7c-4e84-9e84-8029b40613eb"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:50:51.672820 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:51.672798    4425 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/38a2d7f1-ae7c-4e84-9e84-8029b40613eb-tls-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:50:51.672820 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:51.672820    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l49h8\" (UniqueName: \"kubernetes.io/projected/38a2d7f1-ae7c-4e84-9e84-8029b40613eb-kube-api-access-l49h8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:50:51.965093 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:51.965012    4425 generic.go:356] "Generic (PLEG): container finished" podID="38a2d7f1-ae7c-4e84-9e84-8029b40613eb" containerID="44346b8bd6cd2d79d5e2515a54c1c7b8c2b2320e3cf6a92f93059db33083b369" exitCode=0
Oct 30 00:50:51.965093 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:51.965061    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c45845467-796bs"
Oct 30 00:50:51.965267 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:51.965096    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c45845467-796bs" event={"ID":"38a2d7f1-ae7c-4e84-9e84-8029b40613eb","Type":"ContainerDied","Data":"44346b8bd6cd2d79d5e2515a54c1c7b8c2b2320e3cf6a92f93059db33083b369"}
Oct 30 00:50:51.965267 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:51.965138    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c45845467-796bs" event={"ID":"38a2d7f1-ae7c-4e84-9e84-8029b40613eb","Type":"ContainerDied","Data":"2471ca121d32b26b32b851746af7ec6582cf41cffe55bb55ec3074d19ac0f805"}
Oct 30 00:50:51.965267 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:51.965167    4425 scope.go:122] "RemoveContainer" containerID="44346b8bd6cd2d79d5e2515a54c1c7b8c2b2320e3cf6a92f93059db33083b369"
Oct 30 00:50:51.973873 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:51.973858    4425 scope.go:122] "RemoveContainer" containerID="44346b8bd6cd2d79d5e2515a54c1c7b8c2b2320e3cf6a92f93059db33083b369"
Oct 30 00:50:51.974124 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:50:51.974105    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44346b8bd6cd2d79d5e2515a54c1c7b8c2b2320e3cf6a92f93059db33083b369\": container with ID starting with 44346b8bd6cd2d79d5e2515a54c1c7b8c2b2320e3cf6a92f93059db33083b369 not found: ID does not exist" containerID="44346b8bd6cd2d79d5e2515a54c1c7b8c2b2320e3cf6a92f93059db33083b369"
Oct 30 00:50:51.974217 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:51.974130    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44346b8bd6cd2d79d5e2515a54c1c7b8c2b2320e3cf6a92f93059db33083b369"} err="failed to get container status \"44346b8bd6cd2d79d5e2515a54c1c7b8c2b2320e3cf6a92f93059db33083b369\": rpc error: code = NotFound desc = could not find container \"44346b8bd6cd2d79d5e2515a54c1c7b8c2b2320e3cf6a92f93059db33083b369\": container with ID starting with 44346b8bd6cd2d79d5e2515a54c1c7b8c2b2320e3cf6a92f93059db33083b369 not found: ID does not exist"
Oct 30 00:50:52.011723 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:52.011703    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c45845467-796bs"]
Oct 30 00:50:52.021477 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:52.021456    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5c45845467-796bs"]
Oct 30 00:50:53.421320 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:50:53.421289    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a2d7f1-ae7c-4e84-9e84-8029b40613eb" path="/var/lib/kubelet/pods/38a2d7f1-ae7c-4e84-9e84-8029b40613eb/volumes"
Oct 30 00:51:07.553687 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:51:07.553582    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:51:37.552910 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:51:37.552875    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:52:07.553029 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:52:07.552990    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:52:37.553348 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:52:37.553312    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:52:54.996699 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:52:54.996625    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:53:02.640566 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:02.640532    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:53:04.327145 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:04.327106    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"]
Oct 30 00:53:04.327618 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:04.327594    4425 cpu_manager.go:414] "containerMap: removing container" podUID="38a2d7f1-ae7c-4e84-9e84-8029b40613eb" containerName="authorino"
Oct 30 00:53:04.327832 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:04.327619    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a2d7f1-ae7c-4e84-9e84-8029b40613eb" containerName="authorino"
Oct 30 00:53:04.327832 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:04.327727    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a2d7f1-ae7c-4e84-9e84-8029b40613eb" containerName="authorino"
Oct 30 00:53:04.330874 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:04.330858    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"
Oct 30 00:53:04.350202 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:04.350171    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"]
Oct 30 00:53:04.355148 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:04.355129    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3cc211d9\""
Oct 30 00:53:04.355361 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:04.355346    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3cc211d9-dockercfg-w99dw\""
Oct 30 00:53:04.390550 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:04.390517    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbca1e2c-a5d6-4290-8557-542bc335349c-plugins-config-volume\") pod \"payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g\" (UID: \"cbca1e2c-a5d6-4290-8557-542bc335349c\") " pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"
Oct 30 00:53:04.390709 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:04.390682    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz55l\" (UniqueName: \"kubernetes.io/projected/cbca1e2c-a5d6-4290-8557-542bc335349c-kube-api-access-zz55l\") pod \"payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g\" (UID: \"cbca1e2c-a5d6-4290-8557-542bc335349c\") " pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"
Oct 30 00:53:04.491635 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:04.491607    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz55l\" (UniqueName: \"kubernetes.io/projected/cbca1e2c-a5d6-4290-8557-542bc335349c-kube-api-access-zz55l\") pod \"payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g\" (UID: \"cbca1e2c-a5d6-4290-8557-542bc335349c\") " pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"
Oct 30 00:53:04.491805 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:04.491655    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbca1e2c-a5d6-4290-8557-542bc335349c-plugins-config-volume\") pod \"payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g\" (UID: \"cbca1e2c-a5d6-4290-8557-542bc335349c\") " pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"
Oct 30 00:53:04.492526 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:04.492500    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbca1e2c-a5d6-4290-8557-542bc335349c-plugins-config-volume\") pod \"payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g\" (UID: \"cbca1e2c-a5d6-4290-8557-542bc335349c\") " pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"
Oct 30 00:53:04.512673 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:04.512649    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz55l\" (UniqueName: \"kubernetes.io/projected/cbca1e2c-a5d6-4290-8557-542bc335349c-kube-api-access-zz55l\") pod \"payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g\" (UID: \"cbca1e2c-a5d6-4290-8557-542bc335349c\") " pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"
Oct 30 00:53:04.642920 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:04.642802    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"
Oct 30 00:53:04.942875 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:04.942840    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"]
Oct 30 00:53:04.945410 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:53:04.945346    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbca1e2c_a5d6_4290_8557_542bc335349c.slice/crio-1b1002dde10562d789aeca1aca4604c4f27112cd88e3b51f5d643e3dfdec0584 WatchSource:0}: Error finding container 1b1002dde10562d789aeca1aca4604c4f27112cd88e3b51f5d643e3dfdec0584: Status 404 returned error can't find the container with id 1b1002dde10562d789aeca1aca4604c4f27112cd88e3b51f5d643e3dfdec0584
Oct 30 00:53:04.947507 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:04.947484    4425 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:53:05.083135 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:05.083101    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"]
Oct 30 00:53:05.087943 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:05.087920    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"
Oct 30 00:53:05.100291 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:05.100269    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5e61cc8d-dockercfg-wgt4x\""
Oct 30 00:53:05.108673 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:05.108648    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"]
Oct 30 00:53:05.145143 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:05.145121    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5e61cc8d\""
Oct 30 00:53:05.197577 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:05.197511    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de2e8b89-5f6c-4909-b85b-92702a9bd114-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p\" (UID: \"de2e8b89-5f6c-4909-b85b-92702a9bd114\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"
Oct 30 00:53:05.197577 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:05.197543    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbctl\" (UniqueName: \"kubernetes.io/projected/de2e8b89-5f6c-4909-b85b-92702a9bd114-kube-api-access-pbctl\") pod \"payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p\" (UID: \"de2e8b89-5f6c-4909-b85b-92702a9bd114\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"
Oct 30 00:53:05.299028 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:05.298992    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de2e8b89-5f6c-4909-b85b-92702a9bd114-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p\" (UID: \"de2e8b89-5f6c-4909-b85b-92702a9bd114\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"
Oct 30 00:53:05.299028 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:05.299031    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbctl\" (UniqueName: \"kubernetes.io/projected/de2e8b89-5f6c-4909-b85b-92702a9bd114-kube-api-access-pbctl\") pod \"payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p\" (UID: \"de2e8b89-5f6c-4909-b85b-92702a9bd114\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"
Oct 30 00:53:05.299777 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:05.299753    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de2e8b89-5f6c-4909-b85b-92702a9bd114-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p\" (UID: \"de2e8b89-5f6c-4909-b85b-92702a9bd114\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"
Oct 30 00:53:05.320574 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:05.320543    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbctl\" (UniqueName: \"kubernetes.io/projected/de2e8b89-5f6c-4909-b85b-92702a9bd114-kube-api-access-pbctl\") pod \"payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p\" (UID: \"de2e8b89-5f6c-4909-b85b-92702a9bd114\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"
Oct 30 00:53:05.400967 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:05.400935    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"
Oct 30 00:53:05.555427 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:05.555390    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g" event={"ID":"cbca1e2c-a5d6-4290-8557-542bc335349c","Type":"ContainerStarted","Data":"1b1002dde10562d789aeca1aca4604c4f27112cd88e3b51f5d643e3dfdec0584"}
Oct 30 00:53:05.713726 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:05.713522    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"]
Oct 30 00:53:05.717009 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:53:05.716940    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2e8b89_5f6c_4909_b85b_92702a9bd114.slice/crio-611e3adddbe631ad84d6266676fda27c60029cd914a62fa4c8c2f533a61978a8 WatchSource:0}: Error finding container 611e3adddbe631ad84d6266676fda27c60029cd914a62fa4c8c2f533a61978a8: Status 404 returned error can't find the container with id 611e3adddbe631ad84d6266676fda27c60029cd914a62fa4c8c2f533a61978a8
Oct 30 00:53:06.562505 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:06.562460    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p" event={"ID":"de2e8b89-5f6c-4909-b85b-92702a9bd114","Type":"ContainerStarted","Data":"611e3adddbe631ad84d6266676fda27c60029cd914a62fa4c8c2f533a61978a8"}
Oct 30 00:53:07.553336 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:53:07.553301    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:53:07.568512 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:07.568485    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g" event={"ID":"cbca1e2c-a5d6-4290-8557-542bc335349c","Type":"ContainerStarted","Data":"8d94e09cb45115a8b500da9a9d899f714e50f5e73b5784ac13255b678c8abca7"}
Oct 30 00:53:07.568892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:07.568688    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"
Oct 30 00:53:07.570021 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:07.569995    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p" event={"ID":"de2e8b89-5f6c-4909-b85b-92702a9bd114","Type":"ContainerStarted","Data":"8d67fb34deb588d91052e9ebc60a316f2827e0ef469b42369708184ab4fef098"}
Oct 30 00:53:07.570142 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:07.570066    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"
Oct 30 00:53:07.604872 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:07.604832    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g" podStartSLOduration=1.586328956 podStartE2EDuration="3.60482071s" totalImagesPullingTime="2.018491754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:07.603906198 +0000 UTC m=+696.353561506" watchObservedRunningTime="2026-10-30 00:53:07.60482071 +0000 UTC m=+696.354476056"
Oct 30 00:53:07.640744 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:07.640706    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p" podStartSLOduration=1.382465284 podStartE2EDuration="2.640691521s" totalImagesPullingTime="1.258226237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:07.639279525 +0000 UTC m=+696.388934835" watchObservedRunningTime="2026-10-30 00:53:07.640691521 +0000 UTC m=+696.390346830"
Oct 30 00:53:11.030074 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:11.030037    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:53:18.578210 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:18.578149    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"
Oct 30 00:53:18.578657 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:18.578265    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"
Oct 30 00:53:22.263312 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.263276    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"]
Oct 30 00:53:22.267345 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.267320    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"
Oct 30 00:53:22.296402 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.296371    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:53:22.296599 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.296414    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:53:22.296599 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.296455    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:53:22.296793 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.296776    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 00:53:22.299252 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.299227    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"]
Oct 30 00:53:22.351229 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.351201    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc\" (UID: \"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"
Oct 30 00:53:22.351378 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.351275    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjt6h\" (UniqueName: \"kubernetes.io/projected/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-kube-api-access-fjt6h\") pod \"maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc\" (UID: \"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"
Oct 30 00:53:22.351378 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.351344    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc\" (UID: \"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"
Oct 30 00:53:22.429464 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.429430    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:53:22.452236 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.452211    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc\" (UID: \"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"
Oct 30 00:53:22.452396 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.452356    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjt6h\" (UniqueName: \"kubernetes.io/projected/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-kube-api-access-fjt6h\") pod \"maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc\" (UID: \"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"
Oct 30 00:53:22.452396 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.452390    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc\" (UID: \"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"
Oct 30 00:53:22.452945 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.452919    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc\" (UID: \"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"
Oct 30 00:53:22.454706 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.454683    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc\" (UID: \"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"
Oct 30 00:53:22.473034 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.473014    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjt6h\" (UniqueName: \"kubernetes.io/projected/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-kube-api-access-fjt6h\") pod \"maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc\" (UID: \"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"
Oct 30 00:53:22.581885 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.581812    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"
Oct 30 00:53:22.882445 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:22.882148    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"]
Oct 30 00:53:22.884748 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:53:22.884720    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0d74cc_a8f8_48fd_8dfd_fa2ae32b8311.slice/crio-dbd1230bca860d6638ae2ab39994e326d430115e9335730b1d80c80ef0fa036a WatchSource:0}: Error finding container dbd1230bca860d6638ae2ab39994e326d430115e9335730b1d80c80ef0fa036a: Status 404 returned error can't find the container with id dbd1230bca860d6638ae2ab39994e326d430115e9335730b1d80c80ef0fa036a
Oct 30 00:53:23.647266 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:23.647225    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc" event={"ID":"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311","Type":"ContainerStarted","Data":"dbd1230bca860d6638ae2ab39994e326d430115e9335730b1d80c80ef0fa036a"}
Oct 30 00:53:25.658650 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:25.658606    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc" event={"ID":"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311","Type":"ContainerStarted","Data":"eca26c39e700e12ceee1c61deaf51af5f90e4ff097c1f4fe1469e55b5bf4181f"}
Oct 30 00:53:25.694457 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:25.694408    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc" podStartSLOduration=1.749994131 podStartE2EDuration="3.694395204s" totalImagesPullingTime="1.944401073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:25.694110956 +0000 UTC m=+714.443766268" watchObservedRunningTime="2026-10-30 00:53:25.694395204 +0000 UTC m=+714.444050513"
Oct 30 00:53:26.664371 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:26.664336    4425 generic.go:356] "Generic (PLEG): container finished" podID="6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311" containerID="eca26c39e700e12ceee1c61deaf51af5f90e4ff097c1f4fe1469e55b5bf4181f" exitCode=0
Oct 30 00:53:26.664767 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:26.664423    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc" event={"ID":"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311","Type":"ContainerDied","Data":"eca26c39e700e12ceee1c61deaf51af5f90e4ff097c1f4fe1469e55b5bf4181f"}
Oct 30 00:53:26.795291 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:26.795261    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"]
Oct 30 00:53:26.799244 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:26.799223    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"
Oct 30 00:53:26.820207 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:26.820176    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"]
Oct 30 00:53:26.892765 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:26.892740    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx9zw\" (UniqueName: \"kubernetes.io/projected/a844b4d3-22b0-4d52-a6de-456627b1c7d5-kube-api-access-wx9zw\") pod \"maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l\" (UID: \"a844b4d3-22b0-4d52-a6de-456627b1c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"
Oct 30 00:53:26.892918 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:26.892806    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a844b4d3-22b0-4d52-a6de-456627b1c7d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l\" (UID: \"a844b4d3-22b0-4d52-a6de-456627b1c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"
Oct 30 00:53:26.892918 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:26.892899    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a844b4d3-22b0-4d52-a6de-456627b1c7d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l\" (UID: \"a844b4d3-22b0-4d52-a6de-456627b1c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"
Oct 30 00:53:26.993556 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:26.993477    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx9zw\" (UniqueName: \"kubernetes.io/projected/a844b4d3-22b0-4d52-a6de-456627b1c7d5-kube-api-access-wx9zw\") pod \"maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l\" (UID: \"a844b4d3-22b0-4d52-a6de-456627b1c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"
Oct 30 00:53:26.993556 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:26.993536    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a844b4d3-22b0-4d52-a6de-456627b1c7d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l\" (UID: \"a844b4d3-22b0-4d52-a6de-456627b1c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"
Oct 30 00:53:26.993556 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:26.993561    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a844b4d3-22b0-4d52-a6de-456627b1c7d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l\" (UID: \"a844b4d3-22b0-4d52-a6de-456627b1c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"
Oct 30 00:53:26.994165 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:26.994130    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a844b4d3-22b0-4d52-a6de-456627b1c7d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l\" (UID: \"a844b4d3-22b0-4d52-a6de-456627b1c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"
Oct 30 00:53:26.995794 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:26.995776    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a844b4d3-22b0-4d52-a6de-456627b1c7d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l\" (UID: \"a844b4d3-22b0-4d52-a6de-456627b1c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"
Oct 30 00:53:27.014837 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:27.014811    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx9zw\" (UniqueName: \"kubernetes.io/projected/a844b4d3-22b0-4d52-a6de-456627b1c7d5-kube-api-access-wx9zw\") pod \"maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l\" (UID: \"a844b4d3-22b0-4d52-a6de-456627b1c7d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"
Oct 30 00:53:27.110837 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:27.110811    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"
Oct 30 00:53:27.413407 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:27.413379    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"]
Oct 30 00:53:27.416258 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:53:27.416228    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda844b4d3_22b0_4d52_a6de_456627b1c7d5.slice/crio-c04cd8c7a49824525bbed46bed09ca6c0180fff65298f825e0312a09700829f3 WatchSource:0}: Error finding container c04cd8c7a49824525bbed46bed09ca6c0180fff65298f825e0312a09700829f3: Status 404 returned error can't find the container with id c04cd8c7a49824525bbed46bed09ca6c0180fff65298f825e0312a09700829f3
Oct 30 00:53:27.670526 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:27.670458    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l" event={"ID":"a844b4d3-22b0-4d52-a6de-456627b1c7d5","Type":"ContainerStarted","Data":"c04cd8c7a49824525bbed46bed09ca6c0180fff65298f825e0312a09700829f3"}
Oct 30 00:53:27.794733 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:27.794709    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"
Oct 30 00:53:27.902454 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:27.902420    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-maas-api-xks-ca\") pod \"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311\" (UID: \"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311\") "
Oct 30 00:53:27.902588 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:27.902535    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-kube-api-access-fjt6h\" (UniqueName: \"kubernetes.io/projected/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-kube-api-access-fjt6h\") pod \"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311\" (UID: \"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311\") "
Oct 30 00:53:27.902641 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:27.902601    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-maas-api-openshift-ca\") pod \"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311\" (UID: \"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311\") "
Oct 30 00:53:27.902945 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:27.902921    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-maas-api-openshift-ca" pod "6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311" (UID: "6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:27.904674 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:27.904650    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-kube-api-access-fjt6h" pod "6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311" (UID: "6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311"). InnerVolumeSpecName "kube-api-access-fjt6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:27.904779 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:27.904655    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-maas-api-xks-ca" pod "6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311" (UID: "6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:53:28.003906 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:28.003876    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:53:28.004040 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:28.003911    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:53:28.004040 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:28.003928    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjt6h\" (UniqueName: \"kubernetes.io/projected/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311-kube-api-access-fjt6h\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:53:28.676474 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:28.676379    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"
Oct 30 00:53:28.676474 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:28.676393    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc" event={"ID":"6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311","Type":"ContainerDied","Data":"dbd1230bca860d6638ae2ab39994e326d430115e9335730b1d80c80ef0fa036a"}
Oct 30 00:53:28.676474 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:28.676445    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbd1230bca860d6638ae2ab39994e326d430115e9335730b1d80c80ef0fa036a"
Oct 30 00:53:28.677968 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:28.677946    4425 generic.go:356] "Generic (PLEG): container finished" podID="a844b4d3-22b0-4d52-a6de-456627b1c7d5" containerID="9f66848597fe7b880814fa7a7895ce6eeef5573a6238bbe54c5ceebaca3f1842" exitCode=0
Oct 30 00:53:28.678071 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:28.678002    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l" event={"ID":"a844b4d3-22b0-4d52-a6de-456627b1c7d5","Type":"ContainerDied","Data":"9f66848597fe7b880814fa7a7895ce6eeef5573a6238bbe54c5ceebaca3f1842"}
Oct 30 00:53:29.811254 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:29.811230    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"
Oct 30 00:53:29.922522 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:29.922496    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a844b4d3-22b0-4d52-a6de-456627b1c7d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a844b4d3-22b0-4d52-a6de-456627b1c7d5-maas-api-xks-ca\") pod \"a844b4d3-22b0-4d52-a6de-456627b1c7d5\" (UID: \"a844b4d3-22b0-4d52-a6de-456627b1c7d5\") "
Oct 30 00:53:29.922670 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:29.922553    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a844b4d3-22b0-4d52-a6de-456627b1c7d5-kube-api-access-wx9zw\" (UniqueName: \"kubernetes.io/projected/a844b4d3-22b0-4d52-a6de-456627b1c7d5-kube-api-access-wx9zw\") pod \"a844b4d3-22b0-4d52-a6de-456627b1c7d5\" (UID: \"a844b4d3-22b0-4d52-a6de-456627b1c7d5\") "
Oct 30 00:53:29.922670 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:29.922585    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a844b4d3-22b0-4d52-a6de-456627b1c7d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a844b4d3-22b0-4d52-a6de-456627b1c7d5-maas-api-openshift-ca\") pod \"a844b4d3-22b0-4d52-a6de-456627b1c7d5\" (UID: \"a844b4d3-22b0-4d52-a6de-456627b1c7d5\") "
Oct 30 00:53:29.922969 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:29.922944    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a844b4d3-22b0-4d52-a6de-456627b1c7d5-maas-api-openshift-ca" pod "a844b4d3-22b0-4d52-a6de-456627b1c7d5" (UID: "a844b4d3-22b0-4d52-a6de-456627b1c7d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:29.924616 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:29.924590    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a844b4d3-22b0-4d52-a6de-456627b1c7d5-maas-api-xks-ca" pod "a844b4d3-22b0-4d52-a6de-456627b1c7d5" (UID: "a844b4d3-22b0-4d52-a6de-456627b1c7d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:53:29.924707 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:29.924657    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a844b4d3-22b0-4d52-a6de-456627b1c7d5-kube-api-access-wx9zw" pod "a844b4d3-22b0-4d52-a6de-456627b1c7d5" (UID: "a844b4d3-22b0-4d52-a6de-456627b1c7d5"). InnerVolumeSpecName "kube-api-access-wx9zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:30.023651 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:30.023625    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a844b4d3-22b0-4d52-a6de-456627b1c7d5-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:53:30.023651 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:30.023651    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx9zw\" (UniqueName: \"kubernetes.io/projected/a844b4d3-22b0-4d52-a6de-456627b1c7d5-kube-api-access-wx9zw\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:53:30.023803 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:30.023667    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a844b4d3-22b0-4d52-a6de-456627b1c7d5-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:53:30.687527 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:30.687494    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"
Oct 30 00:53:30.687679 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:30.687497    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l" event={"ID":"a844b4d3-22b0-4d52-a6de-456627b1c7d5","Type":"ContainerDied","Data":"c04cd8c7a49824525bbed46bed09ca6c0180fff65298f825e0312a09700829f3"}
Oct 30 00:53:30.687679 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:30.687646    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c04cd8c7a49824525bbed46bed09ca6c0180fff65298f825e0312a09700829f3"
Oct 30 00:53:37.308545 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:37.308504    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"]
Oct 30 00:53:37.556744 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:53:37.556709    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:53:38.146648 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:38.146617    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:53:38.394089 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:38.394055    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"]
Oct 30 00:53:38.403653 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:38.403584    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-aaf98f1a-bzzsc"]
Oct 30 00:53:39.420944 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:39.420915    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311" path="/var/lib/kubelet/pods/6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311/volumes"
Oct 30 00:53:42.309214 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:42.309174    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g" podUID="cbca1e2c-a5d6-4290-8557-542bc335349c" containerName="payload-processing" containerID="cri-o://8d94e09cb45115a8b500da9a9d899f714e50f5e73b5784ac13255b678c8abca7" gracePeriod=25
Oct 30 00:53:42.552748 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:42.552726    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"
Oct 30 00:53:42.740307 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:42.740237    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbca1e2c-a5d6-4290-8557-542bc335349c-kube-api-access-zz55l\" (UniqueName: \"kubernetes.io/projected/cbca1e2c-a5d6-4290-8557-542bc335349c-kube-api-access-zz55l\") pod \"cbca1e2c-a5d6-4290-8557-542bc335349c\" (UID: \"cbca1e2c-a5d6-4290-8557-542bc335349c\") "
Oct 30 00:53:42.740307 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:42.740279    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbca1e2c-a5d6-4290-8557-542bc335349c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbca1e2c-a5d6-4290-8557-542bc335349c-plugins-config-volume\") pod \"cbca1e2c-a5d6-4290-8557-542bc335349c\" (UID: \"cbca1e2c-a5d6-4290-8557-542bc335349c\") "
Oct 30 00:53:42.740701 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:42.740670    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbca1e2c-a5d6-4290-8557-542bc335349c-plugins-config-volume" pod "cbca1e2c-a5d6-4290-8557-542bc335349c" (UID: "cbca1e2c-a5d6-4290-8557-542bc335349c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:53:42.741403 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:42.741381    4425 generic.go:356] "Generic (PLEG): container finished" podID="cbca1e2c-a5d6-4290-8557-542bc335349c" containerID="8d94e09cb45115a8b500da9a9d899f714e50f5e73b5784ac13255b678c8abca7" exitCode=0
Oct 30 00:53:42.741504 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:42.741438    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"
Oct 30 00:53:42.741504 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:42.741465    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g" event={"ID":"cbca1e2c-a5d6-4290-8557-542bc335349c","Type":"ContainerDied","Data":"8d94e09cb45115a8b500da9a9d899f714e50f5e73b5784ac13255b678c8abca7"}
Oct 30 00:53:42.741591 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:42.741539    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g" event={"ID":"cbca1e2c-a5d6-4290-8557-542bc335349c","Type":"ContainerDied","Data":"1b1002dde10562d789aeca1aca4604c4f27112cd88e3b51f5d643e3dfdec0584"}
Oct 30 00:53:42.741591 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:42.741567    4425 scope.go:122] "RemoveContainer" containerID="8d94e09cb45115a8b500da9a9d899f714e50f5e73b5784ac13255b678c8abca7"
Oct 30 00:53:42.742729 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:42.742706    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbca1e2c-a5d6-4290-8557-542bc335349c-kube-api-access-zz55l" pod "cbca1e2c-a5d6-4290-8557-542bc335349c" (UID: "cbca1e2c-a5d6-4290-8557-542bc335349c"). InnerVolumeSpecName "kube-api-access-zz55l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:53:42.753792 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:42.753778    4425 scope.go:122] "RemoveContainer" containerID="8d94e09cb45115a8b500da9a9d899f714e50f5e73b5784ac13255b678c8abca7"
Oct 30 00:53:42.754073 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:53:42.754056    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d94e09cb45115a8b500da9a9d899f714e50f5e73b5784ac13255b678c8abca7\": container with ID starting with 8d94e09cb45115a8b500da9a9d899f714e50f5e73b5784ac13255b678c8abca7 not found: ID does not exist" containerID="8d94e09cb45115a8b500da9a9d899f714e50f5e73b5784ac13255b678c8abca7"
Oct 30 00:53:42.754111 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:42.754086    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d94e09cb45115a8b500da9a9d899f714e50f5e73b5784ac13255b678c8abca7"} err="failed to get container status \"8d94e09cb45115a8b500da9a9d899f714e50f5e73b5784ac13255b678c8abca7\": rpc error: code = NotFound desc = could not find container \"8d94e09cb45115a8b500da9a9d899f714e50f5e73b5784ac13255b678c8abca7\": container with ID starting with 8d94e09cb45115a8b500da9a9d899f714e50f5e73b5784ac13255b678c8abca7 not found: ID does not exist"
Oct 30 00:53:42.841512 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:42.841485    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz55l\" (UniqueName: \"kubernetes.io/projected/cbca1e2c-a5d6-4290-8557-542bc335349c-kube-api-access-zz55l\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:53:42.841593 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:42.841515    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbca1e2c-a5d6-4290-8557-542bc335349c-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:53:43.081099 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:43.081072    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"]
Oct 30 00:53:43.090859 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:43.090833    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3cc211d9-656d69c8bb-pjc8g"]
Oct 30 00:53:43.420869 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:43.420796    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbca1e2c-a5d6-4290-8557-542bc335349c" path="/var/lib/kubelet/pods/cbca1e2c-a5d6-4290-8557-542bc335349c/volumes"
Oct 30 00:53:43.811305 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:43.811274    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"]
Oct 30 00:53:43.820715 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:43.820689    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3cc211d9-vj76l"]
Oct 30 00:53:45.420575 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:45.420538    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a844b4d3-22b0-4d52-a6de-456627b1c7d5" path="/var/lib/kubelet/pods/a844b4d3-22b0-4d52-a6de-456627b1c7d5/volumes"
Oct 30 00:53:49.001287 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:49.001247    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:53:57.875575 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:57.875543    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:53:59.022542 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:53:59.022505    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"]
Oct 30 00:54:04.023859 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:04.023799    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p" podUID="de2e8b89-5f6c-4909-b85b-92702a9bd114" containerName="payload-pre-processing" containerID="cri-o://8d67fb34deb588d91052e9ebc60a316f2827e0ef469b42369708184ab4fef098" gracePeriod=25
Oct 30 00:54:04.358049 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:04.358023    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"
Oct 30 00:54:04.431565 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:04.431539    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de2e8b89-5f6c-4909-b85b-92702a9bd114-kube-api-access-pbctl\" (UniqueName: \"kubernetes.io/projected/de2e8b89-5f6c-4909-b85b-92702a9bd114-kube-api-access-pbctl\") pod \"de2e8b89-5f6c-4909-b85b-92702a9bd114\" (UID: \"de2e8b89-5f6c-4909-b85b-92702a9bd114\") "
Oct 30 00:54:04.431713 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:04.431605    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de2e8b89-5f6c-4909-b85b-92702a9bd114-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de2e8b89-5f6c-4909-b85b-92702a9bd114-plugins-config-volume\") pod \"de2e8b89-5f6c-4909-b85b-92702a9bd114\" (UID: \"de2e8b89-5f6c-4909-b85b-92702a9bd114\") "
Oct 30 00:54:04.431971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:04.431951    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2e8b89-5f6c-4909-b85b-92702a9bd114-plugins-config-volume" pod "de2e8b89-5f6c-4909-b85b-92702a9bd114" (UID: "de2e8b89-5f6c-4909-b85b-92702a9bd114"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:54:04.433621 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:04.433592    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2e8b89-5f6c-4909-b85b-92702a9bd114-kube-api-access-pbctl" pod "de2e8b89-5f6c-4909-b85b-92702a9bd114" (UID: "de2e8b89-5f6c-4909-b85b-92702a9bd114"). InnerVolumeSpecName "kube-api-access-pbctl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:54:04.532796 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:04.532738    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbctl\" (UniqueName: \"kubernetes.io/projected/de2e8b89-5f6c-4909-b85b-92702a9bd114-kube-api-access-pbctl\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:54:04.532796 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:04.532771    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de2e8b89-5f6c-4909-b85b-92702a9bd114-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:54:04.857288 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:04.857256    4425 generic.go:356] "Generic (PLEG): container finished" podID="de2e8b89-5f6c-4909-b85b-92702a9bd114" containerID="8d67fb34deb588d91052e9ebc60a316f2827e0ef469b42369708184ab4fef098" exitCode=0
Oct 30 00:54:04.857472 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:04.857297    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"
Oct 30 00:54:04.857472 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:04.857342    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p" event={"ID":"de2e8b89-5f6c-4909-b85b-92702a9bd114","Type":"ContainerDied","Data":"8d67fb34deb588d91052e9ebc60a316f2827e0ef469b42369708184ab4fef098"}
Oct 30 00:54:04.857472 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:04.857398    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p" event={"ID":"de2e8b89-5f6c-4909-b85b-92702a9bd114","Type":"ContainerDied","Data":"611e3adddbe631ad84d6266676fda27c60029cd914a62fa4c8c2f533a61978a8"}
Oct 30 00:54:04.857472 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:04.857422    4425 scope.go:122] "RemoveContainer" containerID="8d67fb34deb588d91052e9ebc60a316f2827e0ef469b42369708184ab4fef098"
Oct 30 00:54:04.866529 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:04.866502    4425 scope.go:122] "RemoveContainer" containerID="8d67fb34deb588d91052e9ebc60a316f2827e0ef469b42369708184ab4fef098"
Oct 30 00:54:04.866777 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:54:04.866750    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d67fb34deb588d91052e9ebc60a316f2827e0ef469b42369708184ab4fef098\": container with ID starting with 8d67fb34deb588d91052e9ebc60a316f2827e0ef469b42369708184ab4fef098 not found: ID does not exist" containerID="8d67fb34deb588d91052e9ebc60a316f2827e0ef469b42369708184ab4fef098"
Oct 30 00:54:04.866826 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:04.866783    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d67fb34deb588d91052e9ebc60a316f2827e0ef469b42369708184ab4fef098"} err="failed to get container status \"8d67fb34deb588d91052e9ebc60a316f2827e0ef469b42369708184ab4fef098\": rpc error: code = NotFound desc = could not find container \"8d67fb34deb588d91052e9ebc60a316f2827e0ef469b42369708184ab4fef098\": container with ID starting with 8d67fb34deb588d91052e9ebc60a316f2827e0ef469b42369708184ab4fef098 not found: ID does not exist"
Oct 30 00:54:04.903176 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:04.903142    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"]
Oct 30 00:54:04.912885 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:04.912866    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5e61cc8d-59d765895-f5d4p"]
Oct 30 00:54:05.420491 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:05.420459    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de2e8b89-5f6c-4909-b85b-92702a9bd114" path="/var/lib/kubelet/pods/de2e8b89-5f6c-4909-b85b-92702a9bd114/volumes"
Oct 30 00:54:07.553789 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:54:07.553754    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:54:09.963566 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:09.963528    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:54:14.520906 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.520813    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"]
Oct 30 00:54:14.521414 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.521386    4425 cpu_manager.go:414] "containerMap: removing container" podUID="cbca1e2c-a5d6-4290-8557-542bc335349c" containerName="payload-processing"
Oct 30 00:54:14.521414 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.521403    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbca1e2c-a5d6-4290-8557-542bc335349c" containerName="payload-processing"
Oct 30 00:54:14.521494 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.521424    4425 cpu_manager.go:414] "containerMap: removing container" podUID="6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311" containerName="revoke-keys"
Oct 30 00:54:14.521494 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.521430    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311" containerName="revoke-keys"
Oct 30 00:54:14.521494 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.521442    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a844b4d3-22b0-4d52-a6de-456627b1c7d5" containerName="revoke-keys"
Oct 30 00:54:14.521494 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.521447    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a844b4d3-22b0-4d52-a6de-456627b1c7d5" containerName="revoke-keys"
Oct 30 00:54:14.521494 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.521457    4425 cpu_manager.go:414] "containerMap: removing container" podUID="de2e8b89-5f6c-4909-b85b-92702a9bd114" containerName="payload-pre-processing"
Oct 30 00:54:14.521494 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.521462    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de2e8b89-5f6c-4909-b85b-92702a9bd114" containerName="payload-pre-processing"
Oct 30 00:54:14.521671 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.521525    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0d74cc-a8f8-48fd-8dfd-fa2ae32b8311" containerName="revoke-keys"
Oct 30 00:54:14.521671 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.521533    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a844b4d3-22b0-4d52-a6de-456627b1c7d5" containerName="revoke-keys"
Oct 30 00:54:14.521671 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.521540    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="de2e8b89-5f6c-4909-b85b-92702a9bd114" containerName="payload-pre-processing"
Oct 30 00:54:14.521671 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.521548    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbca1e2c-a5d6-4290-8557-542bc335349c" containerName="payload-processing"
Oct 30 00:54:14.525732 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.525711    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.550421 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.550400    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-72ed13\"/\"kube-root-ca.crt\""
Oct 30 00:54:14.550711 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.550687    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-72ed13\"/\"simulator-w2-72ed13-kserve-self-signed-certs\""
Oct 30 00:54:14.550847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.550717    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-72ed13\"/\"default-dockercfg-8zpfr\""
Oct 30 00:54:14.550923 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.550748    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-72ed13\"/\"openshift-service-ca.crt\""
Oct 30 00:54:14.552684 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.552664    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"]
Oct 30 00:54:14.624392 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.624360    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-home\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.624580 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.624412    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp75j\" (UniqueName: \"kubernetes.io/projected/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-kube-api-access-pp75j\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.624580 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.624507    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-tls-certs\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.624699 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.624580    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-model-cache\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.624699 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.624634    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-dshm\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.624699 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.624652    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-tmp-dir\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.725402 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.725366    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp75j\" (UniqueName: \"kubernetes.io/projected/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-kube-api-access-pp75j\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.725402 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.725405    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-tls-certs\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.725649 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.725426    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-model-cache\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.725649 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.725483    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-dshm\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.725649 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.725503    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-tmp-dir\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.725649 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.725562    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-home\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.725929 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.725898    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-model-cache\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.726057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.725952    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-tmp-dir\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.726057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.725988    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-home\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.727688 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.727663    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-dshm\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.727821 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.727803    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-tls-certs\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.754265 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.754242    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp75j\" (UniqueName: \"kubernetes.io/projected/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-kube-api-access-pp75j\") pod \"simulator-w2-72ed13-kserve-554f57dbc7-mxv7z\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") " pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:14.837645 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:14.837567    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:15.140368 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:15.140055    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"]
Oct 30 00:54:15.143437 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:54:15.143407    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4ed47ff_0328_4645_be5f_6a6d51a94fc8.slice/crio-a4997ed4ea51745a47e80e65fe58361c0680f61e07600b22056efde89c3edb3c WatchSource:0}: Error finding container a4997ed4ea51745a47e80e65fe58361c0680f61e07600b22056efde89c3edb3c: Status 404 returned error can't find the container with id a4997ed4ea51745a47e80e65fe58361c0680f61e07600b22056efde89c3edb3c
Oct 30 00:54:15.911488 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:15.911455    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z" event={"ID":"b4ed47ff-0328-4645-be5f-6a6d51a94fc8","Type":"ContainerStarted","Data":"a4997ed4ea51745a47e80e65fe58361c0680f61e07600b22056efde89c3edb3c"}
Oct 30 00:54:16.546538 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:16.546496    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:54:16.917798 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:16.917703    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z" event={"ID":"b4ed47ff-0328-4645-be5f-6a6d51a94fc8","Type":"ContainerStarted","Data":"71280ea5a3fbd2368220d0cdb817112130d7413f6c3f37390d7d5a7c59cb5796"}
Oct 30 00:54:16.956676 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:16.956630    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z" podStartSLOduration=1.8911809800000001 podStartE2EDuration="2.956606551s" totalImagesPullingTime="1.065425571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:16.954957789 +0000 UTC m=+765.704613098" watchObservedRunningTime="2026-10-30 00:54:16.956606551 +0000 UTC m=+765.706261860"
Oct 30 00:54:17.837950 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:17.837913    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:24.838753 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:24.838712    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:24.852333 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:24.852305    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:24.966640 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:24.966610    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:54:26.126255 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.126215    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"]
Oct 30 00:54:26.130340 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.130316    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.140543 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.140514    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-tmp-dir\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.140695 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.140551    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pkww\" (UniqueName: \"kubernetes.io/projected/4d3d278e-ecd0-46ee-a203-33003899a369-kube-api-access-5pkww\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.140695 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.140578    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3d278e-ecd0-46ee-a203-33003899a369-tls-certs\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.140695 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.140673    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-dshm\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.140823 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.140707    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-home\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.140823 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.140726    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-model-cache\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.153881 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.153840    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-39b241\"/\"default-dockercfg-nf2tx\""
Oct 30 00:54:26.154057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.153908    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-39b241\"/\"openshift-service-ca.crt\""
Oct 30 00:54:26.154057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.153918    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-39b241\"/\"kube-root-ca.crt\""
Oct 30 00:54:26.154057 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.153937    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-39b241\"/\"simulator-w3-39b241-kserve-self-signed-certs\""
Oct 30 00:54:26.158598 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.158564    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"]
Oct 30 00:54:26.241795 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.241760    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-dshm\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.241795 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.241806    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-home\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.242067 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.241828    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-model-cache\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.242067 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.241891    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-tmp-dir\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.242067 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.241913    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pkww\" (UniqueName: \"kubernetes.io/projected/4d3d278e-ecd0-46ee-a203-33003899a369-kube-api-access-5pkww\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.242067 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.241934    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3d278e-ecd0-46ee-a203-33003899a369-tls-certs\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.242349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.242298    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-home\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.242349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.242325    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-model-cache\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.242585 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.242564    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-tmp-dir\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.244346 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.244325    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-dshm\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.244797 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.244761    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3d278e-ecd0-46ee-a203-33003899a369-tls-certs\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.263481 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.263445    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pkww\" (UniqueName: \"kubernetes.io/projected/4d3d278e-ecd0-46ee-a203-33003899a369-kube-api-access-5pkww\") pod \"simulator-w3-39b241-kserve-86496dd68c-r5xpw\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") " pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.442074 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.441977    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:26.758044 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.757942    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"]
Oct 30 00:54:26.760604 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:54:26.760575    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d3d278e_ecd0_46ee_a203_33003899a369.slice/crio-dd678f60e3d3bb88413c3e7f17b8b6ceb30d54d58f64d89e0448ebb26eb7fb66 WatchSource:0}: Error finding container dd678f60e3d3bb88413c3e7f17b8b6ceb30d54d58f64d89e0448ebb26eb7fb66: Status 404 returned error can't find the container with id dd678f60e3d3bb88413c3e7f17b8b6ceb30d54d58f64d89e0448ebb26eb7fb66
Oct 30 00:54:26.968007 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.967970    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw" event={"ID":"4d3d278e-ecd0-46ee-a203-33003899a369","Type":"ContainerStarted","Data":"7fd9c865f2a139953b4820d1eae69590fde43ff3c626d9686578af751a1848a8"}
Oct 30 00:54:26.968007 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:26.968013    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw" event={"ID":"4d3d278e-ecd0-46ee-a203-33003899a369","Type":"ContainerStarted","Data":"dd678f60e3d3bb88413c3e7f17b8b6ceb30d54d58f64d89e0448ebb26eb7fb66"}
Oct 30 00:54:27.019264 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:27.019201    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw" podStartSLOduration=1.019181494 podStartE2EDuration="1.019181494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:27.01619623 +0000 UTC m=+775.765851539" watchObservedRunningTime="2026-10-30 00:54:27.019181494 +0000 UTC m=+775.768836800"
Oct 30 00:54:27.442723 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:27.442644    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:29.465685 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:29.465643    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:54:35.733343 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.733308    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"]
Oct 30 00:54:35.740310 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.740279    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.756465 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.756437    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-664c2b\"/\"openshift-service-ca.crt\""
Oct 30 00:54:35.765515 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.765485    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-664c2b\"/\"kube-root-ca.crt\""
Oct 30 00:54:35.765515 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.765489    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-664c2b\"/\"default-dockercfg-p5z47\""
Oct 30 00:54:35.765697 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.765574    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-664c2b\"/\"simulator-w1-664c2b-kserve-self-signed-certs\""
Oct 30 00:54:35.767948 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.767928    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"]
Oct 30 00:54:35.830779 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.830738    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-model-cache\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.830945 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.830786    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-tmp-dir\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.831026 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.830934    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13f688db-25e0-4a88-9d1e-c3fd78b7f659-tls-certs\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.831026 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.830977    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4zdd\" (UniqueName: \"kubernetes.io/projected/13f688db-25e0-4a88-9d1e-c3fd78b7f659-kube-api-access-f4zdd\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.831125 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.831101    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-dshm\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.831125 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.831119    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-home\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.931724 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.931683    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-dshm\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.931724 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.931722    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-home\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.931989 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.931769    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-model-cache\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.931989 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.931789    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-tmp-dir\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.931989 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.931826    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13f688db-25e0-4a88-9d1e-c3fd78b7f659-tls-certs\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.931989 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.931849    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4zdd\" (UniqueName: \"kubernetes.io/projected/13f688db-25e0-4a88-9d1e-c3fd78b7f659-kube-api-access-f4zdd\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.932305 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.932271    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-model-cache\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.932356 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.932313    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-home\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.932356 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.932329    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-tmp-dir\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.934071 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.934050    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-dshm\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.934388 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.934370    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13f688db-25e0-4a88-9d1e-c3fd78b7f659-tls-certs\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:35.952987 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:35.952961    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4zdd\" (UniqueName: \"kubernetes.io/projected/13f688db-25e0-4a88-9d1e-c3fd78b7f659-kube-api-access-f4zdd\") pod \"simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") " pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:36.055353 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:36.055317    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:36.354012 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:36.353815    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"]
Oct 30 00:54:36.355852 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:54:36.355816    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13f688db_25e0_4a88_9d1e_c3fd78b7f659.slice/crio-23f5e141ca0b6f87d77044b423f564af06fb6cf6c3f61b7457ae8b4e3b02be96 WatchSource:0}: Error finding container 23f5e141ca0b6f87d77044b423f564af06fb6cf6c3f61b7457ae8b4e3b02be96: Status 404 returned error can't find the container with id 23f5e141ca0b6f87d77044b423f564af06fb6cf6c3f61b7457ae8b4e3b02be96
Oct 30 00:54:36.443088 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:36.443055    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:36.457556 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:36.457526    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:37.017655 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:37.017615    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2" event={"ID":"13f688db-25e0-4a88-9d1e-c3fd78b7f659","Type":"ContainerStarted","Data":"e3a60e3181e4a5f5f52e0ceb5a8e76bf9fc1d828c354618608640ee44456fd78"}
Oct 30 00:54:37.017655 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:37.017656    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2" event={"ID":"13f688db-25e0-4a88-9d1e-c3fd78b7f659","Type":"ContainerStarted","Data":"23f5e141ca0b6f87d77044b423f564af06fb6cf6c3f61b7457ae8b4e3b02be96"}
Oct 30 00:54:37.029165 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:37.029119    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:54:37.055401 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:37.055344    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2" podStartSLOduration=2.055326507 podStartE2EDuration="2.055326507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:37.054667913 +0000 UTC m=+785.804323223" watchObservedRunningTime="2026-10-30 00:54:37.055326507 +0000 UTC m=+785.804981818"
Oct 30 00:54:37.055609 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:37.055588    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:37.555186 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:54:37.555126    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:54:37.627469 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:37.627432    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:54:42.097200 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.097137    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"]
Oct 30 00:54:42.103015 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.102989    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.128131 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.128101    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-72ed13\"/\"premium-w2-72ed13-kserve-self-signed-certs\""
Oct 30 00:54:42.131832 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.131798    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"]
Oct 30 00:54:42.194250 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.194208    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8d1f900-b8c4-43dc-814e-9491e560ea3f-tls-certs\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.194423 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.194269    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-model-cache\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.194423 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.194307    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-dshm\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.194423 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.194343    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmznk\" (UniqueName: \"kubernetes.io/projected/c8d1f900-b8c4-43dc-814e-9491e560ea3f-kube-api-access-kmznk\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.194423 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.194385    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-home\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.194578 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.194430    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-tmp-dir\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.295120 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.295084    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8d1f900-b8c4-43dc-814e-9491e560ea3f-tls-certs\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.295349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.295139    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-model-cache\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.295349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.295183    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-dshm\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.295349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.295208    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmznk\" (UniqueName: \"kubernetes.io/projected/c8d1f900-b8c4-43dc-814e-9491e560ea3f-kube-api-access-kmznk\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.295349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.295246    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-home\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.295349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.295296    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-tmp-dir\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.295640 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.295615    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-model-cache\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.295715 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.295695    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-home\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.295715 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.295707    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-tmp-dir\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.297824 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.297796    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-dshm\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.298268 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.298247    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8d1f900-b8c4-43dc-814e-9491e560ea3f-tls-certs\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.316530 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.316498    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmznk\" (UniqueName: \"kubernetes.io/projected/c8d1f900-b8c4-43dc-814e-9491e560ea3f-kube-api-access-kmznk\") pod \"premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") " pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.417408 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.417313    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:42.716694 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:42.715961    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"]
Oct 30 00:54:42.719569 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:54:42.719541    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8d1f900_b8c4_43dc_814e_9491e560ea3f.slice/crio-0ab026a093cf666df6998d3effd5932a357c871fb73c5b38e5bc83c213fba5a2 WatchSource:0}: Error finding container 0ab026a093cf666df6998d3effd5932a357c871fb73c5b38e5bc83c213fba5a2: Status 404 returned error can't find the container with id 0ab026a093cf666df6998d3effd5932a357c871fb73c5b38e5bc83c213fba5a2
Oct 30 00:54:43.049880 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:43.049841    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc" event={"ID":"c8d1f900-b8c4-43dc-814e-9491e560ea3f","Type":"ContainerStarted","Data":"dd892509fabdc34976778242280e1b11177b8a51e602f6a706618a784e287036"}
Oct 30 00:54:43.049880 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:43.049889    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc" event={"ID":"c8d1f900-b8c4-43dc-814e-9491e560ea3f","Type":"ContainerStarted","Data":"0ab026a093cf666df6998d3effd5932a357c871fb73c5b38e5bc83c213fba5a2"}
Oct 30 00:54:43.087442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:43.087372    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc" podStartSLOduration=1.087356277 podStartE2EDuration="1.087356277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:43.086402303 +0000 UTC m=+791.836057611" watchObservedRunningTime="2026-10-30 00:54:43.087356277 +0000 UTC m=+791.837011634"
Oct 30 00:54:43.421705 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:43.421620    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:45.406742 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:45.406703    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:54:46.055987 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:46.055950    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:46.069371 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:46.069338    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:46.080840 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:46.080802    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:54:52.418196 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:52.418133    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:52.430539 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:52.430514    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:53.110535 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:53.110500    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:54:59.673043 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.673012    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"]
Oct 30 00:54:59.676887 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.676865    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.689602 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.689580    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d5b2b8\"/\"openshift-service-ca.crt\""
Oct 30 00:54:59.701187 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.699634    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d5b2b8\"/\"default-dockercfg-f9x4g\""
Oct 30 00:54:59.701187 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.699659    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d5b2b8\"/\"premium-w4-d5b2b8-kserve-self-signed-certs\""
Oct 30 00:54:59.701187 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.699802    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d5b2b8\"/\"kube-root-ca.crt\""
Oct 30 00:54:59.701898 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.701869    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"]
Oct 30 00:54:59.767122 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.767082    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-tmp-dir\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.767312 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.767134    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-home\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.767312 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.767176    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-dshm\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.767312 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.767208    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-model-cache\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.767312 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.767233    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npcqd\" (UniqueName: \"kubernetes.io/projected/66a19982-1f16-4a03-a629-bf9b7d6b50d3-kube-api-access-npcqd\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.767312 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.767291    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66a19982-1f16-4a03-a629-bf9b7d6b50d3-tls-certs\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.868657 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.868620    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-tmp-dir\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.868657 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.868662    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-home\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.868921 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.868677    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-dshm\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.868921 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.868695    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-model-cache\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.868921 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.868718    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npcqd\" (UniqueName: \"kubernetes.io/projected/66a19982-1f16-4a03-a629-bf9b7d6b50d3-kube-api-access-npcqd\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.868921 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.868791    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66a19982-1f16-4a03-a629-bf9b7d6b50d3-tls-certs\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.869293 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.869240    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-model-cache\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.869293 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.869268    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-home\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.869568 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.869543    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-tmp-dir\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.872002 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.871978    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66a19982-1f16-4a03-a629-bf9b7d6b50d3-tls-certs\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.872343 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.872319    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-dshm\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.890396 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.890372    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npcqd\" (UniqueName: \"kubernetes.io/projected/66a19982-1f16-4a03-a629-bf9b7d6b50d3-kube-api-access-npcqd\") pod \"premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") " pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:54:59.989902 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:54:59.989819    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:55:00.293439 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:00.293347    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"]
Oct 30 00:55:00.295942 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:55:00.295906    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66a19982_1f16_4a03_a629_bf9b7d6b50d3.slice/crio-ac7be6d725b3206e2181ee61ac791ac1ce867954295b4ba5fbc0ab833adb2869 WatchSource:0}: Error finding container ac7be6d725b3206e2181ee61ac791ac1ce867954295b4ba5fbc0ab833adb2869: Status 404 returned error can't find the container with id ac7be6d725b3206e2181ee61ac791ac1ce867954295b4ba5fbc0ab833adb2869
Oct 30 00:55:01.141875 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:01.141839    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w" event={"ID":"66a19982-1f16-4a03-a629-bf9b7d6b50d3","Type":"ContainerStarted","Data":"45b78d12e45113ece8bb9b46515e5c2aee4dca04a368c534d048dd9b81219e45"}
Oct 30 00:55:01.142278 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:01.141885    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w" event={"ID":"66a19982-1f16-4a03-a629-bf9b7d6b50d3","Type":"ContainerStarted","Data":"ac7be6d725b3206e2181ee61ac791ac1ce867954295b4ba5fbc0ab833adb2869"}
Oct 30 00:55:01.178494 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:01.178449    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w" podStartSLOduration=2.178435436 podStartE2EDuration="2.178435436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:01.177725837 +0000 UTC m=+809.927381158" watchObservedRunningTime="2026-10-30 00:55:01.178435436 +0000 UTC m=+809.928090745"
Oct 30 00:55:01.990579 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:01.990542    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:55:03.287606 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:03.287571    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:55:07.553838 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:55:07.553805    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:55:09.326424 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.326388    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"]
Oct 30 00:55:09.330780 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.330755    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.344970 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.344942    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9bfe36\"/\"openshift-service-ca.crt\""
Oct 30 00:55:09.354608 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.354584    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9bfe36\"/\"simulator-w0-9bfe36-kserve-self-signed-certs\""
Oct 30 00:55:09.354718 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.354608    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9bfe36\"/\"default-dockercfg-phnl6\""
Oct 30 00:55:09.354718 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.354648    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9bfe36\"/\"kube-root-ca.crt\""
Oct 30 00:55:09.356449 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.356426    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"]
Oct 30 00:55:09.459996 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.459953    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-dshm\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.460213 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.460009    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-model-cache\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.460213 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.460177    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-home\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.460348 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.460234    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d97kg\" (UniqueName: \"kubernetes.io/projected/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-kube-api-access-d97kg\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.460348 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.460277    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-tls-certs\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.460348 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.460324    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-tmp-dir\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.561279 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.561233    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d97kg\" (UniqueName: \"kubernetes.io/projected/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-kube-api-access-d97kg\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.561467 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.561286    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-tls-certs\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.561467 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.561339    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-tmp-dir\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.561467 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.561399    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-dshm\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.561467 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.561436    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-model-cache\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.561689 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.561535    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-home\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.561841 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.561811    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-tmp-dir\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.561994 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.561967    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-home\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.561994 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.561983    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-model-cache\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.563735 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.563713    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-dshm\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.564053 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.564035    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-tls-certs\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.582531 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.582473    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d97kg\" (UniqueName: \"kubernetes.io/projected/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-kube-api-access-d97kg\") pod \"simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") " pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.645651 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.645619    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:09.942346 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.942314    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"]
Oct 30 00:55:09.944744 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:55:09.944716    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8b27e40_2516_4d18_8c84_5e0925b3f4d3.slice/crio-49b49b291239bffbb813a19aed6deb69860f697e977e43e5679d9a8508fd49f8 WatchSource:0}: Error finding container 49b49b291239bffbb813a19aed6deb69860f697e977e43e5679d9a8508fd49f8: Status 404 returned error can't find the container with id 49b49b291239bffbb813a19aed6deb69860f697e977e43e5679d9a8508fd49f8
Oct 30 00:55:09.990220 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:09.990191    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:55:10.003118 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:10.003097    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:55:10.187634 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:10.187595    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s" event={"ID":"b8b27e40-2516-4d18-8c84-5e0925b3f4d3","Type":"ContainerStarted","Data":"e3f65bdc61c5ec08e2a303255fb947d0cb3afdf8704bde493afbddd9f7edb1f4"}
Oct 30 00:55:10.187634 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:10.187639    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s" event={"ID":"b8b27e40-2516-4d18-8c84-5e0925b3f4d3","Type":"ContainerStarted","Data":"49b49b291239bffbb813a19aed6deb69860f697e977e43e5679d9a8508fd49f8"}
Oct 30 00:55:10.199260 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:10.199200    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:55:10.225564 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:10.225516    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s" podStartSLOduration=1.225502518 podStartE2EDuration="1.225502518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:55:10.223568426 +0000 UTC m=+818.973223735" watchObservedRunningTime="2026-10-30 00:55:10.225502518 +0000 UTC m=+818.975157821"
Oct 30 00:55:10.645753 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:10.645721    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:19.645968 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:19.645933    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:19.660527 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:19.660496    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:20.261173 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:20.261137    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:55:23.462316 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:23.462282    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:55:37.553307 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:55:37.553251    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:55:39.722503 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:55:39.722467    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:56:05.736082 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:05.736046    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:56:07.553785 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:56:07.553686    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:56:13.616148 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:13.616110    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"]
Oct 30 00:56:13.619387 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:13.619366    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"
Oct 30 00:56:13.641392 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:13.641364    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:56:13.641515 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:13.641446    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:56:13.641592 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:13.641526    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 00:56:13.641592 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:13.641374    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:56:13.643456 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:13.643431    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"]
Oct 30 00:56:13.659987 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:13.659960    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmc98\" (UniqueName: \"kubernetes.io/projected/7164c599-d510-4d81-96f9-c8d14ab111b9-kube-api-access-tmc98\") pod \"maas-api-revoke-sub-4b1406e8-nc7gk\" (UID: \"7164c599-d510-4d81-96f9-c8d14ab111b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"
Oct 30 00:56:13.660105 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:13.660078    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7164c599-d510-4d81-96f9-c8d14ab111b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b1406e8-nc7gk\" (UID: \"7164c599-d510-4d81-96f9-c8d14ab111b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"
Oct 30 00:56:13.660207 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:13.660121    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7164c599-d510-4d81-96f9-c8d14ab111b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b1406e8-nc7gk\" (UID: \"7164c599-d510-4d81-96f9-c8d14ab111b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"
Oct 30 00:56:13.760958 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:13.760932    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmc98\" (UniqueName: \"kubernetes.io/projected/7164c599-d510-4d81-96f9-c8d14ab111b9-kube-api-access-tmc98\") pod \"maas-api-revoke-sub-4b1406e8-nc7gk\" (UID: \"7164c599-d510-4d81-96f9-c8d14ab111b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"
Oct 30 00:56:13.761109 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:13.760990    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7164c599-d510-4d81-96f9-c8d14ab111b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b1406e8-nc7gk\" (UID: \"7164c599-d510-4d81-96f9-c8d14ab111b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"
Oct 30 00:56:13.761109 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:13.761024    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7164c599-d510-4d81-96f9-c8d14ab111b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b1406e8-nc7gk\" (UID: \"7164c599-d510-4d81-96f9-c8d14ab111b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"
Oct 30 00:56:13.761607 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:13.761581    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7164c599-d510-4d81-96f9-c8d14ab111b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b1406e8-nc7gk\" (UID: \"7164c599-d510-4d81-96f9-c8d14ab111b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"
Oct 30 00:56:13.763386 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:13.763363    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7164c599-d510-4d81-96f9-c8d14ab111b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b1406e8-nc7gk\" (UID: \"7164c599-d510-4d81-96f9-c8d14ab111b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"
Oct 30 00:56:13.782118 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:13.782091    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmc98\" (UniqueName: \"kubernetes.io/projected/7164c599-d510-4d81-96f9-c8d14ab111b9-kube-api-access-tmc98\") pod \"maas-api-revoke-sub-4b1406e8-nc7gk\" (UID: \"7164c599-d510-4d81-96f9-c8d14ab111b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"
Oct 30 00:56:13.934361 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:13.934284    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"
Oct 30 00:56:14.233768 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:14.233612    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"]
Oct 30 00:56:14.507654 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:14.507593    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk" event={"ID":"7164c599-d510-4d81-96f9-c8d14ab111b9","Type":"ContainerStarted","Data":"69bc971b8277b0c9ba4856752eeda5eccdccd5be51e176867002322c67f2a53e"}
Oct 30 00:56:14.507654 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:14.507647    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk" event={"ID":"7164c599-d510-4d81-96f9-c8d14ab111b9","Type":"ContainerStarted","Data":"b3cfc4d078db4f81a6c0a45a3f22e15246529862bc21249f3b91a492736c7cc6"}
Oct 30 00:56:14.543810 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:14.543767    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk" podStartSLOduration=1.543755085 podStartE2EDuration="1.543755085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:14.54347186 +0000 UTC m=+883.293127180" watchObservedRunningTime="2026-10-30 00:56:14.543755085 +0000 UTC m=+883.293410390"
Oct 30 00:56:15.514579 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:15.514542    4425 generic.go:356] "Generic (PLEG): container finished" podID="7164c599-d510-4d81-96f9-c8d14ab111b9" containerID="69bc971b8277b0c9ba4856752eeda5eccdccd5be51e176867002322c67f2a53e" exitCode=0
Oct 30 00:56:15.514943 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:15.514634    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk" event={"ID":"7164c599-d510-4d81-96f9-c8d14ab111b9","Type":"ContainerDied","Data":"69bc971b8277b0c9ba4856752eeda5eccdccd5be51e176867002322c67f2a53e"}
Oct 30 00:56:16.658009 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:16.657984    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"
Oct 30 00:56:16.687845 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:16.687818    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7164c599-d510-4d81-96f9-c8d14ab111b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7164c599-d510-4d81-96f9-c8d14ab111b9-maas-api-openshift-ca\") pod \"7164c599-d510-4d81-96f9-c8d14ab111b9\" (UID: \"7164c599-d510-4d81-96f9-c8d14ab111b9\") "
Oct 30 00:56:16.687978 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:16.687876    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7164c599-d510-4d81-96f9-c8d14ab111b9-kube-api-access-tmc98\" (UniqueName: \"kubernetes.io/projected/7164c599-d510-4d81-96f9-c8d14ab111b9-kube-api-access-tmc98\") pod \"7164c599-d510-4d81-96f9-c8d14ab111b9\" (UID: \"7164c599-d510-4d81-96f9-c8d14ab111b9\") "
Oct 30 00:56:16.688067 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:16.688046    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7164c599-d510-4d81-96f9-c8d14ab111b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7164c599-d510-4d81-96f9-c8d14ab111b9-maas-api-xks-ca\") pod \"7164c599-d510-4d81-96f9-c8d14ab111b9\" (UID: \"7164c599-d510-4d81-96f9-c8d14ab111b9\") "
Oct 30 00:56:16.688229 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:16.688207    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7164c599-d510-4d81-96f9-c8d14ab111b9-maas-api-openshift-ca" pod "7164c599-d510-4d81-96f9-c8d14ab111b9" (UID: "7164c599-d510-4d81-96f9-c8d14ab111b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:56:16.688405 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:16.688385    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7164c599-d510-4d81-96f9-c8d14ab111b9-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:56:16.689966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:16.689944    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7164c599-d510-4d81-96f9-c8d14ab111b9-kube-api-access-tmc98" pod "7164c599-d510-4d81-96f9-c8d14ab111b9" (UID: "7164c599-d510-4d81-96f9-c8d14ab111b9"). InnerVolumeSpecName "kube-api-access-tmc98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:16.689966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:16.689951    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7164c599-d510-4d81-96f9-c8d14ab111b9-maas-api-xks-ca" pod "7164c599-d510-4d81-96f9-c8d14ab111b9" (UID: "7164c599-d510-4d81-96f9-c8d14ab111b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:56:16.789640 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:16.789581    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmc98\" (UniqueName: \"kubernetes.io/projected/7164c599-d510-4d81-96f9-c8d14ab111b9-kube-api-access-tmc98\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:56:16.789640 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:16.789604    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7164c599-d510-4d81-96f9-c8d14ab111b9-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:56:17.526176 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:17.526131    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"
Oct 30 00:56:17.526344 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:17.526200    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk" event={"ID":"7164c599-d510-4d81-96f9-c8d14ab111b9","Type":"ContainerDied","Data":"b3cfc4d078db4f81a6c0a45a3f22e15246529862bc21249f3b91a492736c7cc6"}
Oct 30 00:56:17.526344 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:17.526247    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3cfc4d078db4f81a6c0a45a3f22e15246529862bc21249f3b91a492736c7cc6"
Oct 30 00:56:17.970292 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:17.970261    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"]
Oct 30 00:56:17.970768 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:17.970663    4425 cpu_manager.go:414] "containerMap: removing container" podUID="7164c599-d510-4d81-96f9-c8d14ab111b9" containerName="revoke-keys"
Oct 30 00:56:17.970768 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:17.970674    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7164c599-d510-4d81-96f9-c8d14ab111b9" containerName="revoke-keys"
Oct 30 00:56:17.970887 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:17.970781    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7164c599-d510-4d81-96f9-c8d14ab111b9" containerName="revoke-keys"
Oct 30 00:56:17.974171 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:17.974142    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:17.988206 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:17.988185    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-72ed13-openshift-default-dockercfg-tjgpb\""
Oct 30 00:56:18.000389 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.000368    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"]
Oct 30 00:56:18.000674 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.000650    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.000780 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.000690    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.000780 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.000726    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.000936 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.000913    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.000992 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.000974    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.001044 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.001008    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.001142 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.001127    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.001230 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.001190    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.001230 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.001210    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfshk\" (UniqueName: \"kubernetes.io/projected/8d56cb59-ad4a-47ac-9107-5160fe97dfad-kube-api-access-vfshk\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.102040 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.102008    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.102282 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.102058    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.102282 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.102076    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfshk\" (UniqueName: \"kubernetes.io/projected/8d56cb59-ad4a-47ac-9107-5160fe97dfad-kube-api-access-vfshk\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.102405 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.102381    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.102486 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.102430    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.102486 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.102471    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.102589 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.102522    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.102589 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.102559    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.102589 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.102584    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.102717 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.102608    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.103043 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.103019    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.103148 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.103049    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.103148 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.103079    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.103340 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.103318    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.104725 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.104698    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.105230 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.105214    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.122868 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.122838    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfshk\" (UniqueName: \"kubernetes.io/projected/8d56cb59-ad4a-47ac-9107-5160fe97dfad-kube-api-access-vfshk\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.122868 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.122844    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.288018 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.287996    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:18.600861 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.600832    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"]
Oct 30 00:56:18.603147 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:56:18.603120    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d56cb59_ad4a_47ac_9107_5160fe97dfad.slice/crio-09a96230cbcbebe64c67f2e587a9fad2ff46684624fa0c92f8a20accebf54c75 WatchSource:0}: Error finding container 09a96230cbcbebe64c67f2e587a9fad2ff46684624fa0c92f8a20accebf54c75: Status 404 returned error can't find the container with id 09a96230cbcbebe64c67f2e587a9fad2ff46684624fa0c92f8a20accebf54c75
Oct 30 00:56:18.605445 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.605410    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:18.605550 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.605495    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:18.605550 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:18.605525    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:19.538094 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:19.538060    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt" event={"ID":"8d56cb59-ad4a-47ac-9107-5160fe97dfad","Type":"ContainerStarted","Data":"68972de5501186f50b967e0886df7b397fd5dfcc88de0f3292adac2fe54269c7"}
Oct 30 00:56:19.538554 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:19.538113    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt" event={"ID":"8d56cb59-ad4a-47ac-9107-5160fe97dfad","Type":"ContainerStarted","Data":"09a96230cbcbebe64c67f2e587a9fad2ff46684624fa0c92f8a20accebf54c75"}
Oct 30 00:56:19.619109 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:19.619058    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt" podStartSLOduration=2.619046119 podStartE2EDuration="2.619046119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:19.608989639 +0000 UTC m=+888.358644949" watchObservedRunningTime="2026-10-30 00:56:19.619046119 +0000 UTC m=+888.368701487"
Oct 30 00:56:20.289175 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:20.289128    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:20.290534 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:20.290513    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:56:20.290597 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:20.290561    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt" podUID="8d56cb59-ad4a-47ac-9107-5160fe97dfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 00:56:21.288327 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:21.288292    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:56:21.288805 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:21.288349    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt" podUID="8d56cb59-ad4a-47ac-9107-5160fe97dfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 00:56:22.288667 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:22.288635    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:56:22.289094 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:22.288682    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt" podUID="8d56cb59-ad4a-47ac-9107-5160fe97dfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 00:56:23.288402 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:23.288370    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:56:23.288602 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:23.288424    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt" podUID="8d56cb59-ad4a-47ac-9107-5160fe97dfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 00:56:24.289070 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:24.289028    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Oct 30 00:56:24.289494 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:24.289090    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt" podUID="8d56cb59-ad4a-47ac-9107-5160fe97dfad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Oct 30 00:56:25.292244 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:25.292213    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:25.292641 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:25.292492    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:25.293210 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:25.293191    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 00:56:37.555249 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:56:37.555128    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:56:39.208278 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:39.208245    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:56:51.338079 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.338037    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"]
Oct 30 00:56:51.342506 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.342484    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.364503 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.364479    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9bfe36\"/\"distinct-w0-9bfe36-kserve-self-signed-certs\""
Oct 30 00:56:51.366955 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.366930    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"]
Oct 30 00:56:51.455275 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.455245    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834166ed-5c63-4854-9e89-c45041553630-tls-certs\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.455442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.455308    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f8k2\" (UniqueName: \"kubernetes.io/projected/834166ed-5c63-4854-9e89-c45041553630-kube-api-access-8f8k2\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.455442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.455417    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-dshm\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.455549 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.455459    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-model-cache\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.455604 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.455592    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-tmp-dir\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.455657 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.455634    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-home\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.462887 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.462861    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"]
Oct 30 00:56:51.466919 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.466900    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.481246 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.481221    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-39b241-openshift-default-dockercfg-bchvq\""
Oct 30 00:56:51.494994 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.494971    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"]
Oct 30 00:56:51.556606 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.556570    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.556787 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.556645    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.556787 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.556715    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.556906 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.556810    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-tmp-dir\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.556906 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.556888    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.557011 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.556934    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-home\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.557072 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.557038    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834166ed-5c63-4854-9e89-c45041553630-tls-certs\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.557130 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.557098    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.557222 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.557184    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f8k2\" (UniqueName: \"kubernetes.io/projected/834166ed-5c63-4854-9e89-c45041553630-kube-api-access-8f8k2\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.557281 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.557227    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.557281 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.557251    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-tmp-dir\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.557281 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.557251    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbvrx\" (UniqueName: \"kubernetes.io/projected/251fdcfb-b27b-4f4e-854b-9f3f70304ace-kube-api-access-kbvrx\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.557580 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.557558    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-home\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.557750 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.557726    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.557981 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.557946    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.558117 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.557984    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-dshm\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.558117 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.558072    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-model-cache\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.558463 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.558438    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-model-cache\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.561840 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.561818    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834166ed-5c63-4854-9e89-c45041553630-tls-certs\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.562721 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.562695    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-dshm\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.597480 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.597414    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f8k2\" (UniqueName: \"kubernetes.io/projected/834166ed-5c63-4854-9e89-c45041553630-kube-api-access-8f8k2\") pod \"distinct-w0-9bfe36-kserve-bdb675f8-k2dj7\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.655205 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.655186    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:51.659650 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.659627    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.659735 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.659693    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.659735 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.659720    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbvrx\" (UniqueName: \"kubernetes.io/projected/251fdcfb-b27b-4f4e-854b-9f3f70304ace-kube-api-access-kbvrx\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.659847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.659749    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.659847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.659813    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.659950 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.659884    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.659950 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.659935    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.660074 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.660054    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.660134 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.660098    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.660285 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.660257    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.660361 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.660290    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.660527 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.660497    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.660628 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.660607    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.660628 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.660609    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.662028 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.662003    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.662388 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.662371    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.715248 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.712615    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.715248 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.713793    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbvrx\" (UniqueName: \"kubernetes.io/projected/251fdcfb-b27b-4f4e-854b-9f3f70304ace-kube-api-access-kbvrx\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.783435 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.783408    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:51.955823 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:51.955547    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"]
Oct 30 00:56:51.957550 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:56:51.957521    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod834166ed_5c63_4854_9e89_c45041553630.slice/crio-ed73f89f1ad4427985f670a7c69f855663db32459eba6b59441610329f97404f WatchSource:0}: Error finding container ed73f89f1ad4427985f670a7c69f855663db32459eba6b59441610329f97404f: Status 404 returned error can't find the container with id ed73f89f1ad4427985f670a7c69f855663db32459eba6b59441610329f97404f
Oct 30 00:56:52.084531 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:52.084509    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"]
Oct 30 00:56:52.085749 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:56:52.085722    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251fdcfb_b27b_4f4e_854b_9f3f70304ace.slice/crio-a86313568a587a86d9f96ea2aeae02910ecc3cebec6c31e79e90de04db0ccebd WatchSource:0}: Error finding container a86313568a587a86d9f96ea2aeae02910ecc3cebec6c31e79e90de04db0ccebd: Status 404 returned error can't find the container with id a86313568a587a86d9f96ea2aeae02910ecc3cebec6c31e79e90de04db0ccebd
Oct 30 00:56:52.088111 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:52.088079    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:52.088259 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:52.088144    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:52.088259 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:52.088195    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:56:52.736822 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:52.736783    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx" event={"ID":"251fdcfb-b27b-4f4e-854b-9f3f70304ace","Type":"ContainerStarted","Data":"cd20f8e9fa01f73b334530f7f24afffbaf97f5d60cb564880285bc4476ab2058"}
Oct 30 00:56:52.737302 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:52.736839    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx" event={"ID":"251fdcfb-b27b-4f4e-854b-9f3f70304ace","Type":"ContainerStarted","Data":"a86313568a587a86d9f96ea2aeae02910ecc3cebec6c31e79e90de04db0ccebd"}
Oct 30 00:56:52.738420 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:52.738395    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7" event={"ID":"834166ed-5c63-4854-9e89-c45041553630","Type":"ContainerStarted","Data":"1d9fcf60190de75e8075d350084589723f3ee8ea276930870f980a0161106997"}
Oct 30 00:56:52.738537 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:52.738424    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7" event={"ID":"834166ed-5c63-4854-9e89-c45041553630","Type":"ContainerStarted","Data":"ed73f89f1ad4427985f670a7c69f855663db32459eba6b59441610329f97404f"}
Oct 30 00:56:52.783831 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:52.783794    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:52.806603 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:52.806552    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx" podStartSLOduration=1.806538475 podStartE2EDuration="1.806538475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:52.795675285 +0000 UTC m=+921.545330594" watchObservedRunningTime="2026-10-30 00:56:52.806538475 +0000 UTC m=+921.556193783"
Oct 30 00:56:52.832146 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:52.832101    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7" podStartSLOduration=1.8320904310000001 podStartE2EDuration="1.832090431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:56:52.831898058 +0000 UTC m=+921.581553366" watchObservedRunningTime="2026-10-30 00:56:52.832090431 +0000 UTC m=+921.581745739"
Oct 30 00:56:53.667181 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:53.666248    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:56:53.785344 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:53.785308    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 00:56:53.785881 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:53.785369    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx" podUID="251fdcfb-b27b-4f4e-854b-9f3f70304ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 00:56:54.783930 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:54.783895    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 00:56:54.784097 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:54.783946    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx" podUID="251fdcfb-b27b-4f4e-854b-9f3f70304ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 00:56:55.783961 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:55.783928    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 00:56:55.784394 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:55.783982    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx" podUID="251fdcfb-b27b-4f4e-854b-9f3f70304ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 00:56:56.198226 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:56.198132    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:56:56.784262 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:56.784229    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 00:56:56.784685 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:56.784279    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx" podUID="251fdcfb-b27b-4f4e-854b-9f3f70304ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 00:56:57.783908 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:57.783873    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 00:56:57.784350 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:57.784318    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx" podUID="251fdcfb-b27b-4f4e-854b-9f3f70304ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 00:56:58.982068 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:58.982029    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:58.982560 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:58.982537    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:56:58.983230 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:56:58.983206    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 00:57:01.655966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:01.655936    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:57:01.668537 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:01.668512    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:57:01.819732 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:01.819702    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:57:07.553684 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:57:07.553653    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:57:12.550983 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.550949    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"]
Oct 30 00:57:12.555929 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.555908    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.569905 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.569883    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d5b2b8-openshift-default-dockercfg-br5n4\""
Oct 30 00:57:12.587313 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.587286    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"]
Oct 30 00:57:12.670899 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.670872    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be82dda6-937b-4ab8-bf48-1b3471845c9a-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.671042 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.670910    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.671042 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.670931    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.671042 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.671003    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq684\" (UniqueName: \"kubernetes.io/projected/be82dda6-937b-4ab8-bf48-1b3471845c9a-kube-api-access-zq684\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.671240 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.671053    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.671240 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.671137    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.671347 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.671246    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.671347 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.671313    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.671415 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.671388    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.772225 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.772199    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be82dda6-937b-4ab8-bf48-1b3471845c9a-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.772321 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.772245    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.772321 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.772275    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.772321 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.772309    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq684\" (UniqueName: \"kubernetes.io/projected/be82dda6-937b-4ab8-bf48-1b3471845c9a-kube-api-access-zq684\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.772486 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.772336    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.772486 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.772378    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.772486 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.772422    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.772486 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.772477    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.772683 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.772515    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.772855 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.772803    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.773408 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.772858    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.773408 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.772902    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.773408 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.772994    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.773408 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.773040    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be82dda6-937b-4ab8-bf48-1b3471845c9a-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.774789 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.774765    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.775073 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.775047    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.792876 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.792852    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.793288 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.793270    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq684\" (UniqueName: \"kubernetes.io/projected/be82dda6-937b-4ab8-bf48-1b3471845c9a-kube-api-access-zq684\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:12.869220 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:12.869150    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:13.186375 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:13.186127    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"]
Oct 30 00:57:13.188490 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:57:13.188452    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe82dda6_937b_4ab8_bf48_1b3471845c9a.slice/crio-48c466f0d72f07295fb27ec121f2db568719ad35d22561ef52a4544f0762a3a5 WatchSource:0}: Error finding container 48c466f0d72f07295fb27ec121f2db568719ad35d22561ef52a4544f0762a3a5: Status 404 returned error can't find the container with id 48c466f0d72f07295fb27ec121f2db568719ad35d22561ef52a4544f0762a3a5
Oct 30 00:57:13.191535 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:13.191493    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:13.191636 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:13.191581    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:13.191636 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:13.191623    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:13.879667 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:13.879621    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg" event={"ID":"be82dda6-937b-4ab8-bf48-1b3471845c9a","Type":"ContainerStarted","Data":"96da4152bcc19c5aa137686d2562a071a72f0b4e362d65cc7cb06dae5102dab8"}
Oct 30 00:57:13.880035 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:13.879882    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg" event={"ID":"be82dda6-937b-4ab8-bf48-1b3471845c9a","Type":"ContainerStarted","Data":"48c466f0d72f07295fb27ec121f2db568719ad35d22561ef52a4544f0762a3a5"}
Oct 30 00:57:13.940271 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:13.940216    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg" podStartSLOduration=1.940201581 podStartE2EDuration="1.940201581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:13.929387231 +0000 UTC m=+942.679042541" watchObservedRunningTime="2026-10-30 00:57:13.940201581 +0000 UTC m=+942.689856890"
Oct 30 00:57:14.874262 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:14.870459    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:14.875506 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:14.875479    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:57:14.875734 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:14.875708    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg" podUID="be82dda6-937b-4ab8-bf48-1b3471845c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:57:15.874567 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:15.874535    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:57:15.874967 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:15.874589    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg" podUID="be82dda6-937b-4ab8-bf48-1b3471845c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:57:16.176382 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:16.176299    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:57:16.873609 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:16.873574    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:57:16.873916 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:16.873885    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg" podUID="be82dda6-937b-4ab8-bf48-1b3471845c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:57:17.870179 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:17.869511    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:57:17.870179 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:17.869572    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg" podUID="be82dda6-937b-4ab8-bf48-1b3471845c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:57:18.874481 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:18.874446    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:57:18.874856 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:18.874525    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg" podUID="be82dda6-937b-4ab8-bf48-1b3471845c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:57:19.870456 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:19.870423    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:57:19.870615 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:19.870484    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg" podUID="be82dda6-937b-4ab8-bf48-1b3471845c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:57:20.873862 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:20.873827    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:20.874276 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:20.874193    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:20.874855 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:20.874837    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 00:57:29.783756 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.778402    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"]
Oct 30 00:57:29.784525 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.784489    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.810775 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.810750    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9bfe36\"/\"distinct-2-w0-9bfe36-kserve-self-signed-certs\""
Oct 30 00:57:29.814674 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.814649    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"]
Oct 30 00:57:29.843644 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.843613    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gthgp\" (UniqueName: \"kubernetes.io/projected/0824ed99-cf93-4047-b19f-0ad807263acf-kube-api-access-gthgp\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.843778 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.843752    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-dshm\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.843847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.843809    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-model-cache\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.843847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.843844    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-tmp-dir\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.843950 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.843878    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0824ed99-cf93-4047-b19f-0ad807263acf-tls-certs\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.843950 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.843942    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-home\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.944945 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.944906    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-home\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.945138 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.944970    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gthgp\" (UniqueName: \"kubernetes.io/projected/0824ed99-cf93-4047-b19f-0ad807263acf-kube-api-access-gthgp\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.945138 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.945026    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-dshm\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.945138 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.945106    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-model-cache\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.945363 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.945148    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-tmp-dir\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.945363 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.945213    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0824ed99-cf93-4047-b19f-0ad807263acf-tls-certs\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.945470 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.945405    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-home\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.945753 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.945721    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-tmp-dir\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.945973 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.945921    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-model-cache\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.948647 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.948205    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-dshm\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.948647 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.948448    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0824ed99-cf93-4047-b19f-0ad807263acf-tls-certs\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:29.967737 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:29.967712    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gthgp\" (UniqueName: \"kubernetes.io/projected/0824ed99-cf93-4047-b19f-0ad807263acf-kube-api-access-gthgp\") pod \"distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") " pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:30.097391 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:30.097311    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:30.393001 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:30.392974    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"]
Oct 30 00:57:30.395848 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:57:30.395817    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0824ed99_cf93_4047_b19f_0ad807263acf.slice/crio-619cff782a4c0aee4b77c29336f40bf5f8478ec544c360962e163158ce8cf28c WatchSource:0}: Error finding container 619cff782a4c0aee4b77c29336f40bf5f8478ec544c360962e163158ce8cf28c: Status 404 returned error can't find the container with id 619cff782a4c0aee4b77c29336f40bf5f8478ec544c360962e163158ce8cf28c
Oct 30 00:57:30.974407 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:30.974317    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f" event={"ID":"0824ed99-cf93-4047-b19f-0ad807263acf","Type":"ContainerStarted","Data":"b76b959e601a3ff544ff0f9b10cf6f78e86389b1f64eb367563f787bb3be6769"}
Oct 30 00:57:30.974407 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:30.974373    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f" event={"ID":"0824ed99-cf93-4047-b19f-0ad807263acf","Type":"ContainerStarted","Data":"619cff782a4c0aee4b77c29336f40bf5f8478ec544c360962e163158ce8cf28c"}
Oct 30 00:57:31.013927 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:31.013866    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f" podStartSLOduration=2.013851342 podStartE2EDuration="2.013851342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:31.01185141 +0000 UTC m=+959.761506718" watchObservedRunningTime="2026-10-30 00:57:31.013851342 +0000 UTC m=+959.763506655"
Oct 30 00:57:31.098179 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:31.098146    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:36.134026 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:36.133990    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:57:37.554270 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:57:37.554220    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:57:40.097817 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:40.097781    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:40.110863 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:40.110841    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:41.037170 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:41.037128    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:57:50.383582 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.383545    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"]
Oct 30 00:57:50.388064 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.388040    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.400626 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.400606    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9bfe36-openshift-default-dockercfg-4n5g9\""
Oct 30 00:57:50.414250 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.414227    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"]
Oct 30 00:57:50.449803 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.449772    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.449803 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.449807    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.449980 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.449822    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.449980 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.449958    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.450064 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.449998    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.450064 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.450052    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.450136 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.450084    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.450202 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.450144    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.450241 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.450222    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdx2f\" (UniqueName: \"kubernetes.io/projected/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-kube-api-access-sdx2f\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.551801 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.551773    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.551801 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.551800    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.551988 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.551818    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.551988 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.551870    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.551988 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.551899    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.551988 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.551931    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.551988 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.551973    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.552422 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.552013    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.552422 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.552073    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdx2f\" (UniqueName: \"kubernetes.io/projected/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-kube-api-access-sdx2f\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.552422 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.552245    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.552593 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.552522    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.552593 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.552518    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.552678 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.552590    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.552678 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.552604    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.555037 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.554982    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.555698 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.555640    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.574527 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.574502    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdx2f\" (UniqueName: \"kubernetes.io/projected/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-kube-api-access-sdx2f\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.574745 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.574722    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:50.700906 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:50.700841    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:51.012886 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:51.012861    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"]
Oct 30 00:57:51.015279 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:57:51.015250    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd21ddd7_1c9b_4543_9c58_b1656798e9ef.slice/crio-86a8b925e567e8a9cafdeb6f64f2fdec7cc204af28420dc73f75c314634800a3 WatchSource:0}: Error finding container 86a8b925e567e8a9cafdeb6f64f2fdec7cc204af28420dc73f75c314634800a3: Status 404 returned error can't find the container with id 86a8b925e567e8a9cafdeb6f64f2fdec7cc204af28420dc73f75c314634800a3
Oct 30 00:57:51.017609 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:51.017572    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:51.017704 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:51.017637    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:51.017704 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:51.017667    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:57:51.080213 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:51.080188    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg" event={"ID":"fd21ddd7-1c9b-4543-9c58-b1656798e9ef","Type":"ContainerStarted","Data":"86a8b925e567e8a9cafdeb6f64f2fdec7cc204af28420dc73f75c314634800a3"}
Oct 30 00:57:52.087058 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:52.087021    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg" event={"ID":"fd21ddd7-1c9b-4543-9c58-b1656798e9ef","Type":"ContainerStarted","Data":"9b23b87592b851433ec0016cbe7d9fb8e24e5f9c3a337482328ae8b30691a438"}
Oct 30 00:57:52.146381 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:52.146331    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg" podStartSLOduration=2.146316665 podStartE2EDuration="2.146316665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:52.136449952 +0000 UTC m=+980.886105261" watchObservedRunningTime="2026-10-30 00:57:52.146316665 +0000 UTC m=+980.895971974"
Oct 30 00:57:52.701572 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:52.701542    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:52.703117 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:52.703096    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:57:52.703196 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:52.703142    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg" podUID="fd21ddd7-1c9b-4543-9c58-b1656798e9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 00:57:53.701903 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:53.701865    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:57:53.702344 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:53.701925    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg" podUID="fd21ddd7-1c9b-4543-9c58-b1656798e9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 00:57:54.701546 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:54.701513    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:57:54.701729 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:54.701564    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg" podUID="fd21ddd7-1c9b-4543-9c58-b1656798e9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 00:57:55.701531 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:55.701492    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:57:55.701917 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:55.701549    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg" podUID="fd21ddd7-1c9b-4543-9c58-b1656798e9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 00:57:56.674221 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:56.662015    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:57:56.701102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:56.701064    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:57:56.701260 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:56.701126    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg" podUID="fd21ddd7-1c9b-4543-9c58-b1656798e9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 00:57:57.701684 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:57.701640    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 00:57:57.702059 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:57.701742    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg" podUID="fd21ddd7-1c9b-4543-9c58-b1656798e9ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 00:57:58.705735 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:58.705704    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:58.706117 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:58.705950    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:57:58.706612 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:57:58.706597    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 00:58:03.329277 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.329220    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"]
Oct 30 00:58:03.335537 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.335512    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.353090 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.353054    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"]
Oct 30 00:58:03.395434 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.395409    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.395578 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.395463    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.395578 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.395481    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.395660 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.395619    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.395660 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.395647    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.395765 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.395675    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.395824 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.395797    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.395933 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.395917    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.395988 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.395973    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w62qd\" (UniqueName: \"kubernetes.io/projected/83443474-b817-4ba6-afb6-dbe9a7bc24f0-kube-api-access-w62qd\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.497517 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.497486    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.497517 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.497521    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.497752 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.497543    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.497752 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.497567    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.497752 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.497627    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.497752 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.497652    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w62qd\" (UniqueName: \"kubernetes.io/projected/83443474-b817-4ba6-afb6-dbe9a7bc24f0-kube-api-access-w62qd\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.497752 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.497686    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.497752 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.497737    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.498066 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.497760    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.498066 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.498049    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.498192 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.498113    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.498256 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.498203    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.498256 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.498240    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.498595 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.498572    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.499724 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.499704    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.499943 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.499924    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.518114 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.518086    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.518267 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.518250    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w62qd\" (UniqueName: \"kubernetes.io/projected/83443474-b817-4ba6-afb6-dbe9a7bc24f0-kube-api-access-w62qd\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.654020 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.653965    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:03.969638 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.969612    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"]
Oct 30 00:58:03.971859 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:58:03.971833    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83443474_b817_4ba6_afb6_dbe9a7bc24f0.slice/crio-e3e7b0eb9c7df49c5f8ffd67b2cce6ceaedef7b61469ef7d9e7bf61b67d7c655 WatchSource:0}: Error finding container e3e7b0eb9c7df49c5f8ffd67b2cce6ceaedef7b61469ef7d9e7bf61b67d7c655: Status 404 returned error can't find the container with id e3e7b0eb9c7df49c5f8ffd67b2cce6ceaedef7b61469ef7d9e7bf61b67d7c655
Oct 30 00:58:03.977262 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.977226    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:03.977362 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.977297    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:03.977362 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:03.977326    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:04.156325 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:04.156283    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6" event={"ID":"83443474-b817-4ba6-afb6-dbe9a7bc24f0","Type":"ContainerStarted","Data":"88e78fda87118261430affe6efdc5467271c3a3775c574f220e610861b8ea40b"}
Oct 30 00:58:04.156512 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:04.156347    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6" event={"ID":"83443474-b817-4ba6-afb6-dbe9a7bc24f0","Type":"ContainerStarted","Data":"e3e7b0eb9c7df49c5f8ffd67b2cce6ceaedef7b61469ef7d9e7bf61b67d7c655"}
Oct 30 00:58:04.199904 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:04.199846    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6" podStartSLOduration=1.199826614 podStartE2EDuration="1.199826614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:04.195857869 +0000 UTC m=+992.945513182" watchObservedRunningTime="2026-10-30 00:58:04.199826614 +0000 UTC m=+992.949481924"
Oct 30 00:58:04.654971 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:04.654939    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:05.661888 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:05.659266    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:58:05.661888 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:05.659334    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6" podUID="83443474-b817-4ba6-afb6-dbe9a7bc24f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:58:06.656178 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:06.655463    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:58:06.656178 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:06.655522    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6" podUID="83443474-b817-4ba6-afb6-dbe9a7bc24f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:58:07.582310 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:58:07.582271    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:58:07.658183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:07.654670    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:58:07.658183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:07.654729    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6" podUID="83443474-b817-4ba6-afb6-dbe9a7bc24f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:58:08.662345 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:08.662304    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:58:08.662942 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:08.662894    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6" podUID="83443474-b817-4ba6-afb6-dbe9a7bc24f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:58:09.665176 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:09.662244    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:58:09.665176 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:09.662304    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6" podUID="83443474-b817-4ba6-afb6-dbe9a7bc24f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:58:10.654305 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:10.654272    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 00:58:10.654464 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:10.654324    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6" podUID="83443474-b817-4ba6-afb6-dbe9a7bc24f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 00:58:12.667140 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:12.667100    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:58:12.667565 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:12.667179    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6" podUID="83443474-b817-4ba6-afb6-dbe9a7bc24f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:12.741882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:12.741855    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:12.742314 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:12.742288    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:12.743186 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:12.743147    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 00:58:24.244872 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:24.244839    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:58:27.777148 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:27.777113    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"]
Oct 30 00:58:27.781064 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:27.781042    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:27.807780 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:27.807754    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"]
Oct 30 00:58:27.902427 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:27.902387    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:27.902582 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:27.902443    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:27.902582 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:27.902494    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:27.902663 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:27.902619    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:27.902720 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:27.902704    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:27.902820 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:27.902807    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:27.902861 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:27.902849    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:27.902950 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:27.902933    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:27.902995 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:27.902963    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jq77\" (UniqueName: \"kubernetes.io/projected/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-kube-api-access-9jq77\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.004053 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.004022    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.004248 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.004068    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.004248 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.004087    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jq77\" (UniqueName: \"kubernetes.io/projected/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-kube-api-access-9jq77\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.004248 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.004217    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.004436 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.004251    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.004436 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.004273    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.004436 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.004312    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.004436 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.004383    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.004641 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.004513    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.004641 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.004613    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.004641 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.004629    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.004776 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.004639    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.004856 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.004803    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.005206 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.005183    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.006476 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.006455    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.006724 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.006704    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.025503 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.025477    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.025623 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.025606    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jq77\" (UniqueName: \"kubernetes.io/projected/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-kube-api-access-9jq77\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.094813 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.094755    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:28.415170 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.415131    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"]
Oct 30 00:58:28.416991 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:58:28.416967    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4a4f6f_8a1b_4114_be9a_bad89cb053e5.slice/crio-a8d83ef31e6e886a6a9e97155319c71dd8d8a472518d972a732e7b01619d2321 WatchSource:0}: Error finding container a8d83ef31e6e886a6a9e97155319c71dd8d8a472518d972a732e7b01619d2321: Status 404 returned error can't find the container with id a8d83ef31e6e886a6a9e97155319c71dd8d8a472518d972a732e7b01619d2321
Oct 30 00:58:28.419374 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.419336    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:28.419440 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.419426    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:28.419490 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:28.419473    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:29.323125 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:29.323089    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq" event={"ID":"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5","Type":"ContainerStarted","Data":"d49e77c5bc636e47b7173ac6c8d60455a27eb41966adf89adc6060cbbe73d94a"}
Oct 30 00:58:29.323521 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:29.323145    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq" event={"ID":"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5","Type":"ContainerStarted","Data":"a8d83ef31e6e886a6a9e97155319c71dd8d8a472518d972a732e7b01619d2321"}
Oct 30 00:58:29.377386 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:29.377337    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq" podStartSLOduration=2.3773210000000002 podStartE2EDuration="2.377321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:29.375415811 +0000 UTC m=+1018.125071121" watchObservedRunningTime="2026-10-30 00:58:29.377321 +0000 UTC m=+1018.126976308"
Oct 30 00:58:30.095472 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:30.095445    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:30.097098 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:30.097064    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:58:30.097252 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:30.097120    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq" podUID="7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 00:58:30.816036 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:30.816000    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"]
Oct 30 00:58:30.820086 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:30.820063    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"
Oct 30 00:58:30.842082 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:30.842055    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:58:30.842082 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:30.842067    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 00:58:30.842506 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:30.842140    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:58:30.842506 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:30.842243    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:58:30.845266 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:30.845243    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"]
Oct 30 00:58:30.936512 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:30.936431    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a6e97fb-mj6b6\" (UID: \"9752cf4b-2544-4b25-ae84-1b47b48c6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"
Oct 30 00:58:30.936668 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:30.936582    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a6e97fb-mj6b6\" (UID: \"9752cf4b-2544-4b25-ae84-1b47b48c6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"
Oct 30 00:58:30.936745 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:30.936701    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clcrv\" (UniqueName: \"kubernetes.io/projected/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-kube-api-access-clcrv\") pod \"maas-api-revoke-sub-8a6e97fb-mj6b6\" (UID: \"9752cf4b-2544-4b25-ae84-1b47b48c6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"
Oct 30 00:58:31.038103 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:31.038069    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a6e97fb-mj6b6\" (UID: \"9752cf4b-2544-4b25-ae84-1b47b48c6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"
Oct 30 00:58:31.038316 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:31.038130    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clcrv\" (UniqueName: \"kubernetes.io/projected/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-kube-api-access-clcrv\") pod \"maas-api-revoke-sub-8a6e97fb-mj6b6\" (UID: \"9752cf4b-2544-4b25-ae84-1b47b48c6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"
Oct 30 00:58:31.038316 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:31.038238    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a6e97fb-mj6b6\" (UID: \"9752cf4b-2544-4b25-ae84-1b47b48c6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"
Oct 30 00:58:31.039373 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:31.039350    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a6e97fb-mj6b6\" (UID: \"9752cf4b-2544-4b25-ae84-1b47b48c6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"
Oct 30 00:58:31.040627 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:31.040603    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a6e97fb-mj6b6\" (UID: \"9752cf4b-2544-4b25-ae84-1b47b48c6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"
Oct 30 00:58:31.081241 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:31.081212    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clcrv\" (UniqueName: \"kubernetes.io/projected/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-kube-api-access-clcrv\") pod \"maas-api-revoke-sub-8a6e97fb-mj6b6\" (UID: \"9752cf4b-2544-4b25-ae84-1b47b48c6ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"
Oct 30 00:58:31.096010 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:31.095986    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:58:31.096183 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:31.096036    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq" podUID="7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 00:58:31.135576 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:31.135553    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"
Oct 30 00:58:31.480230 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:31.474032    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"]
Oct 30 00:58:31.488737 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:58:31.488705    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9752cf4b_2544_4b25_ae84_1b47b48c6ccb.slice/crio-4f7e31a9db6ffee5af29c46d7290fa1cb0f7b5552d8217e1f0edbd749552e382 WatchSource:0}: Error finding container 4f7e31a9db6ffee5af29c46d7290fa1cb0f7b5552d8217e1f0edbd749552e382: Status 404 returned error can't find the container with id 4f7e31a9db6ffee5af29c46d7290fa1cb0f7b5552d8217e1f0edbd749552e382
Oct 30 00:58:32.095798 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:32.095758    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:58:32.096214 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:32.095841    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq" podUID="7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 00:58:32.341654 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:32.341614    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6" event={"ID":"9752cf4b-2544-4b25-ae84-1b47b48c6ccb","Type":"ContainerStarted","Data":"35d5b78f4c45e6257ee1c13c7785b10c47b7de753cb6ac361c6020306f3629d7"}
Oct 30 00:58:32.341819 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:32.341740    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6" event={"ID":"9752cf4b-2544-4b25-ae84-1b47b48c6ccb","Type":"ContainerStarted","Data":"4f7e31a9db6ffee5af29c46d7290fa1cb0f7b5552d8217e1f0edbd749552e382"}
Oct 30 00:58:32.401188 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:32.400022    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6" podStartSLOduration=2.399484524 podStartE2EDuration="2.399484524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:32.397554577 +0000 UTC m=+1021.147209887" watchObservedRunningTime="2026-10-30 00:58:32.399484524 +0000 UTC m=+1021.149139834"
Oct 30 00:58:33.095847 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:33.095815    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:58:33.096259 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:33.095865    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq" podUID="7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 00:58:33.348133 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:33.348051    4425 generic.go:356] "Generic (PLEG): container finished" podID="9752cf4b-2544-4b25-ae84-1b47b48c6ccb" containerID="35d5b78f4c45e6257ee1c13c7785b10c47b7de753cb6ac361c6020306f3629d7" exitCode=0
Oct 30 00:58:33.348133 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:33.348093    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6" event={"ID":"9752cf4b-2544-4b25-ae84-1b47b48c6ccb","Type":"ContainerDied","Data":"35d5b78f4c45e6257ee1c13c7785b10c47b7de753cb6ac361c6020306f3629d7"}
Oct 30 00:58:34.096004 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:34.095970    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:58:34.096451 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:34.096016    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq" podUID="7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 00:58:34.520327 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:34.520304    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"
Oct 30 00:58:34.701388 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:34.701310    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-kube-api-access-clcrv\" (UniqueName: \"kubernetes.io/projected/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-kube-api-access-clcrv\") pod \"9752cf4b-2544-4b25-ae84-1b47b48c6ccb\" (UID: \"9752cf4b-2544-4b25-ae84-1b47b48c6ccb\") "
Oct 30 00:58:34.701526 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:34.701393    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-maas-api-openshift-ca\") pod \"9752cf4b-2544-4b25-ae84-1b47b48c6ccb\" (UID: \"9752cf4b-2544-4b25-ae84-1b47b48c6ccb\") "
Oct 30 00:58:34.701526 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:34.701480    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-maas-api-xks-ca\") pod \"9752cf4b-2544-4b25-ae84-1b47b48c6ccb\" (UID: \"9752cf4b-2544-4b25-ae84-1b47b48c6ccb\") "
Oct 30 00:58:34.701762 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:34.701731    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-maas-api-openshift-ca" pod "9752cf4b-2544-4b25-ae84-1b47b48c6ccb" (UID: "9752cf4b-2544-4b25-ae84-1b47b48c6ccb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:34.701870 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:34.701856    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:58:34.703403 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:34.703377    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-maas-api-xks-ca" pod "9752cf4b-2544-4b25-ae84-1b47b48c6ccb" (UID: "9752cf4b-2544-4b25-ae84-1b47b48c6ccb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:34.703520 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:34.703510    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-kube-api-access-clcrv" pod "9752cf4b-2544-4b25-ae84-1b47b48c6ccb" (UID: "9752cf4b-2544-4b25-ae84-1b47b48c6ccb"). InnerVolumeSpecName "kube-api-access-clcrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:34.803093 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:34.803064    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:58:34.803093 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:34.803089    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clcrv\" (UniqueName: \"kubernetes.io/projected/9752cf4b-2544-4b25-ae84-1b47b48c6ccb-kube-api-access-clcrv\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:58:35.095335 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:35.095301    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 00:58:35.095530 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:35.095388    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq" podUID="7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 00:58:35.377315 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:35.377238    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"
Oct 30 00:58:35.377671 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:35.377241    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6" event={"ID":"9752cf4b-2544-4b25-ae84-1b47b48c6ccb","Type":"ContainerDied","Data":"4f7e31a9db6ffee5af29c46d7290fa1cb0f7b5552d8217e1f0edbd749552e382"}
Oct 30 00:58:35.377671 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:35.377395    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f7e31a9db6ffee5af29c46d7290fa1cb0f7b5552d8217e1f0edbd749552e382"
Oct 30 00:58:36.099307 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:36.099280    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:36.099650 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:36.099619    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:36.100445 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:36.100425    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 00:58:37.554816 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:58:37.554781    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:58:38.545059 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.545025    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"]
Oct 30 00:58:38.545722 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.545698    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9752cf4b-2544-4b25-ae84-1b47b48c6ccb" containerName="revoke-keys"
Oct 30 00:58:38.545722 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.545720    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9752cf4b-2544-4b25-ae84-1b47b48c6ccb" containerName="revoke-keys"
Oct 30 00:58:38.545914 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.545828    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9752cf4b-2544-4b25-ae84-1b47b48c6ccb" containerName="revoke-keys"
Oct 30 00:58:38.549323 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.549297    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"
Oct 30 00:58:38.561884 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.561863    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:58:38.562192 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.561898    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:58:38.571585 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.571567    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:58:38.571748 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.571578    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 00:58:38.575085 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.575057    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"]
Oct 30 00:58:38.646676 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.646640    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c01ba8f6-4p422\" (UID: \"c80196e9-ead5-4ce0-b6f1-90506f67d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"
Oct 30 00:58:38.646830 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.646684    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7xsc\" (UniqueName: \"kubernetes.io/projected/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-kube-api-access-z7xsc\") pod \"maas-api-revoke-sub-c01ba8f6-4p422\" (UID: \"c80196e9-ead5-4ce0-b6f1-90506f67d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"
Oct 30 00:58:38.646913 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.646896    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c01ba8f6-4p422\" (UID: \"c80196e9-ead5-4ce0-b6f1-90506f67d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"
Oct 30 00:58:38.748360 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.748329    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c01ba8f6-4p422\" (UID: \"c80196e9-ead5-4ce0-b6f1-90506f67d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"
Oct 30 00:58:38.748515 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.748388    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c01ba8f6-4p422\" (UID: \"c80196e9-ead5-4ce0-b6f1-90506f67d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"
Oct 30 00:58:38.748515 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.748418    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7xsc\" (UniqueName: \"kubernetes.io/projected/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-kube-api-access-z7xsc\") pod \"maas-api-revoke-sub-c01ba8f6-4p422\" (UID: \"c80196e9-ead5-4ce0-b6f1-90506f67d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"
Oct 30 00:58:38.749044 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.749019    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c01ba8f6-4p422\" (UID: \"c80196e9-ead5-4ce0-b6f1-90506f67d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"
Oct 30 00:58:38.750675 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.750654    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c01ba8f6-4p422\" (UID: \"c80196e9-ead5-4ce0-b6f1-90506f67d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"
Oct 30 00:58:38.769744 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.769721    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7xsc\" (UniqueName: \"kubernetes.io/projected/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-kube-api-access-z7xsc\") pod \"maas-api-revoke-sub-c01ba8f6-4p422\" (UID: \"c80196e9-ead5-4ce0-b6f1-90506f67d2eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"
Oct 30 00:58:38.862517 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:38.862444    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"
Oct 30 00:58:39.163410 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:39.163386    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"]
Oct 30 00:58:39.166378 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:58:39.166346    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80196e9_ead5_4ce0_b6f1_90506f67d2eb.slice/crio-735775a9a50305350638b03620098827343752d7d8d121ebc8bd38270e6d72dd WatchSource:0}: Error finding container 735775a9a50305350638b03620098827343752d7d8d121ebc8bd38270e6d72dd: Status 404 returned error can't find the container with id 735775a9a50305350638b03620098827343752d7d8d121ebc8bd38270e6d72dd
Oct 30 00:58:39.401100 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:39.401056    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422" event={"ID":"c80196e9-ead5-4ce0-b6f1-90506f67d2eb","Type":"ContainerStarted","Data":"735775a9a50305350638b03620098827343752d7d8d121ebc8bd38270e6d72dd"}
Oct 30 00:58:40.407886 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:40.407853    4425 generic.go:356] "Generic (PLEG): container finished" podID="c80196e9-ead5-4ce0-b6f1-90506f67d2eb" containerID="7c00979705535911102cf16fc745d0e18da9b933c0eb2557f0e28e98eafa2510" exitCode=0
Oct 30 00:58:40.408318 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:40.407908    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422" event={"ID":"c80196e9-ead5-4ce0-b6f1-90506f67d2eb","Type":"ContainerDied","Data":"7c00979705535911102cf16fc745d0e18da9b933c0eb2557f0e28e98eafa2510"}
Oct 30 00:58:41.570705 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.570680    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"
Oct 30 00:58:41.682390 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.682310    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-maas-api-xks-ca\") pod \"c80196e9-ead5-4ce0-b6f1-90506f67d2eb\" (UID: \"c80196e9-ead5-4ce0-b6f1-90506f67d2eb\") "
Oct 30 00:58:41.682535 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.682397    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-kube-api-access-z7xsc\" (UniqueName: \"kubernetes.io/projected/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-kube-api-access-z7xsc\") pod \"c80196e9-ead5-4ce0-b6f1-90506f67d2eb\" (UID: \"c80196e9-ead5-4ce0-b6f1-90506f67d2eb\") "
Oct 30 00:58:41.682578 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.682545    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-maas-api-openshift-ca\") pod \"c80196e9-ead5-4ce0-b6f1-90506f67d2eb\" (UID: \"c80196e9-ead5-4ce0-b6f1-90506f67d2eb\") "
Oct 30 00:58:41.682899 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.682874    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-maas-api-openshift-ca" pod "c80196e9-ead5-4ce0-b6f1-90506f67d2eb" (UID: "c80196e9-ead5-4ce0-b6f1-90506f67d2eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:58:41.682990 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.682977    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:58:41.684420 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.684397    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-maas-api-xks-ca" pod "c80196e9-ead5-4ce0-b6f1-90506f67d2eb" (UID: "c80196e9-ead5-4ce0-b6f1-90506f67d2eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:58:41.684536 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.684505    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-kube-api-access-z7xsc" pod "c80196e9-ead5-4ce0-b6f1-90506f67d2eb" (UID: "c80196e9-ead5-4ce0-b6f1-90506f67d2eb"). InnerVolumeSpecName "kube-api-access-z7xsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:58:41.783727 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.783701    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7xsc\" (UniqueName: \"kubernetes.io/projected/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-kube-api-access-z7xsc\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:58:41.783727 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.783725    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c80196e9-ead5-4ce0-b6f1-90506f67d2eb-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:58:41.960353 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.960266    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"]
Oct 30 00:58:41.960944 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.960923    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c80196e9-ead5-4ce0-b6f1-90506f67d2eb" containerName="revoke-keys"
Oct 30 00:58:41.961015 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.960948    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80196e9-ead5-4ce0-b6f1-90506f67d2eb" containerName="revoke-keys"
Oct 30 00:58:41.961077 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.961065    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80196e9-ead5-4ce0-b6f1-90506f67d2eb" containerName="revoke-keys"
Oct 30 00:58:41.964767 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.964743    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:41.977121 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.977101    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-664c2b\"/\"distinct-w1-664c2b-kserve-self-signed-certs\""
Oct 30 00:58:41.984605 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.984581    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"]
Oct 30 00:58:41.984998 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.984972    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-model-cache\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:41.985118 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.985004    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-tls-certs\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:41.985118 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.985028    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-dshm\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:41.985277 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.985148    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-home\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:41.985277 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.985252    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-tmp-dir\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:41.985413 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:41.985390    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24sdw\" (UniqueName: \"kubernetes.io/projected/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-kube-api-access-24sdw\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:42.086566 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:42.086528    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-tmp-dir\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:42.086744 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:42.086623    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24sdw\" (UniqueName: \"kubernetes.io/projected/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-kube-api-access-24sdw\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:42.086814 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:42.086744    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-model-cache\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:42.086814 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:42.086772    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-tls-certs\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:42.086814 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:42.086802    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-dshm\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:42.086961 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:42.086827    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-home\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:42.087076 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:42.087051    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-tmp-dir\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:42.087265 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:42.087246    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-home\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:42.087436 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:42.087409    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-model-cache\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:42.089641 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:42.089615    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-dshm\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:42.089759 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:42.089734    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-tls-certs\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:42.110189 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:42.108218    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24sdw\" (UniqueName: \"kubernetes.io/projected/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-kube-api-access-24sdw\") pod \"distinct-w1-664c2b-kserve-7555dc55d9-cxvsk\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:42.279750 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:42.279706    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:42.420840 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:42.420813    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"
Oct 30 00:58:42.421002 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:42.420813    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422" event={"ID":"c80196e9-ead5-4ce0-b6f1-90506f67d2eb","Type":"ContainerDied","Data":"735775a9a50305350638b03620098827343752d7d8d121ebc8bd38270e6d72dd"}
Oct 30 00:58:42.421002 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:42.420933    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="735775a9a50305350638b03620098827343752d7d8d121ebc8bd38270e6d72dd"
Oct 30 00:58:42.588214 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:42.588187    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"]
Oct 30 00:58:42.590306 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:58:42.590278    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bc38761_a8e2_4862_b37a_5bcc4e05f3ec.slice/crio-19a0477f9899e2ab0e83b78d420fbafb01d16da5a4478efa214e982d26da482f WatchSource:0}: Error finding container 19a0477f9899e2ab0e83b78d420fbafb01d16da5a4478efa214e982d26da482f: Status 404 returned error can't find the container with id 19a0477f9899e2ab0e83b78d420fbafb01d16da5a4478efa214e982d26da482f
Oct 30 00:58:43.444792 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:43.444746    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" event={"ID":"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec","Type":"ContainerStarted","Data":"a2f8ad05532131ac2b563286ebb4bbf05c3ee95767e7372c61abe82741feddfd"}
Oct 30 00:58:43.444792 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:43.444797    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" event={"ID":"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec","Type":"ContainerStarted","Data":"19a0477f9899e2ab0e83b78d420fbafb01d16da5a4478efa214e982d26da482f"}
Oct 30 00:58:43.485735 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:43.483875    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" podStartSLOduration=2.483853308 podStartE2EDuration="2.483853308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:43.48163048 +0000 UTC m=+1032.231285815" watchObservedRunningTime="2026-10-30 00:58:43.483853308 +0000 UTC m=+1032.233508612"
Oct 30 00:58:44.286191 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:44.283242    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:44.982180 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:44.970417    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z" podUID="b4ed47ff-0328-4645-be5f-6a6d51a94fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:58:48.477975 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.477939    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"]
Oct 30 00:58:48.481931 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.481901    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.515972 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.515941    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.516131 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.515994    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcfc6\" (UniqueName: \"kubernetes.io/projected/998232e0-4d7b-4a4b-9fd1-5eb005157b11-kube-api-access-lcfc6\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.516131 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.516059    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.516131 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.516087    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.516354 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.516136    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.516354 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.516312    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.516472 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.516444    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.516526 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.516474    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.516526 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.516494    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.535798 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.535770    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"]
Oct 30 00:58:48.617854 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.617824    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.618012 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.617900    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.618012 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.617922    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.618012 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.617937    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.618012 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.617975    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.618267 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.618014    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcfc6\" (UniqueName: \"kubernetes.io/projected/998232e0-4d7b-4a4b-9fd1-5eb005157b11-kube-api-access-lcfc6\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.618267 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.618047    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.618267 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.618074    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.618267 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.618143    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.618467 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.618397    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-data\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.618541 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.618517    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-workload-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.618608 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.618584    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-workload-certs\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.618707 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.618684    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-credential-socket\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.618958 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.618941    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istiod-ca-cert\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.620169 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.620137    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-envoy\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.620412 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.620395    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-podinfo\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.639020 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.638998    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-token\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.639120 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.639087    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcfc6\" (UniqueName: \"kubernetes.io/projected/998232e0-4d7b-4a4b-9fd1-5eb005157b11-kube-api-access-lcfc6\") pod \"e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") " pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:48.796939 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:48.796894    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:49.112554 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:49.112521    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"]
Oct 30 00:58:49.113919 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:58:49.113894    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998232e0_4d7b_4a4b_9fd1_5eb005157b11.slice/crio-8ef843bf9c9f1cd2f45cb1889d256d110c8bb7f0ab84f58ccb36d66ae765e8bd WatchSource:0}: Error finding container 8ef843bf9c9f1cd2f45cb1889d256d110c8bb7f0ab84f58ccb36d66ae765e8bd: Status 404 returned error can't find the container with id 8ef843bf9c9f1cd2f45cb1889d256d110c8bb7f0ab84f58ccb36d66ae765e8bd
Oct 30 00:58:49.120613 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:49.120582    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:49.120676 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:49.120645    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:49.120722 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:49.120676    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:49.504004 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:49.503915    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr" event={"ID":"998232e0-4d7b-4a4b-9fd1-5eb005157b11","Type":"ContainerStarted","Data":"cde5f6edf92232a90174fc5976205803909583e7b1e3994bf76707307795412f"}
Oct 30 00:58:49.504004 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:49.503975    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr" event={"ID":"998232e0-4d7b-4a4b-9fd1-5eb005157b11","Type":"ContainerStarted","Data":"8ef843bf9c9f1cd2f45cb1889d256d110c8bb7f0ab84f58ccb36d66ae765e8bd"}
Oct 30 00:58:49.545687 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:49.545636    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr" podStartSLOduration=1.5456221970000001 podStartE2EDuration="1.545622197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:49.544680085 +0000 UTC m=+1038.294335394" watchObservedRunningTime="2026-10-30 00:58:49.545622197 +0000 UTC m=+1038.295277506"
Oct 30 00:58:49.797816 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:49.797771    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:50.799044 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:50.799004    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 00:58:50.799448 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:50.799067    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr" podUID="998232e0-4d7b-4a4b-9fd1-5eb005157b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 00:58:51.797746 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.797707    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 00:58:51.797920 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.797777    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr" podUID="998232e0-4d7b-4a4b-9fd1-5eb005157b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 00:58:51.803690 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.803665    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"]
Oct 30 00:58:51.807817 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.807796    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.832265 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.832237    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"]
Oct 30 00:58:51.873871 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.873826    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.874626 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.874593    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.874946 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.874886    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421b4772-a3d2-4cf1-aefb-abc655989ad9-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.875261 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.875215    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.875492 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.875474    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.875730 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.875709    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.875969 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.875917    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.876241 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.876221    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.876546 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.876526    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdwsz\" (UniqueName: \"kubernetes.io/projected/421b4772-a3d2-4cf1-aefb-abc655989ad9-kube-api-access-bdwsz\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.977845 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.977811    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.978069 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.977876    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.978069 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.977915    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.978069 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.977937    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.978069 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.977972    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.978069 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.978030    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwsz\" (UniqueName: \"kubernetes.io/projected/421b4772-a3d2-4cf1-aefb-abc655989ad9-kube-api-access-bdwsz\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.978358 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.978079    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.978358 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.978144    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.978358 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.978216    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421b4772-a3d2-4cf1-aefb-abc655989ad9-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.978358 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.978347    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.978640 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.978618    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.978717 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.978694    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.978884 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.978861    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.978960 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.978947    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421b4772-a3d2-4cf1-aefb-abc655989ad9-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.980970 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.980946    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:51.981166 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:51.981139    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:52.000071 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:52.000046    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdwsz\" (UniqueName: \"kubernetes.io/projected/421b4772-a3d2-4cf1-aefb-abc655989ad9-kube-api-access-bdwsz\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:52.000177 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:52.000048    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:52.123682 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:52.123606    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:52.281270 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:52.280719    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:52.298711 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:52.298463    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:52.479481 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:52.479451    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"]
Oct 30 00:58:52.479597 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:58:52.479502    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421b4772_a3d2_4cf1_aefb_abc655989ad9.slice/crio-df9ac6f381e407faad8c50de5e6831e4981600d38b007821ef0f33733a2641f9 WatchSource:0}: Error finding container df9ac6f381e407faad8c50de5e6831e4981600d38b007821ef0f33733a2641f9: Status 404 returned error can't find the container with id df9ac6f381e407faad8c50de5e6831e4981600d38b007821ef0f33733a2641f9
Oct 30 00:58:52.481950 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:52.481918    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:52.482041 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:52.481978    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:52.482041 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:52.482006    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:58:52.522689 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:52.522664    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls" event={"ID":"421b4772-a3d2-4cf1-aefb-abc655989ad9","Type":"ContainerStarted","Data":"df9ac6f381e407faad8c50de5e6831e4981600d38b007821ef0f33733a2641f9"}
Oct 30 00:58:52.534249 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:52.534228    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:58:52.797731 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:52.797699    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 00:58:52.797933 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:52.797748    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr" podUID="998232e0-4d7b-4a4b-9fd1-5eb005157b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 00:58:53.531526 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:53.531496    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls" event={"ID":"421b4772-a3d2-4cf1-aefb-abc655989ad9","Type":"ContainerStarted","Data":"21de5a9a998b22a17887be397d1bf9ba7c70d1fbcdf0433d0548786413cf2ea3"}
Oct 30 00:58:53.605474 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:53.605416    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls" podStartSLOduration=2.605396302 podStartE2EDuration="2.605396302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:53.602563418 +0000 UTC m=+1042.352218729" watchObservedRunningTime="2026-10-30 00:58:53.605396302 +0000 UTC m=+1042.355051612"
Oct 30 00:58:53.798245 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:53.797648    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 00:58:53.798245 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:53.797706    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr" podUID="998232e0-4d7b-4a4b-9fd1-5eb005157b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 00:58:54.124604 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:54.124516    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:58:54.125724 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:54.125691    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:58:54.125866 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:54.125739    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls" podUID="421b4772-a3d2-4cf1-aefb-abc655989ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 00:58:54.801180 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:54.800426    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 00:58:54.801180 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:54.800486    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr" podUID="998232e0-4d7b-4a4b-9fd1-5eb005157b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 00:58:55.124783 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:55.124689    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:58:55.124783 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:55.124748    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls" podUID="421b4772-a3d2-4cf1-aefb-abc655989ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 00:58:55.797985 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:55.797948    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 00:58:55.798150 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:55.798000    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr" podUID="998232e0-4d7b-4a4b-9fd1-5eb005157b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 00:58:56.124507 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:56.124419    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:58:56.124507 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:56.124482    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls" podUID="421b4772-a3d2-4cf1-aefb-abc655989ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 00:58:56.802325 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:56.802294    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:56.802608 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:56.802584    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:56.803234 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:56.803218    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 00:58:57.124220 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:57.124128    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:58:57.124368 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:57.124216    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls" podUID="421b4772-a3d2-4cf1-aefb-abc655989ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 00:58:58.124781 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:58.124745    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:58:58.125208 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:58.124796    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls" podUID="421b4772-a3d2-4cf1-aefb-abc655989ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 00:58:58.427296 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:58.427213    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:58:59.124681 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:59.124644    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 00:58:59.124894 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:58:59.124714    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls" podUID="421b4772-a3d2-4cf1-aefb-abc655989ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 00:59:00.652207 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:00.652179    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:59:00.654341 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:00.654307    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:59:00.659464 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:00.659441    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 00:59:01.673304 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:01.673255    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"]
Oct 30 00:59:01.679313 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:01.679289    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"
Oct 30 00:59:01.693256 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:01.693233    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:59:01.702815 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:01.702792    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:59:01.705428 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:01.705401    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"]
Oct 30 00:59:01.750868 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:01.750839    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c903a9d4-xcn9n\" (UID: \"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"
Oct 30 00:59:01.751006 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:01.750883    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ts4h\" (UniqueName: \"kubernetes.io/projected/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-kube-api-access-8ts4h\") pod \"maas-api-revoke-sub-c903a9d4-xcn9n\" (UID: \"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"
Oct 30 00:59:01.751212 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:01.751179    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c903a9d4-xcn9n\" (UID: \"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"
Oct 30 00:59:01.772873 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:01.772854    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:59:01.773003 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:01.772914    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 00:59:01.852380 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:01.852352    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c903a9d4-xcn9n\" (UID: \"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"
Oct 30 00:59:01.852498 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:01.852387    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ts4h\" (UniqueName: \"kubernetes.io/projected/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-kube-api-access-8ts4h\") pod \"maas-api-revoke-sub-c903a9d4-xcn9n\" (UID: \"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"
Oct 30 00:59:01.852498 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:01.852476    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c903a9d4-xcn9n\" (UID: \"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"
Oct 30 00:59:01.853226 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:01.853206    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c903a9d4-xcn9n\" (UID: \"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"
Oct 30 00:59:01.854796 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:01.854775    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c903a9d4-xcn9n\" (UID: \"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"
Oct 30 00:59:01.873454 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:01.873428    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ts4h\" (UniqueName: \"kubernetes.io/projected/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-kube-api-access-8ts4h\") pod \"maas-api-revoke-sub-c903a9d4-xcn9n\" (UID: \"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"
Oct 30 00:59:01.995121 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:01.995051    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"
Oct 30 00:59:02.303203 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.303181    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"]
Oct 30 00:59:02.304884 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:59:02.304858    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c9c70b_b6c8_43e7_b276_43b1b5ac20ad.slice/crio-d0e58f2e6c3d460c44780bd921a1e22a981f2d9167705d3cd3a3cc860df3c438 WatchSource:0}: Error finding container d0e58f2e6c3d460c44780bd921a1e22a981f2d9167705d3cd3a3cc860df3c438: Status 404 returned error can't find the container with id d0e58f2e6c3d460c44780bd921a1e22a981f2d9167705d3cd3a3cc860df3c438
Oct 30 00:59:02.346874 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.346849    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"]
Oct 30 00:59:02.352264 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.352243    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.372889 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.372864    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"]
Oct 30 00:59:02.376378 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.376358    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-664c2b\"/\"distinct-2-w1-664c2b-kserve-self-signed-certs\""
Oct 30 00:59:02.459557 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.459519    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16b18eab-634a-4cc9-8f05-10a2e3b937f2-tls-certs\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.459725 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.459573    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-dshm\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.459782 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.459714    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-tmp-dir\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.459882 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.459864    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-home\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.459926 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.459911    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-model-cache\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.459972 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.459956    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhkzb\" (UniqueName: \"kubernetes.io/projected/16b18eab-634a-4cc9-8f05-10a2e3b937f2-kube-api-access-rhkzb\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.564084 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.561053    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-tmp-dir\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.564084 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.561260    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-home\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.564084 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.561304    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-model-cache\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.564084 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.561347    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhkzb\" (UniqueName: \"kubernetes.io/projected/16b18eab-634a-4cc9-8f05-10a2e3b937f2-kube-api-access-rhkzb\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.564084 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.561408    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16b18eab-634a-4cc9-8f05-10a2e3b937f2-tls-certs\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.564084 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.561441    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-dshm\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.564084 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.561640    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-tmp-dir\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.564084 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.562620    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-home\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.564084 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.562745    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-model-cache\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.565447 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.565424    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-dshm\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.570395 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.570371    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16b18eab-634a-4cc9-8f05-10a2e3b937f2-tls-certs\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.583247 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.583219    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhkzb\" (UniqueName: \"kubernetes.io/projected/16b18eab-634a-4cc9-8f05-10a2e3b937f2-kube-api-access-rhkzb\") pod \"distinct-2-w1-664c2b-kserve-5c78796994-5qr9j\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") " pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.633221 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.633180    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n" event={"ID":"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad","Type":"ContainerStarted","Data":"d0e58f2e6c3d460c44780bd921a1e22a981f2d9167705d3cd3a3cc860df3c438"}
Oct 30 00:59:02.666892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.666863    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:02.970907 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:02.970637    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"]
Oct 30 00:59:03.639789 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:03.639758    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" event={"ID":"16b18eab-634a-4cc9-8f05-10a2e3b937f2","Type":"ContainerStarted","Data":"efb5688e0c2981dac4d9c1f551158004328395fc7cd8f2b349c21b7c8deb8ade"}
Oct 30 00:59:03.639989 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:03.639803    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" event={"ID":"16b18eab-634a-4cc9-8f05-10a2e3b937f2","Type":"ContainerStarted","Data":"f35a5a639e2054cc98fd3f5f8d67a90a5aa831c4e76bb0f45e19e52aa417130d"}
Oct 30 00:59:03.641456 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:03.641427    4425 generic.go:356] "Generic (PLEG): container finished" podID="f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad" containerID="ceae8b352770ffe17708d8e59b3b8d9891aa3ad2e3bd8af56d0a267a91bcb6d4" exitCode=0
Oct 30 00:59:03.641573 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:03.641460    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n" event={"ID":"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad","Type":"ContainerDied","Data":"ceae8b352770ffe17708d8e59b3b8d9891aa3ad2e3bd8af56d0a267a91bcb6d4"}
Oct 30 00:59:03.667970 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:03.667949    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:03.676852 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:03.676814    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" podStartSLOduration=1.676801907 podStartE2EDuration="1.676801907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:03.675917043 +0000 UTC m=+1052.425572362" watchObservedRunningTime="2026-10-30 00:59:03.676801907 +0000 UTC m=+1052.426457216"
Oct 30 00:59:04.791881 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:04.791852    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"
Oct 30 00:59:04.895743 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:04.895715    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-kube-api-access-8ts4h\" (UniqueName: \"kubernetes.io/projected/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-kube-api-access-8ts4h\") pod \"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad\" (UID: \"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad\") "
Oct 30 00:59:04.895907 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:04.895782    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-maas-api-openshift-ca\") pod \"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad\" (UID: \"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad\") "
Oct 30 00:59:04.895907 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:04.895837    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-maas-api-xks-ca\") pod \"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad\" (UID: \"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad\") "
Oct 30 00:59:04.896192 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:04.896142    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-maas-api-openshift-ca" pod "f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad" (UID: "f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:04.897781 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:04.897747    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-kube-api-access-8ts4h" pod "f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad" (UID: "f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad"). InnerVolumeSpecName "kube-api-access-8ts4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:04.897872 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:04.897833    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-maas-api-xks-ca" pod "f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad" (UID: "f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:59:04.997142 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:04.997115    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ts4h\" (UniqueName: \"kubernetes.io/projected/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-kube-api-access-8ts4h\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:59:04.997142 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:04.997140    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:59:04.997325 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:04.997151    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:59:05.654938 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:05.654903    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"
Oct 30 00:59:05.655182 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:05.654903    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n" event={"ID":"f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad","Type":"ContainerDied","Data":"d0e58f2e6c3d460c44780bd921a1e22a981f2d9167705d3cd3a3cc860df3c438"}
Oct 30 00:59:05.655182 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:05.655017    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0e58f2e6c3d460c44780bd921a1e22a981f2d9167705d3cd3a3cc860df3c438"
Oct 30 00:59:07.553648 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:59:07.553617    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:59:12.667598 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:12.667566    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:12.679981 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:12.679955    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:12.704968 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:12.704938    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:12.902679 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:12.902652    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"]
Oct 30 00:59:12.903259 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:12.903244    4425 cpu_manager.go:414] "containerMap: removing container" podUID="f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad" containerName="revoke-keys"
Oct 30 00:59:12.903312 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:12.903262    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad" containerName="revoke-keys"
Oct 30 00:59:12.903352 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:12.903343    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad" containerName="revoke-keys"
Oct 30 00:59:12.907888 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:12.907864    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:12.932500 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:12.932443    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"]
Oct 30 00:59:13.083572 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.083542    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jrpt\" (UniqueName: \"kubernetes.io/projected/11e986da-b068-404f-91f7-4a8f109465d7-kube-api-access-8jrpt\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.083737 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.083579    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e986da-b068-404f-91f7-4a8f109465d7-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.083737 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.083666    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.083811 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.083749    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.083811 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.083777    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.083904 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.083890    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.083950 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.083939    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e986da-b068-404f-91f7-4a8f109465d7-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.084004 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.083961    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e986da-b068-404f-91f7-4a8f109465d7-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.084046 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.084019    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.185308 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.185227    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e986da-b068-404f-91f7-4a8f109465d7-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.185308 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.185273    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e986da-b068-404f-91f7-4a8f109465d7-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.185549 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.185527    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.185641 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.185627    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jrpt\" (UniqueName: \"kubernetes.io/projected/11e986da-b068-404f-91f7-4a8f109465d7-kube-api-access-8jrpt\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.185773 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.185745    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e986da-b068-404f-91f7-4a8f109465d7-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.185933 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.185810    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.185933 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.185843    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.185933 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.185888    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.186122 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.185993    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.186122 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.186014    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.186539 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.186445    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.186539 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.186474    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.186539 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.186490    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.186752 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.186578    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e986da-b068-404f-91f7-4a8f109465d7-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.188089 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.188063    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e986da-b068-404f-91f7-4a8f109465d7-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.188556 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.188536    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.206265 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.206242    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e986da-b068-404f-91f7-4a8f109465d7-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.206442 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.206421    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jrpt\" (UniqueName: \"kubernetes.io/projected/11e986da-b068-404f-91f7-4a8f109465d7-kube-api-access-8jrpt\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.222079 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.222006    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:13.541630 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.541605    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"]
Oct 30 00:59:13.542583 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:59:13.542558    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e986da_b068_404f_91f7_4a8f109465d7.slice/crio-79de5a34941ff05a07a21dc6a3996e2825eecf2aae2387ebf0c6b54ac5a4a52a WatchSource:0}: Error finding container 79de5a34941ff05a07a21dc6a3996e2825eecf2aae2387ebf0c6b54ac5a4a52a: Status 404 returned error can't find the container with id 79de5a34941ff05a07a21dc6a3996e2825eecf2aae2387ebf0c6b54ac5a4a52a
Oct 30 00:59:13.545014 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.544978    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:13.545117 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.545055    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:13.545209 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.545111    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:13.702432 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.702399    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j" event={"ID":"11e986da-b068-404f-91f7-4a8f109465d7","Type":"ContainerStarted","Data":"2e4d5be578242ec18254ac72479feeebd360f514144efa950b13c5605f6a0208"}
Oct 30 00:59:13.702915 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.702459    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j" event={"ID":"11e986da-b068-404f-91f7-4a8f109465d7","Type":"ContainerStarted","Data":"79de5a34941ff05a07a21dc6a3996e2825eecf2aae2387ebf0c6b54ac5a4a52a"}
Oct 30 00:59:13.745852 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:13.745795    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j" podStartSLOduration=1.745780154 podStartE2EDuration="1.745780154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:13.742471987 +0000 UTC m=+1062.492127297" watchObservedRunningTime="2026-10-30 00:59:13.745780154 +0000 UTC m=+1062.495435462"
Oct 30 00:59:14.225175 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:14.223462    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:14.225487 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:14.225444    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 00:59:14.225631 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:14.225507    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j" podUID="11e986da-b068-404f-91f7-4a8f109465d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 00:59:15.222984 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:15.222948    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 00:59:15.223389 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:15.223012    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j" podUID="11e986da-b068-404f-91f7-4a8f109465d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 00:59:16.222666 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:16.222623    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 00:59:16.222864 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:16.222675    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j" podUID="11e986da-b068-404f-91f7-4a8f109465d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 00:59:17.223170 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:17.223122    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 00:59:17.223593 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:17.223220    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j" podUID="11e986da-b068-404f-91f7-4a8f109465d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 00:59:18.223601 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:18.223535    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 00:59:18.224204 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:18.224171    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j" podUID="11e986da-b068-404f-91f7-4a8f109465d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 00:59:19.082297 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:19.082245    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2" podUID="13f688db-25e0-4a88-9d1e-c3fd78b7f659" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:19.224533 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:19.224496    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 00:59:19.225114 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:19.225072    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j" podUID="11e986da-b068-404f-91f7-4a8f109465d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 00:59:19.256283 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:19.256239    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s" podUID="b8b27e40-2516-4d18-8c84-5e0925b3f4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:19.697368 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:19.697308    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:20.235181 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:20.225324    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 00:59:20.235181 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:20.225392    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j" podUID="11e986da-b068-404f-91f7-4a8f109465d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 00:59:20.861288 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:20.861234    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"]
Oct 30 00:59:20.929500 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:20.929470    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.024046 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.024006    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"]
Oct 30 00:59:21.056093 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.046894    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.056093 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.046974    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.056093 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.047083    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.056093 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.047113    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.056093 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.047146    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.056093 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.047187    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.056093 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.047236    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64zhw\" (UniqueName: \"kubernetes.io/projected/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-kube-api-access-64zhw\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.056093 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.047299    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.056093 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.047335    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.070202 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.067529    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"]
Oct 30 00:59:21.080179 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.079521    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"]
Oct 30 00:59:21.081878 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.081844    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.156907 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.156838    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.156907 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.156884    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.157142 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.156941    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.157276 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.157260    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.157316 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.157291    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.157440 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.157315    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wmf9\" (UniqueName: \"kubernetes.io/projected/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-kube-api-access-6wmf9\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.157497 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.157441    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.157546 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.157518    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.157602 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.157552    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.157602 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.157587    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64zhw\" (UniqueName: \"kubernetes.io/projected/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-kube-api-access-64zhw\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.157720 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.157658    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.157720 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.157674    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.157720 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.157698    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.157880 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.157767    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.157880 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.157815    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.157880 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.157849    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.158049 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.157894    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.158049 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.157926    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.158049 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.157953    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.158049 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.158018    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.158304 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.158095    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.158304 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.158133    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.158522 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.158497    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.160300 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.160279    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.166666 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.166642    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.185894 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.185865    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.187252 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.187225    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64zhw\" (UniqueName: \"kubernetes.io/projected/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-kube-api-access-64zhw\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.223242 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.223216    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 00:59:21.223464 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.223266    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j" podUID="11e986da-b068-404f-91f7-4a8f109465d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 00:59:21.259002 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.258967    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.259425 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.259013    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.259425 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.259048    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.259425 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.259072    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.259425 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.259250    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.259425 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.259309    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wmf9\" (UniqueName: \"kubernetes.io/projected/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-kube-api-access-6wmf9\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.259425 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.259360    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.259425 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.259417    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.259820 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.259439    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.259820 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.259514    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.259929 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.259849    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.260125 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.260102    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.260291 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.260135    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.260349 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.260335    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.262277 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.262253    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.262398 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.262288    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.282015 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.281990    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.282578 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.282556    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wmf9\" (UniqueName: \"kubernetes.io/projected/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-kube-api-access-6wmf9\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.324690 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.324661    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:21.398237 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.397739    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:21.653952 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.653925    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"]
Oct 30 00:59:21.655897 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:59:21.655868    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5a0c33_0c8e_45a8_9fcb_433a6132af2b.slice/crio-75411d3469d4c80a2450230b271210c5e5a4738b73b6ba539292039a6ee79d70 WatchSource:0}: Error finding container 75411d3469d4c80a2450230b271210c5e5a4738b73b6ba539292039a6ee79d70: Status 404 returned error can't find the container with id 75411d3469d4c80a2450230b271210c5e5a4738b73b6ba539292039a6ee79d70
Oct 30 00:59:21.658662 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.658629    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:21.658751 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.658694    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:21.658751 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.658729    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:21.720265 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.720239    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"]
Oct 30 00:59:21.722040 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:59:21.722014    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee90c29_d7d2_41cc_b02d_5cbcf06c9b29.slice/crio-abb1e9febe37bae9e8d147aa6d0f1941e5c95c37ed676f910f7b1a532ca331aa WatchSource:0}: Error finding container abb1e9febe37bae9e8d147aa6d0f1941e5c95c37ed676f910f7b1a532ca331aa: Status 404 returned error can't find the container with id abb1e9febe37bae9e8d147aa6d0f1941e5c95c37ed676f910f7b1a532ca331aa
Oct 30 00:59:21.724966 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.724925    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:21.725064 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.725014    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:21.725126 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.725082    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:21.858480 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.857788    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl" event={"ID":"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b","Type":"ContainerStarted","Data":"63fb51d1789a4aa53e528ed5d079550ca126b5f77774cd9b93405eb79f753094"}
Oct 30 00:59:21.858480 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.857847    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl" event={"ID":"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b","Type":"ContainerStarted","Data":"75411d3469d4c80a2450230b271210c5e5a4738b73b6ba539292039a6ee79d70"}
Oct 30 00:59:21.859861 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.859825    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f" event={"ID":"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29","Type":"ContainerStarted","Data":"b75cf59de7e09c63251c75abadd6ecc32d6c5ee048c04035b3a1ba935d2e2113"}
Oct 30 00:59:21.860012 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.859882    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f" event={"ID":"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29","Type":"ContainerStarted","Data":"abb1e9febe37bae9e8d147aa6d0f1941e5c95c37ed676f910f7b1a532ca331aa"}
Oct 30 00:59:21.905117 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.903686    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl" podStartSLOduration=1.9036696800000001 podStartE2EDuration="1.90366968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:21.899840779 +0000 UTC m=+1070.649496090" watchObservedRunningTime="2026-10-30 00:59:21.90366968 +0000 UTC m=+1070.653324990"
Oct 30 00:59:21.940458 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:21.940364    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f" podStartSLOduration=1.940352301 podStartE2EDuration="1.940352301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:21.939273067 +0000 UTC m=+1070.688928376" watchObservedRunningTime="2026-10-30 00:59:21.940352301 +0000 UTC m=+1070.690007610"
Oct 30 00:59:22.325546 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:22.325510    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:22.327150 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:22.327120    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 00:59:22.327404 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:22.327374    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl" podUID="7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 00:59:22.402180 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:22.401894    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:22.404249 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:22.402378    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:59:22.404249 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:22.402438    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f" podUID="cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 00:59:23.279180 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:23.269314    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:59:23.279180 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:23.269375    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j" podUID="11e986da-b068-404f-91f7-4a8f109465d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:23.330200 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:23.330027    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 00:59:23.330200 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:23.330086    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl" podUID="7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 00:59:23.453493 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:23.445574    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:59:23.453493 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:23.445634    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f" podUID="cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 00:59:24.321179 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:24.318200    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:59:24.321179 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:24.318260    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j" podUID="11e986da-b068-404f-91f7-4a8f109465d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:24.326179 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:24.325474    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 00:59:24.326179 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:24.325527    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl" podUID="7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 00:59:24.418178 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:24.402506    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:59:24.418178 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:24.402568    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f" podUID="cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 00:59:24.625833 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:24.625746    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:24.647668 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:24.647586    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:24.657415 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:24.656534    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 00:59:25.325401 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:25.325368    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 00:59:25.325596 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:25.325419    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl" podUID="7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 00:59:25.401179 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:25.400529    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:59:25.401179 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:25.400589    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f" podUID="cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 00:59:26.037810 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.036950    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"]
Oct 30 00:59:26.042368 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.042345    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.074340 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.074300    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-664c2b-openshift-default-dockercfg-vvttz\""
Oct 30 00:59:26.087942 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.087863    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"]
Oct 30 00:59:26.153896 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.153859    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.158500 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.158471    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db496a54-58b2-4800-9cd5-0610da69a340-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.158715 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.158697    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.158884 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.158867    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db496a54-58b2-4800-9cd5-0610da69a340-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.158999 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.158984    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.176177 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.159118    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5q2z\" (UniqueName: \"kubernetes.io/projected/db496a54-58b2-4800-9cd5-0610da69a340-kube-api-access-z5q2z\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.177083 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.177058    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.177293 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.177261    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db496a54-58b2-4800-9cd5-0610da69a340-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.177427 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.177406    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.295234 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.294949    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.295234 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.295018    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db496a54-58b2-4800-9cd5-0610da69a340-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.295234 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.295052    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.295234 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.295112    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db496a54-58b2-4800-9cd5-0610da69a340-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.295234 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.295138    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.295234 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.295207    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5q2z\" (UniqueName: \"kubernetes.io/projected/db496a54-58b2-4800-9cd5-0610da69a340-kube-api-access-z5q2z\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.295691 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.295262    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.295691 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.295301    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db496a54-58b2-4800-9cd5-0610da69a340-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.295691 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.295331    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.303258 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.295844    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.303258 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.296109    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.303258 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.299830    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db496a54-58b2-4800-9cd5-0610da69a340-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.303258 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.300087    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.303258 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.300289    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"]
Oct 30 00:59:26.303258 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.302598    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.303745 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.303708    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db496a54-58b2-4800-9cd5-0610da69a340-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.305148 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.305128    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"
Oct 30 00:59:26.307816 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.307794    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.312242 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.312220    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"]
Oct 30 00:59:26.325392 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.325271    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 00:59:26.325392 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.325322    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl" podUID="7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 00:59:26.327125 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.327106    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:59:26.327338 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.327318    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:59:26.347059 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.347030    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db496a54-58b2-4800-9cd5-0610da69a340-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.347170 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.347088    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5q2z\" (UniqueName: \"kubernetes.io/projected/db496a54-58b2-4800-9cd5-0610da69a340-kube-api-access-z5q2z\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.393177 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.392302    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:59:26.393177 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.392460    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 00:59:26.393177 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.392716    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:26.397173 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.397034    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rksl4\" (UniqueName: \"kubernetes.io/projected/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-kube-api-access-rksl4\") pod \"maas-api-revoke-sub-b9a8b147-5wj8f\" (UID: \"365ddc41-5d93-42dd-9b56-6109b3cd0dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"
Oct 30 00:59:26.411707 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.411243    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9a8b147-5wj8f\" (UID: \"365ddc41-5d93-42dd-9b56-6109b3cd0dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"
Oct 30 00:59:26.411707 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.411304    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9a8b147-5wj8f\" (UID: \"365ddc41-5d93-42dd-9b56-6109b3cd0dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"
Oct 30 00:59:26.413295 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.413270    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:59:26.413397 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.413323    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f" podUID="cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 00:59:26.512513 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.512477    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rksl4\" (UniqueName: \"kubernetes.io/projected/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-kube-api-access-rksl4\") pod \"maas-api-revoke-sub-b9a8b147-5wj8f\" (UID: \"365ddc41-5d93-42dd-9b56-6109b3cd0dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"
Oct 30 00:59:26.512705 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.512537    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9a8b147-5wj8f\" (UID: \"365ddc41-5d93-42dd-9b56-6109b3cd0dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"
Oct 30 00:59:26.512705 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.512564    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9a8b147-5wj8f\" (UID: \"365ddc41-5d93-42dd-9b56-6109b3cd0dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"
Oct 30 00:59:26.513364 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.513337    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9a8b147-5wj8f\" (UID: \"365ddc41-5d93-42dd-9b56-6109b3cd0dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"
Oct 30 00:59:26.515092 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.515067    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9a8b147-5wj8f\" (UID: \"365ddc41-5d93-42dd-9b56-6109b3cd0dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"
Oct 30 00:59:26.579233 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.566406    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rksl4\" (UniqueName: \"kubernetes.io/projected/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-kube-api-access-rksl4\") pod \"maas-api-revoke-sub-b9a8b147-5wj8f\" (UID: \"365ddc41-5d93-42dd-9b56-6109b3cd0dc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"
Oct 30 00:59:26.692179 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.649237    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"
Oct 30 00:59:26.811202 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.809674    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"]
Oct 30 00:59:26.816195 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:59:26.815937    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb496a54_58b2_4800_9cd5_0610da69a340.slice/crio-029cdcc7e544a79cd11701d2ec392a3db06f18281cdadecb50dd24cd50201d57 WatchSource:0}: Error finding container 029cdcc7e544a79cd11701d2ec392a3db06f18281cdadecb50dd24cd50201d57: Status 404 returned error can't find the container with id 029cdcc7e544a79cd11701d2ec392a3db06f18281cdadecb50dd24cd50201d57
Oct 30 00:59:26.818941 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.818898    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:26.819019 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.818995    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:26.819063 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:26.819037    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:27.012017 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:27.011992    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d" event={"ID":"db496a54-58b2-4800-9cd5-0610da69a340","Type":"ContainerStarted","Data":"029cdcc7e544a79cd11701d2ec392a3db06f18281cdadecb50dd24cd50201d57"}
Oct 30 00:59:27.113768 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:59:27.113731    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365ddc41_5d93_42dd_9b56_6109b3cd0dc7.slice/crio-538560309cdcb1b41aa0785ce8e58c382f172e6000bb7d4a9e971b328c146162 WatchSource:0}: Error finding container 538560309cdcb1b41aa0785ce8e58c382f172e6000bb7d4a9e971b328c146162: Status 404 returned error can't find the container with id 538560309cdcb1b41aa0785ce8e58c382f172e6000bb7d4a9e971b328c146162
Oct 30 00:59:27.117491 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:27.116412    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"]
Oct 30 00:59:27.326109 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:27.326069    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 00:59:27.326335 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:27.326131    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl" podUID="7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 00:59:27.401220 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:27.400496    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:59:27.401220 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:27.400554    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f" podUID="cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 00:59:28.027271 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:28.027238    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f" event={"ID":"365ddc41-5d93-42dd-9b56-6109b3cd0dc7","Type":"ContainerStarted","Data":"b3a18ce18824968f8ac7e40152f5927c449e18e790e50a0ef737f3fddd0a35fb"}
Oct 30 00:59:28.027490 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:28.027476    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f" event={"ID":"365ddc41-5d93-42dd-9b56-6109b3cd0dc7","Type":"ContainerStarted","Data":"538560309cdcb1b41aa0785ce8e58c382f172e6000bb7d4a9e971b328c146162"}
Oct 30 00:59:28.030008 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:28.029985    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d" event={"ID":"db496a54-58b2-4800-9cd5-0610da69a340","Type":"ContainerStarted","Data":"398df3c35b00bbeb936eecedfbb3ff55de67bef2db8bb9fc54ae0dcb4be0c694"}
Oct 30 00:59:28.089177 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:28.072611    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f" podStartSLOduration=2.072591568 podStartE2EDuration="2.072591568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:28.067637872 +0000 UTC m=+1076.817293182" watchObservedRunningTime="2026-10-30 00:59:28.072591568 +0000 UTC m=+1076.822246877"
Oct 30 00:59:28.112923 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:28.112856    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d" podStartSLOduration=2.112839096 podStartE2EDuration="2.112839096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:28.110748064 +0000 UTC m=+1076.860403374" watchObservedRunningTime="2026-10-30 00:59:28.112839096 +0000 UTC m=+1076.862494418"
Oct 30 00:59:28.325256 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:28.325151    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 00:59:28.325256 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:28.325225    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl" podUID="7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 00:59:28.394719 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:28.394684    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:28.396396 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:28.396368    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:59:28.396505 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:28.396425    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d" podUID="db496a54-58b2-4800-9cd5-0610da69a340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 00:59:28.398750 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:28.398724    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 00:59:28.398835 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:28.398771    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f" podUID="cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 00:59:29.041648 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:29.041610    4425 generic.go:356] "Generic (PLEG): container finished" podID="365ddc41-5d93-42dd-9b56-6109b3cd0dc7" containerID="b3a18ce18824968f8ac7e40152f5927c449e18e790e50a0ef737f3fddd0a35fb" exitCode=0
Oct 30 00:59:29.041822 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:29.041704    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f" event={"ID":"365ddc41-5d93-42dd-9b56-6109b3cd0dc7","Type":"ContainerDied","Data":"b3a18ce18824968f8ac7e40152f5927c449e18e790e50a0ef737f3fddd0a35fb"}
Oct 30 00:59:29.395886 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:29.395802    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:59:29.396902 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:29.396860    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d" podUID="db496a54-58b2-4800-9cd5-0610da69a340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 00:59:29.768502 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:29.768465    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:29.771316 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:29.771289    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:29.784568 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:29.783385    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 00:59:29.889264 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:29.889221    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:29.890321 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:29.890298    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:29.891517 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:29.891495    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 00:59:29.944707 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:29.944665    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 00:59:30.288969 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:30.288943    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"
Oct 30 00:59:30.394740 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:30.394662    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:59:30.394740 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:30.394716    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d" podUID="db496a54-58b2-4800-9cd5-0610da69a340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 00:59:30.423328 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:30.423300    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-kube-api-access-rksl4\" (UniqueName: \"kubernetes.io/projected/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-kube-api-access-rksl4\") pod \"365ddc41-5d93-42dd-9b56-6109b3cd0dc7\" (UID: \"365ddc41-5d93-42dd-9b56-6109b3cd0dc7\") "
Oct 30 00:59:30.423746 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:30.423342    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-maas-api-openshift-ca\") pod \"365ddc41-5d93-42dd-9b56-6109b3cd0dc7\" (UID: \"365ddc41-5d93-42dd-9b56-6109b3cd0dc7\") "
Oct 30 00:59:30.423746 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:30.423391    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-maas-api-xks-ca\") pod \"365ddc41-5d93-42dd-9b56-6109b3cd0dc7\" (UID: \"365ddc41-5d93-42dd-9b56-6109b3cd0dc7\") "
Oct 30 00:59:30.423746 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:30.423683    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-maas-api-openshift-ca" pod "365ddc41-5d93-42dd-9b56-6109b3cd0dc7" (UID: "365ddc41-5d93-42dd-9b56-6109b3cd0dc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:30.423872 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:30.423824    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:59:30.425421 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:30.425398    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-maas-api-xks-ca" pod "365ddc41-5d93-42dd-9b56-6109b3cd0dc7" (UID: "365ddc41-5d93-42dd-9b56-6109b3cd0dc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:59:30.425619 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:30.425604    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-kube-api-access-rksl4" pod "365ddc41-5d93-42dd-9b56-6109b3cd0dc7" (UID: "365ddc41-5d93-42dd-9b56-6109b3cd0dc7"). InnerVolumeSpecName "kube-api-access-rksl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:30.524602 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:30.524573    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:59:30.524602 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:30.524598    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rksl4\" (UniqueName: \"kubernetes.io/projected/365ddc41-5d93-42dd-9b56-6109b3cd0dc7-kube-api-access-rksl4\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:59:31.085180 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:31.083682    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"
Oct 30 00:59:31.085394 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:31.085253    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f" event={"ID":"365ddc41-5d93-42dd-9b56-6109b3cd0dc7","Type":"ContainerDied","Data":"538560309cdcb1b41aa0785ce8e58c382f172e6000bb7d4a9e971b328c146162"}
Oct 30 00:59:31.085394 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:31.085307    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="538560309cdcb1b41aa0785ce8e58c382f172e6000bb7d4a9e971b328c146162"
Oct 30 00:59:31.440245 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:31.438122    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:59:31.440245 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:31.438208    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d" podUID="db496a54-58b2-4800-9cd5-0610da69a340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 00:59:31.591700 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:31.591668    4425 scope.go:122] "RemoveContainer" containerID="9f66848597fe7b880814fa7a7895ce6eeef5573a6238bbe54c5ceebaca3f1842"
Oct 30 00:59:31.645861 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:31.645717    4425 scope.go:122] "RemoveContainer" containerID="eca26c39e700e12ceee1c61deaf51af5f90e4ff097c1f4fe1469e55b5bf4181f"
Oct 30 00:59:32.414745 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:32.414249    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:59:32.414745 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:32.414327    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d" podUID="db496a54-58b2-4800-9cd5-0610da69a340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 00:59:32.570372 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:32.570320    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" podUID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:32.674241 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:32.674129    4425 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7" podUID="834166ed-5c63-4854-9e89-c45041553630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:32.704396 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:32.703369    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:32.830189 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:32.830037    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7" podUID="834166ed-5c63-4854-9e89-c45041553630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 00:59:33.106322 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:33.106278    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc" podUID="c8d1f900-b8c4-43dc-814e-9491e560ea3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 00:59:33.194331 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:33.194268    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w" podUID="66a19982-1f16-4a03-a629-bf9b7d6b50d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 00:59:33.281300 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:33.281250    4425 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" podUID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:33.401634 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:33.401543    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:59:33.402670 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:33.402636    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d" podUID="db496a54-58b2-4800-9cd5-0610da69a340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 00:59:33.592188 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:33.583807    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" podUID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:33.619302 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:33.619269    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:59:33.695782 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:33.695685    4425 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:33.813181 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:33.749567    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:33.813181 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:33.749998    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:34.118527 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:34.118473    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc" podUID="c8d1f900-b8c4-43dc-814e-9491e560ea3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:34.494967 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:34.118991    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw" podUID="4d3d278e-ecd0-46ee-a203-33003899a369" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 00:59:34.494967 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:34.119453    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f" podUID="0824ed99-cf93-4047-b19f-0ad807263acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 00:59:34.494967 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:34.258955    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s" podUID="b8b27e40-2516-4d18-8c84-5e0925b3f4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:34.494967 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:34.259059    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w" podUID="66a19982-1f16-4a03-a629-bf9b7d6b50d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 00:59:34.494967 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:34.365049    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:59:34.494967 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:34.440840    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:59:34.494967 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:34.440924    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:59:34.523361 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:34.518419    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:59:34.523361 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:34.518507    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d" podUID="db496a54-58b2-4800-9cd5-0610da69a340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 00:59:34.758061 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:34.757966    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 00:59:34.830303 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:34.830259    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7" podUID="834166ed-5c63-4854-9e89-c45041553630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:35.398182 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:35.397831    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:59:35.398182 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:35.397907    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d" podUID="db496a54-58b2-4800-9cd5-0610da69a340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 00:59:35.426416 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:35.426365    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc" podUID="c8d1f900-b8c4-43dc-814e-9491e560ea3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:35.427102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:35.426652    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw" podUID="4d3d278e-ecd0-46ee-a203-33003899a369" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:35.450190 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:35.448226    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w" podUID="66a19982-1f16-4a03-a629-bf9b7d6b50d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:35.495085 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:35.495054    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:59:35.539940 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:35.539916    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:35.706138 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:35.706057    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:59:35.741735 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:35.740562    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:59:35.789222 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:35.789193    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:59:36.394607 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:36.394570    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 00:59:36.394796 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:36.394619    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d" podUID="db496a54-58b2-4800-9cd5-0610da69a340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 00:59:37.398592 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:37.398563    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:37.399053 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:37.398931    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:37.399576 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:37.399555    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 00:59:37.561597 ip-10-0-133-32 kubenswrapper[4425]: E1030 00:59:37.561559    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 00:59:37.793863 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:37.793830    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"]
Oct 30 00:59:37.794599 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:37.794583    4425 cpu_manager.go:414] "containerMap: removing container" podUID="365ddc41-5d93-42dd-9b56-6109b3cd0dc7" containerName="revoke-keys"
Oct 30 00:59:37.794665 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:37.794602    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365ddc41-5d93-42dd-9b56-6109b3cd0dc7" containerName="revoke-keys"
Oct 30 00:59:37.794748 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:37.794733    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="365ddc41-5d93-42dd-9b56-6109b3cd0dc7" containerName="revoke-keys"
Oct 30 00:59:37.800038 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:37.800014    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"
Oct 30 00:59:37.815112 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:37.815087    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 00:59:37.815843 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:37.815731    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:59:37.815968 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:37.815804    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 00:59:37.816690 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:37.816667    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"]
Oct 30 00:59:37.820929 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:37.820894    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:59:37.911513 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:37.911473    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fjzx\" (UniqueName: \"kubernetes.io/projected/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-kube-api-access-5fjzx\") pod \"maas-api-revoke-sub-771f502b-b5lnt\" (UID: \"47cca8f2-075c-46d2-9d06-07c5de2e2fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"
Oct 30 00:59:37.911706 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:37.911626    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-771f502b-b5lnt\" (UID: \"47cca8f2-075c-46d2-9d06-07c5de2e2fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"
Oct 30 00:59:37.911706 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:37.911699    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-771f502b-b5lnt\" (UID: \"47cca8f2-075c-46d2-9d06-07c5de2e2fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"
Oct 30 00:59:38.013226 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:38.013185    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fjzx\" (UniqueName: \"kubernetes.io/projected/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-kube-api-access-5fjzx\") pod \"maas-api-revoke-sub-771f502b-b5lnt\" (UID: \"47cca8f2-075c-46d2-9d06-07c5de2e2fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"
Oct 30 00:59:38.013399 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:38.013333    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-771f502b-b5lnt\" (UID: \"47cca8f2-075c-46d2-9d06-07c5de2e2fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"
Oct 30 00:59:38.013399 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:38.013391    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-771f502b-b5lnt\" (UID: \"47cca8f2-075c-46d2-9d06-07c5de2e2fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"
Oct 30 00:59:38.014017 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:38.013997    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-771f502b-b5lnt\" (UID: \"47cca8f2-075c-46d2-9d06-07c5de2e2fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"
Oct 30 00:59:38.015921 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:38.015892    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-771f502b-b5lnt\" (UID: \"47cca8f2-075c-46d2-9d06-07c5de2e2fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"
Oct 30 00:59:38.034403 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:38.034369    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fjzx\" (UniqueName: \"kubernetes.io/projected/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-kube-api-access-5fjzx\") pod \"maas-api-revoke-sub-771f502b-b5lnt\" (UID: \"47cca8f2-075c-46d2-9d06-07c5de2e2fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"
Oct 30 00:59:38.116041 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:38.115966    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"
Oct 30 00:59:38.421977 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:38.421900    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"]
Oct 30 00:59:38.426077 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:59:38.426041    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47cca8f2_075c_46d2_9d06_07c5de2e2fc8.slice/crio-b38bf84cbe7622b24d1c7b0a85aa79b73bacec420ef95feadef6a424ac63498a WatchSource:0}: Error finding container b38bf84cbe7622b24d1c7b0a85aa79b73bacec420ef95feadef6a424ac63498a: Status 404 returned error can't find the container with id b38bf84cbe7622b24d1c7b0a85aa79b73bacec420ef95feadef6a424ac63498a
Oct 30 00:59:39.341755 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:39.341720    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt" event={"ID":"47cca8f2-075c-46d2-9d06-07c5de2e2fc8","Type":"ContainerStarted","Data":"87b6d1cb4d299319c2929ce02154c5611b156c411be4d9ccb52cc2673990cbdf"}
Oct 30 00:59:39.341976 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:39.341773    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt" event={"ID":"47cca8f2-075c-46d2-9d06-07c5de2e2fc8","Type":"ContainerStarted","Data":"b38bf84cbe7622b24d1c7b0a85aa79b73bacec420ef95feadef6a424ac63498a"}
Oct 30 00:59:39.379196 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:39.379122    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt" podStartSLOduration=2.379106578 podStartE2EDuration="2.379106578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:39.376759461 +0000 UTC m=+1088.126414769" watchObservedRunningTime="2026-10-30 00:59:39.379106578 +0000 UTC m=+1088.128761886"
Oct 30 00:59:40.348654 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:40.348620    4425 generic.go:356] "Generic (PLEG): container finished" podID="47cca8f2-075c-46d2-9d06-07c5de2e2fc8" containerID="87b6d1cb4d299319c2929ce02154c5611b156c411be4d9ccb52cc2673990cbdf" exitCode=0
Oct 30 00:59:40.349080 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:40.348711    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt" event={"ID":"47cca8f2-075c-46d2-9d06-07c5de2e2fc8","Type":"ContainerDied","Data":"87b6d1cb4d299319c2929ce02154c5611b156c411be4d9ccb52cc2673990cbdf"}
Oct 30 00:59:41.088022 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.087984    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"]
Oct 30 00:59:41.092133 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.092108    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.120327 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.120289    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"]
Oct 30 00:59:41.151761 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.151711    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.151761 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.151761    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.151983 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.151883    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.151983 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.151922    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.152066 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.152044    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.152102 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.152087    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.152136 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.152105    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.152248 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.152235    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4hts\" (UniqueName: \"kubernetes.io/projected/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-kube-api-access-w4hts\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.152291 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.152260    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.253080 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.253043    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.253080 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.253088    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.253341 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.253145    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.253341 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.253214    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.253341 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.253282    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.253341 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.253315    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.253341 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.253337    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.253582 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.253406    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4hts\" (UniqueName: \"kubernetes.io/projected/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-kube-api-access-w4hts\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.253582 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.253437    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.253582 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.253564    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.253746 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.253616    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.253746 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.253668    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.253843 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.253741    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.254139 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.254117    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.255997 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.255960    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.256138 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.256115    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.274783 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.274749    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.275318 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.275293    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4hts\" (UniqueName: \"kubernetes.io/projected/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-kube-api-access-w4hts\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.412004 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.411963    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:41.514409 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.514165    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"
Oct 30 00:59:41.571216 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.571143    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-maas-api-xks-ca\") pod \"47cca8f2-075c-46d2-9d06-07c5de2e2fc8\" (UID: \"47cca8f2-075c-46d2-9d06-07c5de2e2fc8\") "
Oct 30 00:59:41.571399 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.571285    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-maas-api-openshift-ca\") pod \"47cca8f2-075c-46d2-9d06-07c5de2e2fc8\" (UID: \"47cca8f2-075c-46d2-9d06-07c5de2e2fc8\") "
Oct 30 00:59:41.571467 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.571444    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-kube-api-access-5fjzx\" (UniqueName: \"kubernetes.io/projected/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-kube-api-access-5fjzx\") pod \"47cca8f2-075c-46d2-9d06-07c5de2e2fc8\" (UID: \"47cca8f2-075c-46d2-9d06-07c5de2e2fc8\") "
Oct 30 00:59:41.571668 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.571638    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-maas-api-openshift-ca" pod "47cca8f2-075c-46d2-9d06-07c5de2e2fc8" (UID: "47cca8f2-075c-46d2-9d06-07c5de2e2fc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:41.572035 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.572012    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:59:41.573728 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.573699    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-maas-api-xks-ca" pod "47cca8f2-075c-46d2-9d06-07c5de2e2fc8" (UID: "47cca8f2-075c-46d2-9d06-07c5de2e2fc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:59:41.573912 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.573887    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-kube-api-access-5fjzx" pod "47cca8f2-075c-46d2-9d06-07c5de2e2fc8" (UID: "47cca8f2-075c-46d2-9d06-07c5de2e2fc8"). InnerVolumeSpecName "kube-api-access-5fjzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:41.673113 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.673013    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fjzx\" (UniqueName: \"kubernetes.io/projected/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-kube-api-access-5fjzx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:59:41.673113 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.673047    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47cca8f2-075c-46d2-9d06-07c5de2e2fc8-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 00:59:41.759056 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.759023    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"]
Oct 30 00:59:41.760567 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:59:41.760534    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba50e09_63b0_4d2e_b18e_f8fa1a30ef79.slice/crio-40a7f1e996f80e064f3f8c18ce94d9b0558e99a193ca1e44c64ff6b8f894620c WatchSource:0}: Error finding container 40a7f1e996f80e064f3f8c18ce94d9b0558e99a193ca1e44c64ff6b8f894620c: Status 404 returned error can't find the container with id 40a7f1e996f80e064f3f8c18ce94d9b0558e99a193ca1e44c64ff6b8f894620c
Oct 30 00:59:41.763134 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.763103    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:41.763237 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.763202    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:41.763282 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:41.763246    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:42.374849 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:42.374810    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"
Oct 30 00:59:42.375105 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:42.374822    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt" event={"ID":"47cca8f2-075c-46d2-9d06-07c5de2e2fc8","Type":"ContainerDied","Data":"b38bf84cbe7622b24d1c7b0a85aa79b73bacec420ef95feadef6a424ac63498a"}
Oct 30 00:59:42.375105 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:42.374944    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b38bf84cbe7622b24d1c7b0a85aa79b73bacec420ef95feadef6a424ac63498a"
Oct 30 00:59:42.377115 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:42.377077    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24" event={"ID":"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79","Type":"ContainerStarted","Data":"e717335be880557407fe751ef248ff254376d4f1323e707e0d42a2135a255f66"}
Oct 30 00:59:42.377290 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:42.377139    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24" event={"ID":"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79","Type":"ContainerStarted","Data":"40a7f1e996f80e064f3f8c18ce94d9b0558e99a193ca1e44c64ff6b8f894620c"}
Oct 30 00:59:42.412568 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:42.412535    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:42.413994 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:42.413958    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:59:42.414145 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:42.414006    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24" podUID="3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:59:42.428766 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:42.428713    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24" podStartSLOduration=1.428700076 podStartE2EDuration="1.428700076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:42.420442301 +0000 UTC m=+1091.170097604" watchObservedRunningTime="2026-10-30 00:59:42.428700076 +0000 UTC m=+1091.178355385"
Oct 30 00:59:43.413067 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:43.413027    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:59:43.413649 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:43.413092    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24" podUID="3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:59:44.421323 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:44.421282    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:59:44.421892 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:44.421360    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24" podUID="3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:59:45.462692 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:45.462635    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:59:45.569960 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:45.569910    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24" podUID="3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:59:45.631181 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:45.563736    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:45.775306 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:45.775260    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w" podUID="66a19982-1f16-4a03-a629-bf9b7d6b50d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:45.804187 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:45.791624    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw" podUID="4d3d278e-ecd0-46ee-a203-33003899a369" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:45.810370 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:45.810328    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7" podUID="834166ed-5c63-4854-9e89-c45041553630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:45.856198 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:45.855280    4425 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z" podUID="b4ed47ff-0328-4645-be5f-6a6d51a94fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:45.961312 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:45.960253    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z" podUID="b4ed47ff-0328-4645-be5f-6a6d51a94fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:46.037361 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:46.037256    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f" podUID="0824ed99-cf93-4047-b19f-0ad807263acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:46.083615 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:46.083568    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2" podUID="13f688db-25e0-4a88-9d1e-c3fd78b7f659" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:46.260188 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:46.255272    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s" podUID="b8b27e40-2516-4d18-8c84-5e0925b3f4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 00:59:46.362342 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:46.362235    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" podUID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:46.431182 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:46.430280    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:59:46.431182 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:46.430361    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24" podUID="3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:59:46.472181 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:46.471493    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc" podUID="c8d1f900-b8c4-43dc-814e-9491e560ea3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:46.682977 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:46.682886    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 00:59:46.695022 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:46.693972    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:46.783853 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:46.781512    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w" podUID="66a19982-1f16-4a03-a629-bf9b7d6b50d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:46.783853 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:46.782229    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:59:46.816451 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:46.816376    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7" podUID="834166ed-5c63-4854-9e89-c45041553630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:46.828338 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:46.817283    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:59:46.828338 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:46.819187    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw" podUID="4d3d278e-ecd0-46ee-a203-33003899a369" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:46.828338 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:46.819492    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:59:47.003227 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:46.987280    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z" podUID="b4ed47ff-0328-4645-be5f-6a6d51a94fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:47.003227 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:46.988071    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:59:47.029700 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.029674    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 00:59:47.043605 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.043234    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f" podUID="0824ed99-cf93-4047-b19f-0ad807263acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:47.045587 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.045565    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:59:47.047477 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.047459    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 00:59:47.062552 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.062502    4425 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2" podUID="13f688db-25e0-4a88-9d1e-c3fd78b7f659" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Oct 30 00:59:47.098620 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.093374    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2" podUID="13f688db-25e0-4a88-9d1e-c3fd78b7f659" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:47.098620 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.098335    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:59:47.443189 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.427589    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" podUID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 00:59:47.443189 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.428239    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:59:47.483263 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.482202    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc" podUID="c8d1f900-b8c4-43dc-814e-9491e560ea3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:47.652250 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.652207    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s" podUID="b8b27e40-2516-4d18-8c84-5e0925b3f4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:47.708538 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.708410    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:47.747743 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.747708    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:59:47.748034 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.748004    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24" podUID="3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:59:47.791439 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.791397    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w" podUID="66a19982-1f16-4a03-a629-bf9b7d6b50d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:47.818864 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.778474    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:59:47.819284 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.819264    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:59:47.829178 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.819126    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7" podUID="834166ed-5c63-4854-9e89-c45041553630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:47.988964 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:47.988888    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 00:59:48.046875 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:48.046272    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f" podUID="0824ed99-cf93-4047-b19f-0ad807263acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:59:48.082300 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:48.082272    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 00:59:48.098646 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:48.090922    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 00:59:48.125293 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:48.122688    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2" podUID="13f688db-25e0-4a88-9d1e-c3fd78b7f659" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 00:59:48.210741 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:48.210692    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 00:59:48.213884 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:48.213861    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 00:59:48.216355 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:48.216333    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 00:59:48.224519 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:48.224498    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 00:59:48.235927 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:48.235907    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 00:59:48.412673 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:48.412634    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:59:48.412869 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:48.412695    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24" podUID="3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:59:49.412946 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:49.412909    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:59:49.413513 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:49.412976    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24" podUID="3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:59:50.769333 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:50.769272    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:50.770273 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:50.770233    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:50.770657 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:50.770639    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 00:59:51.982997 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:51.982946    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"]
Oct 30 00:59:51.983778 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:51.983752    4425 cpu_manager.go:414] "containerMap: removing container" podUID="47cca8f2-075c-46d2-9d06-07c5de2e2fc8" containerName="revoke-keys"
Oct 30 00:59:51.983778 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:51.983776    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47cca8f2-075c-46d2-9d06-07c5de2e2fc8" containerName="revoke-keys"
Oct 30 00:59:51.983992 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:51.983963    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="47cca8f2-075c-46d2-9d06-07c5de2e2fc8" containerName="revoke-keys"
Oct 30 00:59:51.987345 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:51.987325    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.018411 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.018376    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"]
Oct 30 00:59:52.030328 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.030272    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.030660 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.030630    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8653a528-c49c-41e3-abbb-739353e81e66-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.030833 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.030703    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8653a528-c49c-41e3-abbb-739353e81e66-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.030833 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.030798    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.030976 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.030891    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.030976 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.030914    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.030976 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.030968    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkrkt\" (UniqueName: \"kubernetes.io/projected/8653a528-c49c-41e3-abbb-739353e81e66-kube-api-access-wkrkt\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.031135 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.030996    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.031135 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.031020    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8653a528-c49c-41e3-abbb-739353e81e66-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.132151 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.132119    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.132151 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.132181    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.132447 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.132216    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkrkt\" (UniqueName: \"kubernetes.io/projected/8653a528-c49c-41e3-abbb-739353e81e66-kube-api-access-wkrkt\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.132447 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.132235    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.132447 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.132251    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8653a528-c49c-41e3-abbb-739353e81e66-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.132447 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.132304    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.132447 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.132381    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8653a528-c49c-41e3-abbb-739353e81e66-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.132447 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.132432    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8653a528-c49c-41e3-abbb-739353e81e66-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.132755 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.132530    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.132755 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.132637    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.132868 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.132815    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.132868 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.132835    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.132939 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.132905    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.133380 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.133358    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8653a528-c49c-41e3-abbb-739353e81e66-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.134754 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.134730    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.135014 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.134996    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8653a528-c49c-41e3-abbb-739353e81e66-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.153268 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.153245    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8653a528-c49c-41e3-abbb-739353e81e66-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.153527 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.153510    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkrkt\" (UniqueName: \"kubernetes.io/projected/8653a528-c49c-41e3-abbb-739353e81e66-kube-api-access-wkrkt\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.303164 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.303133    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:52.622273 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.622236    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"]
Oct 30 00:59:52.623615 ip-10-0-133-32 kubenswrapper[4425]: W1030 00:59:52.623581    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8653a528_c49c_41e3_abbb_739353e81e66.slice/crio-0bf8cc9ff78c68892db3f69e0224ea50b19bbde3005db5ead12ef4285dffa2ed WatchSource:0}: Error finding container 0bf8cc9ff78c68892db3f69e0224ea50b19bbde3005db5ead12ef4285dffa2ed: Status 404 returned error can't find the container with id 0bf8cc9ff78c68892db3f69e0224ea50b19bbde3005db5ead12ef4285dffa2ed
Oct 30 00:59:52.626100 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.626070    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:52.626190 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.626136    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:52.626229 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.626195    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 00:59:52.915984 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.915896    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp" event={"ID":"8653a528-c49c-41e3-abbb-739353e81e66","Type":"ContainerStarted","Data":"4484e85f1819f19c2fca0c087e0f2b601ff27352a2667988ce51170c18edab6f"}
Oct 30 00:59:52.915984 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.915965    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp" event={"ID":"8653a528-c49c-41e3-abbb-739353e81e66","Type":"ContainerStarted","Data":"0bf8cc9ff78c68892db3f69e0224ea50b19bbde3005db5ead12ef4285dffa2ed"}
Oct 30 00:59:52.958302 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:52.958244    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp" podStartSLOduration=1.958226298 podStartE2EDuration="1.958226298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:52.955823249 +0000 UTC m=+1101.705478558" watchObservedRunningTime="2026-10-30 00:59:52.958226298 +0000 UTC m=+1101.707881608"
Oct 30 00:59:53.303658 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:53.303625    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 00:59:53.305079 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:53.305048    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:59:53.305260 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:53.305130    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp" podUID="8653a528-c49c-41e3-abbb-739353e81e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:59:54.304108 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:54.303754    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:59:54.304108 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:54.303811    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp" podUID="8653a528-c49c-41e3-abbb-739353e81e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:59:55.304340 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:55.304303    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:59:55.304747 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:55.304352    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp" podUID="8653a528-c49c-41e3-abbb-739353e81e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:59:56.313452 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:56.313408    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:59:56.313989 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:56.313961    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp" podUID="8653a528-c49c-41e3-abbb-739353e81e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:59:57.304538 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:57.304496    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:59:57.304749 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:57.304567    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp" podUID="8653a528-c49c-41e3-abbb-739353e81e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:59:58.304531 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:58.304490    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:59:58.304925 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:58.304554    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp" podUID="8653a528-c49c-41e3-abbb-739353e81e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:59:59.304462 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:59.304434    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:59:59.304637 ip-10-0-133-32 kubenswrapper[4425]: I1030 00:59:59.304481    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp" podUID="8653a528-c49c-41e3-abbb-739353e81e66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 01:00:00.163995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.163963    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75"]
Oct 30 01:00:00.169356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.169333    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt"]
Oct 30 01:00:00.169481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.169442    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75"
Oct 30 01:00:00.173971 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.173948    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt"
Oct 30 01:00:00.182928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.182909    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns"]
Oct 30 01:00:00.187228 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.187200    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75"]
Oct 30 01:00:00.187344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.187299    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns"
Oct 30 01:00:00.191551 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.191530    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt"]
Oct 30 01:00:00.192915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.192683    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:00:00.193521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.192814    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:00:00.206797 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.206777    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns"]
Oct 30 01:00:00.264379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.264350    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5llnd\" (UniqueName: \"kubernetes.io/projected/110d4727-ac79-4852-8437-f971afc20244-kube-api-access-5llnd\") pod \"maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75\" (UID: \"110d4727-ac79-4852-8437-f971afc20244\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75"
Oct 30 01:00:00.264379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.264381    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rttxx\" (UniqueName: \"kubernetes.io/projected/078d4d89-6bcf-41a5-9755-c8929d96a4c0-kube-api-access-rttxx\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt\" (UID: \"078d4d89-6bcf-41a5-9755-c8929d96a4c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt"
Oct 30 01:00:00.264571 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.264415    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns\" (UID: \"75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns"
Oct 30 01:00:00.264606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.264589    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b642\" (UniqueName: \"kubernetes.io/projected/75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a-kube-api-access-8b642\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns\" (UID: \"75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns"
Oct 30 01:00:00.264653 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.264617    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110d4727-ac79-4852-8437-f971afc20244-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75\" (UID: \"110d4727-ac79-4852-8437-f971afc20244\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75"
Oct 30 01:00:00.264720 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.264709    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/078d4d89-6bcf-41a5-9755-c8929d96a4c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt\" (UID: \"078d4d89-6bcf-41a5-9755-c8929d96a4c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt"
Oct 30 01:00:00.288237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.288214    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:00:00.307605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.307574    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 01:00:00.308040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.308024    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 01:00:00.308770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.308753    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 01:00:00.365328 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.365297    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5llnd\" (UniqueName: \"kubernetes.io/projected/110d4727-ac79-4852-8437-f971afc20244-kube-api-access-5llnd\") pod \"maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75\" (UID: \"110d4727-ac79-4852-8437-f971afc20244\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75"
Oct 30 01:00:00.365442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.365333    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rttxx\" (UniqueName: \"kubernetes.io/projected/078d4d89-6bcf-41a5-9755-c8929d96a4c0-kube-api-access-rttxx\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt\" (UID: \"078d4d89-6bcf-41a5-9755-c8929d96a4c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt"
Oct 30 01:00:00.365442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.365367    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns\" (UID: \"75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns"
Oct 30 01:00:00.365579 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.365487    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b642\" (UniqueName: \"kubernetes.io/projected/75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a-kube-api-access-8b642\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns\" (UID: \"75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns"
Oct 30 01:00:00.365579 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.365516    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110d4727-ac79-4852-8437-f971afc20244-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75\" (UID: \"110d4727-ac79-4852-8437-f971afc20244\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75"
Oct 30 01:00:00.365691 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.365584    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/078d4d89-6bcf-41a5-9755-c8929d96a4c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt\" (UID: \"078d4d89-6bcf-41a5-9755-c8929d96a4c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt"
Oct 30 01:00:00.366127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.366109    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns\" (UID: \"75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns"
Oct 30 01:00:00.366301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.366285    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110d4727-ac79-4852-8437-f971afc20244-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75\" (UID: \"110d4727-ac79-4852-8437-f971afc20244\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75"
Oct 30 01:00:00.366412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.366392    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/078d4d89-6bcf-41a5-9755-c8929d96a4c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt\" (UID: \"078d4d89-6bcf-41a5-9755-c8929d96a4c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt"
Oct 30 01:00:00.386286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.386259    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rttxx\" (UniqueName: \"kubernetes.io/projected/078d4d89-6bcf-41a5-9755-c8929d96a4c0-kube-api-access-rttxx\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt\" (UID: \"078d4d89-6bcf-41a5-9755-c8929d96a4c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt"
Oct 30 01:00:00.386508 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.386487    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b642\" (UniqueName: \"kubernetes.io/projected/75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a-kube-api-access-8b642\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns\" (UID: \"75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns"
Oct 30 01:00:00.386594 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.386580    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5llnd\" (UniqueName: \"kubernetes.io/projected/110d4727-ac79-4852-8437-f971afc20244-kube-api-access-5llnd\") pod \"maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75\" (UID: \"110d4727-ac79-4852-8437-f971afc20244\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75"
Oct 30 01:00:00.484909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.484830    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75"
Oct 30 01:00:00.499030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.499010    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt"
Oct 30 01:00:00.513207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.513187    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns"
Oct 30 01:00:00.794990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.794966    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75"]
Oct 30 01:00:00.795499 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:00:00.795472    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110d4727_ac79_4852_8437_f971afc20244.slice/crio-7e82a91d68d9a83bf4b8d23c138223a14100468693b9393dbf33367dd496086f WatchSource:0}: Error finding container 7e82a91d68d9a83bf4b8d23c138223a14100468693b9393dbf33367dd496086f: Status 404 returned error can't find the container with id 7e82a91d68d9a83bf4b8d23c138223a14100468693b9393dbf33367dd496086f
Oct 30 01:00:00.829077 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.829053    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns"]
Oct 30 01:00:00.831404 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:00:00.831376    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d085f5_018a_4f9b_a6e4_7fb0b1d40e3a.slice/crio-312725ddcb501873fd41aec745682fb1ed9bfc35d6e5d15335072938528e98ad WatchSource:0}: Error finding container 312725ddcb501873fd41aec745682fb1ed9bfc35d6e5d15335072938528e98ad: Status 404 returned error can't find the container with id 312725ddcb501873fd41aec745682fb1ed9bfc35d6e5d15335072938528e98ad
Oct 30 01:00:00.836250 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.836224    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt"]
Oct 30 01:00:00.844266 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:00:00.844242    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078d4d89_6bcf_41a5_9755_c8929d96a4c0.slice/crio-0056523f3a82358323a80ce037331ef63eb5ab62f242fa774a78456e0a6e6c1b WatchSource:0}: Error finding container 0056523f3a82358323a80ce037331ef63eb5ab62f242fa774a78456e0a6e6c1b: Status 404 returned error can't find the container with id 0056523f3a82358323a80ce037331ef63eb5ab62f242fa774a78456e0a6e6c1b
Oct 30 01:00:00.982714 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.982689    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt" event={"ID":"078d4d89-6bcf-41a5-9755-c8929d96a4c0","Type":"ContainerStarted","Data":"0056523f3a82358323a80ce037331ef63eb5ab62f242fa774a78456e0a6e6c1b"}
Oct 30 01:00:00.984052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.984028    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75" event={"ID":"110d4727-ac79-4852-8437-f971afc20244","Type":"ContainerStarted","Data":"7e82a91d68d9a83bf4b8d23c138223a14100468693b9393dbf33367dd496086f"}
Oct 30 01:00:00.985388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:00.985366    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns" event={"ID":"75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a","Type":"ContainerStarted","Data":"312725ddcb501873fd41aec745682fb1ed9bfc35d6e5d15335072938528e98ad"}
Oct 30 01:00:01.995408 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:01.995334    4425 generic.go:356] "Generic (PLEG): container finished" podID="75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a" containerID="8cf684535e4aefc61d8b03649a6a13a6f68339788f37c5ab44250f3b442ef51e" exitCode=60
Oct 30 01:00:01.995774 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:01.995415    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns" event={"ID":"75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a","Type":"ContainerDied","Data":"8cf684535e4aefc61d8b03649a6a13a6f68339788f37c5ab44250f3b442ef51e"}
Oct 30 01:00:01.995774 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:01.995629    4425 scope.go:122] "RemoveContainer" containerID="8cf684535e4aefc61d8b03649a6a13a6f68339788f37c5ab44250f3b442ef51e"
Oct 30 01:00:01.997097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:01.997057    4425 generic.go:356] "Generic (PLEG): container finished" podID="078d4d89-6bcf-41a5-9755-c8929d96a4c0" containerID="eccc3bcea268dbf211a8b75806a357e0e8deb7bcf59467f950260a129eb5330d" exitCode=60
Oct 30 01:00:01.997201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:01.997102    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt" event={"ID":"078d4d89-6bcf-41a5-9755-c8929d96a4c0","Type":"ContainerDied","Data":"eccc3bcea268dbf211a8b75806a357e0e8deb7bcf59467f950260a129eb5330d"}
Oct 30 01:00:01.997387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:01.997368    4425 scope.go:122] "RemoveContainer" containerID="eccc3bcea268dbf211a8b75806a357e0e8deb7bcf59467f950260a129eb5330d"
Oct 30 01:00:01.999006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:01.998986    4425 generic.go:356] "Generic (PLEG): container finished" podID="110d4727-ac79-4852-8437-f971afc20244" containerID="d98ab402b4f3007c63b2bab30da9464b01b780a0ea31341e986b5beb4f0ab49e" exitCode=60
Oct 30 01:00:01.999101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:01.999057    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75" event={"ID":"110d4727-ac79-4852-8437-f971afc20244","Type":"ContainerDied","Data":"d98ab402b4f3007c63b2bab30da9464b01b780a0ea31341e986b5beb4f0ab49e"}
Oct 30 01:00:01.999371 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:01.999353    4425 scope.go:122] "RemoveContainer" containerID="d98ab402b4f3007c63b2bab30da9464b01b780a0ea31341e986b5beb4f0ab49e"
Oct 30 01:00:03.008629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:03.008595    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt" event={"ID":"078d4d89-6bcf-41a5-9755-c8929d96a4c0","Type":"ContainerStarted","Data":"1e19cad8f1421cb890361a6c113f4c671a5a793c2d02584b96c53cfaa985b0d4"}
Oct 30 01:00:03.010254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:03.010233    4425 generic.go:356] "Generic (PLEG): container finished" podID="110d4727-ac79-4852-8437-f971afc20244" containerID="9e79714ff7abccf309ce13ec0b10dae4981cf51f6e8f8f6e4d0f2577ce0b9ad6" exitCode=60
Oct 30 01:00:03.010364 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:03.010301    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75" event={"ID":"110d4727-ac79-4852-8437-f971afc20244","Type":"ContainerDied","Data":"9e79714ff7abccf309ce13ec0b10dae4981cf51f6e8f8f6e4d0f2577ce0b9ad6"}
Oct 30 01:00:03.010364 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:03.010349    4425 scope.go:122] "RemoveContainer" containerID="d98ab402b4f3007c63b2bab30da9464b01b780a0ea31341e986b5beb4f0ab49e"
Oct 30 01:00:03.010548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:03.010526    4425 scope.go:122] "RemoveContainer" containerID="9e79714ff7abccf309ce13ec0b10dae4981cf51f6e8f8f6e4d0f2577ce0b9ad6"
Oct 30 01:00:03.010794 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:00:03.010775    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75_odh-ai-gateway-infra(110d4727-ac79-4852-8437-f971afc20244)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75" podUID="110d4727-ac79-4852-8437-f971afc20244"
Oct 30 01:00:03.012284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:03.012265    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns" event={"ID":"75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a","Type":"ContainerStarted","Data":"1d299966db5d0dcb37ab2af293559eb4bdf67288ad387f26684d06550d9b4fc1"}
Oct 30 01:00:03.049122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:03.049054    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt" podStartSLOduration=3.049034609 podStartE2EDuration="3.049034609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:03.046960414 +0000 UTC m=+1111.796615723" watchObservedRunningTime="2026-10-30 01:00:03.049034609 +0000 UTC m=+1111.798689918"
Oct 30 01:00:03.127333 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:03.127262    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns" podStartSLOduration=3.127242374 podStartE2EDuration="3.127242374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:03.122495067 +0000 UTC m=+1111.872150376" watchObservedRunningTime="2026-10-30 01:00:03.127242374 +0000 UTC m=+1111.876897686"
Oct 30 01:00:04.020801 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:04.020761    4425 generic.go:356] "Generic (PLEG): container finished" podID="078d4d89-6bcf-41a5-9755-c8929d96a4c0" containerID="1e19cad8f1421cb890361a6c113f4c671a5a793c2d02584b96c53cfaa985b0d4" exitCode=60
Oct 30 01:00:04.021263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:04.020836    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt" event={"ID":"078d4d89-6bcf-41a5-9755-c8929d96a4c0","Type":"ContainerDied","Data":"1e19cad8f1421cb890361a6c113f4c671a5a793c2d02584b96c53cfaa985b0d4"}
Oct 30 01:00:04.021263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:04.020893    4425 scope.go:122] "RemoveContainer" containerID="eccc3bcea268dbf211a8b75806a357e0e8deb7bcf59467f950260a129eb5330d"
Oct 30 01:00:04.021263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:04.021078    4425 scope.go:122] "RemoveContainer" containerID="1e19cad8f1421cb890361a6c113f4c671a5a793c2d02584b96c53cfaa985b0d4"
Oct 30 01:00:04.021418 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:00:04.021398    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt_odh-ai-gateway-infra(078d4d89-6bcf-41a5-9755-c8929d96a4c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt" podUID="078d4d89-6bcf-41a5-9755-c8929d96a4c0"
Oct 30 01:00:04.022916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:04.022893    4425 scope.go:122] "RemoveContainer" containerID="9e79714ff7abccf309ce13ec0b10dae4981cf51f6e8f8f6e4d0f2577ce0b9ad6"
Oct 30 01:00:04.023191 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:00:04.023170    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75_odh-ai-gateway-infra(110d4727-ac79-4852-8437-f971afc20244)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75" podUID="110d4727-ac79-4852-8437-f971afc20244"
Oct 30 01:00:04.024540 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:04.024516    4425 generic.go:356] "Generic (PLEG): container finished" podID="75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a" containerID="1d299966db5d0dcb37ab2af293559eb4bdf67288ad387f26684d06550d9b4fc1" exitCode=60
Oct 30 01:00:04.024647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:04.024564    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns" event={"ID":"75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a","Type":"ContainerDied","Data":"1d299966db5d0dcb37ab2af293559eb4bdf67288ad387f26684d06550d9b4fc1"}
Oct 30 01:00:04.024784 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:04.024771    4425 scope.go:122] "RemoveContainer" containerID="1d299966db5d0dcb37ab2af293559eb4bdf67288ad387f26684d06550d9b4fc1"
Oct 30 01:00:04.024960 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:00:04.024943    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns_odh-ai-gateway-infra(75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns" podUID="75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a"
Oct 30 01:00:04.032974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:04.032962    4425 scope.go:122] "RemoveContainer" containerID="8cf684535e4aefc61d8b03649a6a13a6f68339788f37c5ab44250f3b442ef51e"
Oct 30 01:00:04.556610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:04.556570    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:00:05.033034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:05.033006    4425 scope.go:122] "RemoveContainer" containerID="1d299966db5d0dcb37ab2af293559eb4bdf67288ad387f26684d06550d9b4fc1"
Oct 30 01:00:05.033493 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:00:05.033338    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns_odh-ai-gateway-infra(75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns" podUID="75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a"
Oct 30 01:00:05.034974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:05.034954    4425 scope.go:122] "RemoveContainer" containerID="1e19cad8f1421cb890361a6c113f4c671a5a793c2d02584b96c53cfaa985b0d4"
Oct 30 01:00:05.035202 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:00:05.035181    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt_odh-ai-gateway-infra(078d4d89-6bcf-41a5-9755-c8929d96a4c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt" podUID="078d4d89-6bcf-41a5-9755-c8929d96a4c0"
Oct 30 01:00:06.080074 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.080032    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"]
Oct 30 01:00:06.104680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.104648    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"]
Oct 30 01:00:06.105959 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.105032    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.131198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.130110    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8363a40b\"/\"openshift-service-ca.crt\""
Oct 30 01:00:06.135382 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.135360    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8363a40b\"/\"kube-root-ca.crt\""
Oct 30 01:00:06.140552 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.140368    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8363a40b\"/\"default-dockercfg-hbrhl\""
Oct 30 01:00:06.140552 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.140462    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8363a40b\"/\"e2e-lifecycle-model-8363a40b-kserve-self-signed-certs\""
Oct 30 01:00:06.147034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.147014    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-tmp-dir\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.147139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.147119    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-dshm\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.147343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.147324    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-home\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.147420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.147356    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-model-cache\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.147474 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.147440    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hk85\" (UniqueName: \"kubernetes.io/projected/0f9fbe79-74be-44e1-977c-963f67b0f054-kube-api-access-6hk85\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.147535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.147519    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f9fbe79-74be-44e1-977c-963f67b0f054-tls-certs\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.255175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.252120    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-dshm\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.255175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.252214    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-home\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.255175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.252246    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-model-cache\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.255175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.252331    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hk85\" (UniqueName: \"kubernetes.io/projected/0f9fbe79-74be-44e1-977c-963f67b0f054-kube-api-access-6hk85\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.255175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.252404    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f9fbe79-74be-44e1-977c-963f67b0f054-tls-certs\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.255175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.252498    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-tmp-dir\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.255175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.252924    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-tmp-dir\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.255175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.253466    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-model-cache\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.255175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.253678    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-home\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.276225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.259533    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-dshm\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.276225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.260891    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f9fbe79-74be-44e1-977c-963f67b0f054-tls-certs\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.278993 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.278964    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hk85\" (UniqueName: \"kubernetes.io/projected/0f9fbe79-74be-44e1-977c-963f67b0f054-kube-api-access-6hk85\") pod \"e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") " pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:06.456229 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:06.455373    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:07.193180 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:00:07.192670    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f9fbe79_74be_44e1_977c_963f67b0f054.slice/crio-5d0039b8b4c5567c1a891ff1c01e9c51cdf1a75c33663cf03fbd77edb64138f4 WatchSource:0}: Error finding container 5d0039b8b4c5567c1a891ff1c01e9c51cdf1a75c33663cf03fbd77edb64138f4: Status 404 returned error can't find the container with id 5d0039b8b4c5567c1a891ff1c01e9c51cdf1a75c33663cf03fbd77edb64138f4
Oct 30 01:00:07.413459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:07.407103    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"]
Oct 30 01:00:07.789199 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:00:07.787383    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:00:08.264177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:08.261258    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2" podUID="13f688db-25e0-4a88-9d1e-c3fd78b7f659" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:08.408445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:08.407474    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h" event={"ID":"0f9fbe79-74be-44e1-977c-963f67b0f054","Type":"ContainerStarted","Data":"5d0039b8b4c5567c1a891ff1c01e9c51cdf1a75c33663cf03fbd77edb64138f4"}
Oct 30 01:00:08.728229 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:08.713258    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:08.849181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:08.839247    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7" podUID="834166ed-5c63-4854-9e89-c45041553630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:09.054259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:09.054218    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f" podUID="0824ed99-cf93-4047-b19f-0ad807263acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:09.227630 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:09.226693    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" podUID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:09.595912 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:09.595873    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h" event={"ID":"0f9fbe79-74be-44e1-977c-963f67b0f054","Type":"ContainerStarted","Data":"281cfab270ca913f082dad01e974c36e715f36e28265979a8ac43af1f8979afb"}
Oct 30 01:00:10.457830 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:10.457792    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:15.415324 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:15.415247    4425 scope.go:122] "RemoveContainer" containerID="9e79714ff7abccf309ce13ec0b10dae4981cf51f6e8f8f6e4d0f2577ce0b9ad6"
Oct 30 01:00:15.415770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:15.415349    4425 scope.go:122] "RemoveContainer" containerID="1d299966db5d0dcb37ab2af293559eb4bdf67288ad387f26684d06550d9b4fc1"
Oct 30 01:00:16.457892 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:16.457865    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:16.470683 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:16.470659    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:16.511410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:16.511364    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h" podStartSLOduration=10.511352513 podStartE2EDuration="10.511352513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:09.565585846 +0000 UTC m=+1118.315241157" watchObservedRunningTime="2026-10-30 01:00:16.511352513 +0000 UTC m=+1125.261007822"
Oct 30 01:00:16.531433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:16.531363    4425 generic.go:356] "Generic (PLEG): container finished" podID="110d4727-ac79-4852-8437-f971afc20244" containerID="f2fb4b4a8eac8fafc05cca4f1f7be105c3e216a1ccc78fa875aa94be6840acf3" exitCode=60
Oct 30 01:00:16.531560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:16.531438    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75" event={"ID":"110d4727-ac79-4852-8437-f971afc20244","Type":"ContainerDied","Data":"f2fb4b4a8eac8fafc05cca4f1f7be105c3e216a1ccc78fa875aa94be6840acf3"}
Oct 30 01:00:16.531560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:16.531490    4425 scope.go:122] "RemoveContainer" containerID="9e79714ff7abccf309ce13ec0b10dae4981cf51f6e8f8f6e4d0f2577ce0b9ad6"
Oct 30 01:00:16.531839 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:16.531820    4425 scope.go:122] "RemoveContainer" containerID="f2fb4b4a8eac8fafc05cca4f1f7be105c3e216a1ccc78fa875aa94be6840acf3"
Oct 30 01:00:16.532137 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:00:16.532113    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75_odh-ai-gateway-infra(110d4727-ac79-4852-8437-f971afc20244)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75" podUID="110d4727-ac79-4852-8437-f971afc20244"
Oct 30 01:00:16.533583 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:16.533562    4425 generic.go:356] "Generic (PLEG): container finished" podID="75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a" containerID="99855aae458e735ae91a154f89faee90224e4235c3c1623d3b2b07e4602fb6e0" exitCode=60
Oct 30 01:00:16.533676 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:16.533638    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns" event={"ID":"75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a","Type":"ContainerDied","Data":"99855aae458e735ae91a154f89faee90224e4235c3c1623d3b2b07e4602fb6e0"}
Oct 30 01:00:16.534098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:16.534079    4425 scope.go:122] "RemoveContainer" containerID="99855aae458e735ae91a154f89faee90224e4235c3c1623d3b2b07e4602fb6e0"
Oct 30 01:00:16.534373 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:00:16.534352    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns_odh-ai-gateway-infra(75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns" podUID="75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a"
Oct 30 01:00:16.543927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:16.543906    4425 scope.go:122] "RemoveContainer" containerID="1d299966db5d0dcb37ab2af293559eb4bdf67288ad387f26684d06550d9b4fc1"
Oct 30 01:00:16.545362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:16.545333    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:00:17.415295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:17.415267    4425 scope.go:122] "RemoveContainer" containerID="1e19cad8f1421cb890361a6c113f4c671a5a793c2d02584b96c53cfaa985b0d4"
Oct 30 01:00:17.578123 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:17.578088    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75"]
Oct 30 01:00:17.652934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:17.652901    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns"]
Oct 30 01:00:17.745434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:17.745410    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75"
Oct 30 01:00:17.841668 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:17.841642    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns"
Oct 30 01:00:17.863416 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:17.863384    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/110d4727-ac79-4852-8437-f971afc20244-kube-api-access-5llnd\" (UniqueName: \"kubernetes.io/projected/110d4727-ac79-4852-8437-f971afc20244-kube-api-access-5llnd\") pod \"110d4727-ac79-4852-8437-f971afc20244\" (UID: \"110d4727-ac79-4852-8437-f971afc20244\") "
Oct 30 01:00:17.863563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:17.863535    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/110d4727-ac79-4852-8437-f971afc20244-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110d4727-ac79-4852-8437-f971afc20244-maas-api-service-ca\") pod \"110d4727-ac79-4852-8437-f971afc20244\" (UID: \"110d4727-ac79-4852-8437-f971afc20244\") "
Oct 30 01:00:17.863859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:17.863837    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110d4727-ac79-4852-8437-f971afc20244-maas-api-service-ca" pod "110d4727-ac79-4852-8437-f971afc20244" (UID: "110d4727-ac79-4852-8437-f971afc20244"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:17.864039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:17.864027    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110d4727-ac79-4852-8437-f971afc20244-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:00:17.865426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:17.865406    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110d4727-ac79-4852-8437-f971afc20244-kube-api-access-5llnd" pod "110d4727-ac79-4852-8437-f971afc20244" (UID: "110d4727-ac79-4852-8437-f971afc20244"). InnerVolumeSpecName "kube-api-access-5llnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:17.964505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:17.964425    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a-kube-api-access-8b642\" (UniqueName: \"kubernetes.io/projected/75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a-kube-api-access-8b642\") pod \"75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a\" (UID: \"75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a\") "
Oct 30 01:00:17.964669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:17.964611    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a-maas-api-service-ca\") pod \"75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a\" (UID: \"75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a\") "
Oct 30 01:00:17.964935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:17.964912    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a-maas-api-service-ca" pod "75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a" (UID: "75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:17.965150 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:17.965136    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5llnd\" (UniqueName: \"kubernetes.io/projected/110d4727-ac79-4852-8437-f971afc20244-kube-api-access-5llnd\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:00:17.965212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:17.965167    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:00:17.966447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:17.966426    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a-kube-api-access-8b642" pod "75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a" (UID: "75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a"). InnerVolumeSpecName "kube-api-access-8b642". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:18.066709 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:18.066677    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b642\" (UniqueName: \"kubernetes.io/projected/75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a-kube-api-access-8b642\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:00:18.565599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:18.556463    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75"
Oct 30 01:00:18.565599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:18.556463    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75" event={"ID":"110d4727-ac79-4852-8437-f971afc20244","Type":"ContainerDied","Data":"7e82a91d68d9a83bf4b8d23c138223a14100468693b9393dbf33367dd496086f"}
Oct 30 01:00:18.565599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:18.556719    4425 scope.go:122] "RemoveContainer" containerID="f2fb4b4a8eac8fafc05cca4f1f7be105c3e216a1ccc78fa875aa94be6840acf3"
Oct 30 01:00:18.565599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:18.559969    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns" event={"ID":"75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a","Type":"ContainerDied","Data":"312725ddcb501873fd41aec745682fb1ed9bfc35d6e5d15335072938528e98ad"}
Oct 30 01:00:18.565599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:18.560117    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns"
Oct 30 01:00:18.583887 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:18.583862    4425 generic.go:356] "Generic (PLEG): container finished" podID="078d4d89-6bcf-41a5-9755-c8929d96a4c0" containerID="1a834f582a31e0f315c82ae7f483deca1e3400e73c53de47623cac88ae04c7ff" exitCode=60
Oct 30 01:00:18.584333 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:18.583913    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt" event={"ID":"078d4d89-6bcf-41a5-9755-c8929d96a4c0","Type":"ContainerDied","Data":"1a834f582a31e0f315c82ae7f483deca1e3400e73c53de47623cac88ae04c7ff"}
Oct 30 01:00:18.584439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:18.584352    4425 scope.go:122] "RemoveContainer" containerID="1a834f582a31e0f315c82ae7f483deca1e3400e73c53de47623cac88ae04c7ff"
Oct 30 01:00:18.584733 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:00:18.584711    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt_odh-ai-gateway-infra(078d4d89-6bcf-41a5-9755-c8929d96a4c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt" podUID="078d4d89-6bcf-41a5-9755-c8929d96a4c0"
Oct 30 01:00:18.600538 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:18.600058    4425 scope.go:122] "RemoveContainer" containerID="99855aae458e735ae91a154f89faee90224e4235c3c1623d3b2b07e4602fb6e0"
Oct 30 01:00:18.612233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:18.612128    4425 scope.go:122] "RemoveContainer" containerID="1e19cad8f1421cb890361a6c113f4c671a5a793c2d02584b96c53cfaa985b0d4"
Oct 30 01:00:18.662738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:18.662708    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns"]
Oct 30 01:00:18.670485 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:18.670459    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888700-755ns"]
Oct 30 01:00:18.699739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:18.699710    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75"]
Oct 30 01:00:18.703768 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:18.703746    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8363a40b-29888700-hsk75"]
Oct 30 01:00:19.429390 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:19.429358    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="110d4727-ac79-4852-8437-f971afc20244" path="/var/lib/kubelet/pods/110d4727-ac79-4852-8437-f971afc20244/volumes"
Oct 30 01:00:19.429779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:19.429765    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a" path="/var/lib/kubelet/pods/75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a/volumes"
Oct 30 01:00:19.639204 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:19.639173    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt"]
Oct 30 01:00:19.785525 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:19.785504    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt"
Oct 30 01:00:19.893371 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:19.893344    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/078d4d89-6bcf-41a5-9755-c8929d96a4c0-kube-api-access-rttxx\" (UniqueName: \"kubernetes.io/projected/078d4d89-6bcf-41a5-9755-c8929d96a4c0-kube-api-access-rttxx\") pod \"078d4d89-6bcf-41a5-9755-c8929d96a4c0\" (UID: \"078d4d89-6bcf-41a5-9755-c8929d96a4c0\") "
Oct 30 01:00:19.893371 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:19.893376    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/078d4d89-6bcf-41a5-9755-c8929d96a4c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/078d4d89-6bcf-41a5-9755-c8929d96a4c0-maas-api-service-ca\") pod \"078d4d89-6bcf-41a5-9755-c8929d96a4c0\" (UID: \"078d4d89-6bcf-41a5-9755-c8929d96a4c0\") "
Oct 30 01:00:19.893736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:19.893709    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078d4d89-6bcf-41a5-9755-c8929d96a4c0-maas-api-service-ca" pod "078d4d89-6bcf-41a5-9755-c8929d96a4c0" (UID: "078d4d89-6bcf-41a5-9755-c8929d96a4c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:19.893867 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:19.893846    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/078d4d89-6bcf-41a5-9755-c8929d96a4c0-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:00:19.895331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:19.895309    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078d4d89-6bcf-41a5-9755-c8929d96a4c0-kube-api-access-rttxx" pod "078d4d89-6bcf-41a5-9755-c8929d96a4c0" (UID: "078d4d89-6bcf-41a5-9755-c8929d96a4c0"). InnerVolumeSpecName "kube-api-access-rttxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:19.995303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:19.995245    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rttxx\" (UniqueName: \"kubernetes.io/projected/078d4d89-6bcf-41a5-9755-c8929d96a4c0-kube-api-access-rttxx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:00:20.600513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:20.600483    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt"
Oct 30 01:00:20.600513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:20.600505    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt" event={"ID":"078d4d89-6bcf-41a5-9755-c8929d96a4c0","Type":"ContainerDied","Data":"0056523f3a82358323a80ce037331ef63eb5ab62f242fa774a78456e0a6e6c1b"}
Oct 30 01:00:20.600895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:20.600572    4425 scope.go:122] "RemoveContainer" containerID="1a834f582a31e0f315c82ae7f483deca1e3400e73c53de47623cac88ae04c7ff"
Oct 30 01:00:20.645535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:20.645509    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt"]
Oct 30 01:00:20.655061 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:20.655037    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888700-cxfwt"]
Oct 30 01:00:21.423067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:21.423036    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="078d4d89-6bcf-41a5-9755-c8929d96a4c0" path="/var/lib/kubelet/pods/078d4d89-6bcf-41a5-9755-c8929d96a4c0/volumes"
Oct 30 01:00:26.231479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.231445    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"]
Oct 30 01:00:26.232166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232127    4425 cpu_manager.go:414] "containerMap: removing container" podUID="110d4727-ac79-4852-8437-f971afc20244" containerName="cleanup"
Oct 30 01:00:26.232278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232167    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110d4727-ac79-4852-8437-f971afc20244" containerName="cleanup"
Oct 30 01:00:26.232278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232180    4425 cpu_manager.go:414] "containerMap: removing container" podUID="75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a" containerName="cleanup"
Oct 30 01:00:26.232278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232187    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a" containerName="cleanup"
Oct 30 01:00:26.232278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232198    4425 cpu_manager.go:414] "containerMap: removing container" podUID="078d4d89-6bcf-41a5-9755-c8929d96a4c0" containerName="cleanup"
Oct 30 01:00:26.232278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232204    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078d4d89-6bcf-41a5-9755-c8929d96a4c0" containerName="cleanup"
Oct 30 01:00:26.232278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232229    4425 cpu_manager.go:414] "containerMap: removing container" podUID="75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a" containerName="cleanup"
Oct 30 01:00:26.232278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232236    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a" containerName="cleanup"
Oct 30 01:00:26.232278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232253    4425 cpu_manager.go:414] "containerMap: removing container" podUID="110d4727-ac79-4852-8437-f971afc20244" containerName="cleanup"
Oct 30 01:00:26.232278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232259    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110d4727-ac79-4852-8437-f971afc20244" containerName="cleanup"
Oct 30 01:00:26.232688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232335    4425 cpu_manager.go:414] "containerMap: removing container" podUID="078d4d89-6bcf-41a5-9755-c8929d96a4c0" containerName="cleanup"
Oct 30 01:00:26.232688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232344    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078d4d89-6bcf-41a5-9755-c8929d96a4c0" containerName="cleanup"
Oct 30 01:00:26.232688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232476    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a" containerName="cleanup"
Oct 30 01:00:26.232688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232488    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="110d4727-ac79-4852-8437-f971afc20244" containerName="cleanup"
Oct 30 01:00:26.232688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232500    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="078d4d89-6bcf-41a5-9755-c8929d96a4c0" containerName="cleanup"
Oct 30 01:00:26.232688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232515    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a" containerName="cleanup"
Oct 30 01:00:26.232688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232526    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="110d4727-ac79-4852-8437-f971afc20244" containerName="cleanup"
Oct 30 01:00:26.232688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232536    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="078d4d89-6bcf-41a5-9755-c8929d96a4c0" containerName="cleanup"
Oct 30 01:00:26.232688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.232543    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="110d4727-ac79-4852-8437-f971afc20244" containerName="cleanup"
Oct 30 01:00:26.237714 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.237692    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.272396 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.272371    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"]
Oct 30 01:00:26.394655 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.394619    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.394655 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.394655    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.394903 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.394731    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.394903 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.394785    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.394903 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.394862    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.395051 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.394932    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.395051 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.395015    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.395125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.395066    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.395125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.395118    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv6rn\" (UniqueName: \"kubernetes.io/projected/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-kube-api-access-kv6rn\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.495834 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.495766    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.495834 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.495801    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.495834 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.495834    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv6rn\" (UniqueName: \"kubernetes.io/projected/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-kube-api-access-kv6rn\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.496082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.495891    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.496082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.495907    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.496082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.495961    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.496082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.496036    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.496082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.496069    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.496387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.496113    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.496387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.496333    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.496625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.496599    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.496727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.496605    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.496727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.496679    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.496727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.496711    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.498776 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.498753    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.498936 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.498917    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.516460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.516429    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.516722 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.516703    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv6rn\" (UniqueName: \"kubernetes.io/projected/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-kube-api-access-kv6rn\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.556827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.556804    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:26.870059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.870031    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"]
Oct 30 01:00:26.871058 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:00:26.871032    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30db65f6_70ad_4a3b_bed9_fb4f74533b3a.slice/crio-2a40707767bbaa26283d8c0f6868289a27dc8daad0470316439960fc5befa78d WatchSource:0}: Error finding container 2a40707767bbaa26283d8c0f6868289a27dc8daad0470316439960fc5befa78d: Status 404 returned error can't find the container with id 2a40707767bbaa26283d8c0f6868289a27dc8daad0470316439960fc5befa78d
Oct 30 01:00:26.873522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.873485    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:26.873616 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.873565    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:26.873616 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:26.873610    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:27.649206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:27.649131    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg" event={"ID":"30db65f6-70ad-4a3b-bed9-fb4f74533b3a","Type":"ContainerStarted","Data":"f746ab01ba64474fcb169e668a363861456a436abd2c5118ea64be738cfa167e"}
Oct 30 01:00:27.649728 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:27.649234    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg" event={"ID":"30db65f6-70ad-4a3b-bed9-fb4f74533b3a","Type":"ContainerStarted","Data":"2a40707767bbaa26283d8c0f6868289a27dc8daad0470316439960fc5befa78d"}
Oct 30 01:00:27.710812 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:27.710747    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg" podStartSLOduration=1.710727652 podStartE2EDuration="1.710727652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:27.708986915 +0000 UTC m=+1136.458642224" watchObservedRunningTime="2026-10-30 01:00:27.710727652 +0000 UTC m=+1136.460382962"
Oct 30 01:00:28.571179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:28.566780    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:28.571179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:28.566898    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 01:00:28.571179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:28.566952    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg" podUID="30db65f6-70ad-4a3b-bed9-fb4f74533b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 01:00:29.108527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.108485    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"]
Oct 30 01:00:29.109268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.109246    4425 cpu_manager.go:414] "containerMap: removing container" podUID="110d4727-ac79-4852-8437-f971afc20244" containerName="cleanup"
Oct 30 01:00:29.109268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.109268    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110d4727-ac79-4852-8437-f971afc20244" containerName="cleanup"
Oct 30 01:00:29.109447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.109282    4425 cpu_manager.go:414] "containerMap: removing container" podUID="078d4d89-6bcf-41a5-9755-c8929d96a4c0" containerName="cleanup"
Oct 30 01:00:29.109447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.109290    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078d4d89-6bcf-41a5-9755-c8929d96a4c0" containerName="cleanup"
Oct 30 01:00:29.109447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.109313    4425 cpu_manager.go:414] "containerMap: removing container" podUID="75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a" containerName="cleanup"
Oct 30 01:00:29.109447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.109321    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a" containerName="cleanup"
Oct 30 01:00:29.109635 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.109490    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d085f5-018a-4f9b-a6e4-7fb0b1d40e3a" containerName="cleanup"
Oct 30 01:00:29.109635 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.109504    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="078d4d89-6bcf-41a5-9755-c8929d96a4c0" containerName="cleanup"
Oct 30 01:00:29.113067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.113044    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.137423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.137374    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d084a21c-a3d0-4f2a-99ac-412c560f0e03-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.137589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.137567    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d084a21c-a3d0-4f2a-99ac-412c560f0e03-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.137744 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.137726    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7p76\" (UniqueName: \"kubernetes.io/projected/d084a21c-a3d0-4f2a-99ac-412c560f0e03-kube-api-access-j7p76\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.137883 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.137868    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d084a21c-a3d0-4f2a-99ac-412c560f0e03-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.137995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.137982    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d084a21c-a3d0-4f2a-99ac-412c560f0e03-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.138105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.138091    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d084a21c-a3d0-4f2a-99ac-412c560f0e03-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.138248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.138231    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d084a21c-a3d0-4f2a-99ac-412c560f0e03-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.138400 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.138384    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d084a21c-a3d0-4f2a-99ac-412c560f0e03-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.138584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.138570    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d084a21c-a3d0-4f2a-99ac-412c560f0e03-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.186007 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.185896    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"]
Oct 30 01:00:29.239562 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.239532    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7p76\" (UniqueName: \"kubernetes.io/projected/d084a21c-a3d0-4f2a-99ac-412c560f0e03-kube-api-access-j7p76\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.239740 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.239574    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d084a21c-a3d0-4f2a-99ac-412c560f0e03-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.239740 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.239592    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d084a21c-a3d0-4f2a-99ac-412c560f0e03-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.239740 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.239609    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d084a21c-a3d0-4f2a-99ac-412c560f0e03-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.239740 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.239636    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d084a21c-a3d0-4f2a-99ac-412c560f0e03-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.239740 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.239675    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d084a21c-a3d0-4f2a-99ac-412c560f0e03-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.239740 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.239724    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d084a21c-a3d0-4f2a-99ac-412c560f0e03-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.240039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.239769    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d084a21c-a3d0-4f2a-99ac-412c560f0e03-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.240039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.239785    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d084a21c-a3d0-4f2a-99ac-412c560f0e03-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.240140 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.240100    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d084a21c-a3d0-4f2a-99ac-412c560f0e03-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.241553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.241220    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d084a21c-a3d0-4f2a-99ac-412c560f0e03-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.241553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.241300    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d084a21c-a3d0-4f2a-99ac-412c560f0e03-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.241553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.241482    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d084a21c-a3d0-4f2a-99ac-412c560f0e03-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.241791 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.241711    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d084a21c-a3d0-4f2a-99ac-412c560f0e03-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.242592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.242569    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d084a21c-a3d0-4f2a-99ac-412c560f0e03-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.243837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.243816    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d084a21c-a3d0-4f2a-99ac-412c560f0e03-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.280197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.272692    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7p76\" (UniqueName: \"kubernetes.io/projected/d084a21c-a3d0-4f2a-99ac-412c560f0e03-kube-api-access-j7p76\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.283662 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.283604    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d084a21c-a3d0-4f2a-99ac-412c560f0e03-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-m5c9r\" (UID: \"d084a21c-a3d0-4f2a-99ac-412c560f0e03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.461665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.461354    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:29.564517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.562490    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 01:00:29.564517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.562553    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg" podUID="30db65f6-70ad-4a3b-bed9-fb4f74533b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 01:00:29.841454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.841416    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"]
Oct 30 01:00:29.845500 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:00:29.845475    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd084a21c_a3d0_4f2a_99ac_412c560f0e03.slice/crio-846dff5eb0011e1c7d9f425e5ad8892016d8d5f2b036310a49615a590c456464 WatchSource:0}: Error finding container 846dff5eb0011e1c7d9f425e5ad8892016d8d5f2b036310a49615a590c456464: Status 404 returned error can't find the container with id 846dff5eb0011e1c7d9f425e5ad8892016d8d5f2b036310a49615a590c456464
Oct 30 01:00:29.848141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.848107    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:29.848248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.848208    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:29.848290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:29.848252    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:30.560181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:30.557608    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 01:00:30.560181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:30.557690    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg" podUID="30db65f6-70ad-4a3b-bed9-fb4f74533b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 01:00:30.702054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:30.702014    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r" event={"ID":"d084a21c-a3d0-4f2a-99ac-412c560f0e03","Type":"ContainerStarted","Data":"51cef9cf7c588bc2bfb82142142036b03405733ec1ef76aec988f3072bf39f22"}
Oct 30 01:00:30.702253 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:30.702106    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r" event={"ID":"d084a21c-a3d0-4f2a-99ac-412c560f0e03","Type":"ContainerStarted","Data":"846dff5eb0011e1c7d9f425e5ad8892016d8d5f2b036310a49615a590c456464"}
Oct 30 01:00:31.480180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:31.468514    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:31.489327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:31.483395    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m5c9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 01:00:31.489327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:31.483451    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r" podUID="d084a21c-a3d0-4f2a-99ac-412c560f0e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 01:00:31.574726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:31.572257    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 01:00:31.574726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:31.572317    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg" podUID="30db65f6-70ad-4a3b-bed9-fb4f74533b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 01:00:32.463531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:32.463494    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m5c9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 01:00:32.463813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:32.463784    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r" podUID="d084a21c-a3d0-4f2a-99ac-412c560f0e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 01:00:32.567181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:32.557475    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 01:00:32.567181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:32.557530    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg" podUID="30db65f6-70ad-4a3b-bed9-fb4f74533b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 01:00:33.464684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:33.464645    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m5c9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 01:00:33.465337 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:33.465305    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r" podUID="d084a21c-a3d0-4f2a-99ac-412c560f0e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 01:00:33.561574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:33.561539    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 01:00:33.561814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:33.561788    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg" podUID="30db65f6-70ad-4a3b-bed9-fb4f74533b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 01:00:34.472965 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:34.472928    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m5c9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 01:00:34.473603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:34.473570    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r" podUID="d084a21c-a3d0-4f2a-99ac-412c560f0e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 01:00:35.464187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:35.463530    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m5c9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 01:00:35.464187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:35.463592    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r" podUID="d084a21c-a3d0-4f2a-99ac-412c560f0e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 01:00:35.565330 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:35.565268    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:00:35.565970 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:35.565940    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg" podUID="30db65f6-70ad-4a3b-bed9-fb4f74533b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:35.857249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:35.840339    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7" podUID="834166ed-5c63-4854-9e89-c45041553630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:35.873176 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:35.867246    4425 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z" podUID="b4ed47ff-0328-4645-be5f-6a6d51a94fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:36.063343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:36.063292    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z" podUID="b4ed47ff-0328-4645-be5f-6a6d51a94fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 01:00:36.063931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:36.063898    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f" podUID="0824ed99-cf93-4047-b19f-0ad807263acf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 01:00:36.065628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:36.065601    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw" podUID="4d3d278e-ecd0-46ee-a203-33003899a369" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 01:00:36.250249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:36.233246    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc" podUID="c8d1f900-b8c4-43dc-814e-9491e560ea3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 30 01:00:36.250249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:36.235323    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" podUID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:36.250249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:36.235405    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s" podUID="b8b27e40-2516-4d18-8c84-5e0925b3f4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 01:00:36.250249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:36.239302    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2" podUID="13f688db-25e0-4a88-9d1e-c3fd78b7f659" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:36.463477 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:36.463428    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m5c9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 01:00:36.463732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:36.463699    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r" podUID="d084a21c-a3d0-4f2a-99ac-412c560f0e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 01:00:36.618198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:36.616910    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:00:36.618198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:36.617011    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg" podUID="30db65f6-70ad-4a3b-bed9-fb4f74533b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:37.152032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:37.151999    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:37.153002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:37.152975    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:37.153725 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:37.153704    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:00:37.209503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:37.209444    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r" podStartSLOduration=8.209424254 podStartE2EDuration="8.209424254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:30.774573711 +0000 UTC m=+1139.524229022" watchObservedRunningTime="2026-10-30 01:00:37.209424254 +0000 UTC m=+1145.959079564"
Oct 30 01:00:37.463450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:37.463364    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m5c9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 01:00:37.463589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:37.463447    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r" podUID="d084a21c-a3d0-4f2a-99ac-412c560f0e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 01:00:37.560021 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:00:37.559974    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:00:38.462624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:38.462589    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m5c9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 30 01:00:38.463026 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:38.462672    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r" podUID="d084a21c-a3d0-4f2a-99ac-412c560f0e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 30 01:00:40.504326 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:40.504276    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m5c9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:00:40.505351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:40.504359    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r" podUID="d084a21c-a3d0-4f2a-99ac-412c560f0e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:41.547282 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:41.547252    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-m5c9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:00:41.547682 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:41.547298    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r" podUID="d084a21c-a3d0-4f2a-99ac-412c560f0e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:00:41.671017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:41.670988    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:41.671560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:41.671530    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:41.672411 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:41.672393    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-m5c9r"
Oct 30 01:00:54.253511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.253481    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"]
Oct 30 01:00:54.257562 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.257543    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"
Oct 30 01:00:54.276319 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.276299    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"]
Oct 30 01:00:54.282192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.281593    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:00:54.282192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.281663    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:00:54.282192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.281718    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:00:54.282192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.281734    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:00:54.339126 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.339093    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff836440-d844-4126-a495-1f8cf256dccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d27e976-cvbz8\" (UID: \"ff836440-d844-4126-a495-1f8cf256dccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"
Oct 30 01:00:54.339313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.339145    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qzjd\" (UniqueName: \"kubernetes.io/projected/ff836440-d844-4126-a495-1f8cf256dccc-kube-api-access-7qzjd\") pod \"maas-api-revoke-sub-3d27e976-cvbz8\" (UID: \"ff836440-d844-4126-a495-1f8cf256dccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"
Oct 30 01:00:54.339368 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.339345    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff836440-d844-4126-a495-1f8cf256dccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d27e976-cvbz8\" (UID: \"ff836440-d844-4126-a495-1f8cf256dccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"
Oct 30 01:00:54.440645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.440608    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff836440-d844-4126-a495-1f8cf256dccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d27e976-cvbz8\" (UID: \"ff836440-d844-4126-a495-1f8cf256dccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"
Oct 30 01:00:54.440817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.440672    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzjd\" (UniqueName: \"kubernetes.io/projected/ff836440-d844-4126-a495-1f8cf256dccc-kube-api-access-7qzjd\") pod \"maas-api-revoke-sub-3d27e976-cvbz8\" (UID: \"ff836440-d844-4126-a495-1f8cf256dccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"
Oct 30 01:00:54.440817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.440766    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff836440-d844-4126-a495-1f8cf256dccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d27e976-cvbz8\" (UID: \"ff836440-d844-4126-a495-1f8cf256dccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"
Oct 30 01:00:54.441444 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.441420    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff836440-d844-4126-a495-1f8cf256dccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d27e976-cvbz8\" (UID: \"ff836440-d844-4126-a495-1f8cf256dccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"
Oct 30 01:00:54.443055 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.443029    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff836440-d844-4126-a495-1f8cf256dccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d27e976-cvbz8\" (UID: \"ff836440-d844-4126-a495-1f8cf256dccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"
Oct 30 01:00:54.461484 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.461461    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qzjd\" (UniqueName: \"kubernetes.io/projected/ff836440-d844-4126-a495-1f8cf256dccc-kube-api-access-7qzjd\") pod \"maas-api-revoke-sub-3d27e976-cvbz8\" (UID: \"ff836440-d844-4126-a495-1f8cf256dccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"
Oct 30 01:00:54.570522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.570498    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"
Oct 30 01:00:54.876304 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:54.876277    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"]
Oct 30 01:00:54.877736 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:00:54.877712    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff836440_d844_4126_a495_1f8cf256dccc.slice/crio-d6866bef7fc1d2d3437aa3de9231837566697084191941099234c6be40c4f083 WatchSource:0}: Error finding container d6866bef7fc1d2d3437aa3de9231837566697084191941099234c6be40c4f083: Status 404 returned error can't find the container with id d6866bef7fc1d2d3437aa3de9231837566697084191941099234c6be40c4f083
Oct 30 01:00:55.196284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:55.196255    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8" event={"ID":"ff836440-d844-4126-a495-1f8cf256dccc","Type":"ContainerStarted","Data":"fe010553a6472c59f38a9ac20d969ff5448ddb3c333c758d4b31d539729453d1"}
Oct 30 01:00:55.196394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:55.196313    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8" event={"ID":"ff836440-d844-4126-a495-1f8cf256dccc","Type":"ContainerStarted","Data":"d6866bef7fc1d2d3437aa3de9231837566697084191941099234c6be40c4f083"}
Oct 30 01:00:55.242032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:55.241991    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8" podStartSLOduration=1.241979567 podStartE2EDuration="1.241979567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:55.230988649 +0000 UTC m=+1163.980643956" watchObservedRunningTime="2026-10-30 01:00:55.241979567 +0000 UTC m=+1163.991634876"
Oct 30 01:00:56.203423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:56.203384    4425 generic.go:356] "Generic (PLEG): container finished" podID="ff836440-d844-4126-a495-1f8cf256dccc" containerID="fe010553a6472c59f38a9ac20d969ff5448ddb3c333c758d4b31d539729453d1" exitCode=0
Oct 30 01:00:56.203812 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:56.203470    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8" event={"ID":"ff836440-d844-4126-a495-1f8cf256dccc","Type":"ContainerDied","Data":"fe010553a6472c59f38a9ac20d969ff5448ddb3c333c758d4b31d539729453d1"}
Oct 30 01:00:57.379449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:57.379425    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"
Oct 30 01:00:57.477211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:57.477128    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff836440-d844-4126-a495-1f8cf256dccc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff836440-d844-4126-a495-1f8cf256dccc-maas-api-xks-ca\") pod \"ff836440-d844-4126-a495-1f8cf256dccc\" (UID: \"ff836440-d844-4126-a495-1f8cf256dccc\") "
Oct 30 01:00:57.477350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:57.477232    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff836440-d844-4126-a495-1f8cf256dccc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff836440-d844-4126-a495-1f8cf256dccc-maas-api-openshift-ca\") pod \"ff836440-d844-4126-a495-1f8cf256dccc\" (UID: \"ff836440-d844-4126-a495-1f8cf256dccc\") "
Oct 30 01:00:57.477350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:57.477271    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff836440-d844-4126-a495-1f8cf256dccc-kube-api-access-7qzjd\" (UniqueName: \"kubernetes.io/projected/ff836440-d844-4126-a495-1f8cf256dccc-kube-api-access-7qzjd\") pod \"ff836440-d844-4126-a495-1f8cf256dccc\" (UID: \"ff836440-d844-4126-a495-1f8cf256dccc\") "
Oct 30 01:00:57.477618 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:57.477595    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff836440-d844-4126-a495-1f8cf256dccc-maas-api-openshift-ca" pod "ff836440-d844-4126-a495-1f8cf256dccc" (UID: "ff836440-d844-4126-a495-1f8cf256dccc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:57.477857 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:57.477841    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff836440-d844-4126-a495-1f8cf256dccc-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:00:57.479089 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:57.479063    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff836440-d844-4126-a495-1f8cf256dccc-maas-api-xks-ca" pod "ff836440-d844-4126-a495-1f8cf256dccc" (UID: "ff836440-d844-4126-a495-1f8cf256dccc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:57.479299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:57.479275    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff836440-d844-4126-a495-1f8cf256dccc-kube-api-access-7qzjd" pod "ff836440-d844-4126-a495-1f8cf256dccc" (UID: "ff836440-d844-4126-a495-1f8cf256dccc"). InnerVolumeSpecName "kube-api-access-7qzjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:57.578978 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:57.578949    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff836440-d844-4126-a495-1f8cf256dccc-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:00:57.578978 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:57.578986    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qzjd\" (UniqueName: \"kubernetes.io/projected/ff836440-d844-4126-a495-1f8cf256dccc-kube-api-access-7qzjd\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:00:58.216075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:58.216036    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8" event={"ID":"ff836440-d844-4126-a495-1f8cf256dccc","Type":"ContainerDied","Data":"d6866bef7fc1d2d3437aa3de9231837566697084191941099234c6be40c4f083"}
Oct 30 01:00:58.216289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:58.216102    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6866bef7fc1d2d3437aa3de9231837566697084191941099234c6be40c4f083"
Oct 30 01:00:58.216289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:58.216053    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"
Oct 30 01:00:59.181515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.181431    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"]
Oct 30 01:00:59.182239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.182192    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ff836440-d844-4126-a495-1f8cf256dccc" containerName="revoke-keys"
Oct 30 01:00:59.182239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.182214    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff836440-d844-4126-a495-1f8cf256dccc" containerName="revoke-keys"
Oct 30 01:00:59.182428 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.182414    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff836440-d844-4126-a495-1f8cf256dccc" containerName="revoke-keys"
Oct 30 01:00:59.186117 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.186046    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.207858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.207820    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.208011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.207869    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.208011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.207904    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.208011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.207971    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.208275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.208201    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae18a2f-da25-474a-af36-6be757b43311-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.208275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.208227    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtmhn\" (UniqueName: \"kubernetes.io/projected/dae18a2f-da25-474a-af36-6be757b43311-kube-api-access-dtmhn\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.208388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.208305    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae18a2f-da25-474a-af36-6be757b43311-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.208388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.208337    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae18a2f-da25-474a-af36-6be757b43311-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.208388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.208366    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.210847 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.210826    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"]
Oct 30 01:00:59.309700 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.309667    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.309700 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.309704    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.309976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.309729    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.309976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.309764    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.309976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.309815    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae18a2f-da25-474a-af36-6be757b43311-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.309976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.309830    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtmhn\" (UniqueName: \"kubernetes.io/projected/dae18a2f-da25-474a-af36-6be757b43311-kube-api-access-dtmhn\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.309976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.309938    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae18a2f-da25-474a-af36-6be757b43311-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.310284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.309995    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae18a2f-da25-474a-af36-6be757b43311-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.310284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.310032    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.310284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.310192    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.310427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.310313    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.310427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.310375    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.310521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.310499    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.310888 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.310862    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae18a2f-da25-474a-af36-6be757b43311-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.312321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.312296    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.312492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.312475    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae18a2f-da25-474a-af36-6be757b43311-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.330687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.330658    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae18a2f-da25-474a-af36-6be757b43311-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.330862 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.330842    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtmhn\" (UniqueName: \"kubernetes.io/projected/dae18a2f-da25-474a-af36-6be757b43311-kube-api-access-dtmhn\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ltz8j\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.500146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.500061    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:00:59.820802 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.820773    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"]
Oct 30 01:00:59.820985 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:00:59.820962    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae18a2f_da25_474a_af36_6be757b43311.slice/crio-fdc4529e78a174f383c5582b8c1c8cea050b26a6809421e69e8456777f29ad96 WatchSource:0}: Error finding container fdc4529e78a174f383c5582b8c1c8cea050b26a6809421e69e8456777f29ad96: Status 404 returned error can't find the container with id fdc4529e78a174f383c5582b8c1c8cea050b26a6809421e69e8456777f29ad96
Oct 30 01:00:59.823465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.823429    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:59.823585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.823495    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:00:59.823585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:00:59.823523    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:01:00.240445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:00.240367    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j" event={"ID":"dae18a2f-da25-474a-af36-6be757b43311","Type":"ContainerStarted","Data":"df0be71d9ef343d2168a976ab5974e6f22bfb29a0a44bb1d27dd4046d8d33b03"}
Oct 30 01:01:00.240445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:00.240419    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j" event={"ID":"dae18a2f-da25-474a-af36-6be757b43311","Type":"ContainerStarted","Data":"fdc4529e78a174f383c5582b8c1c8cea050b26a6809421e69e8456777f29ad96"}
Oct 30 01:01:00.286700 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:00.286639    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j" podStartSLOduration=1.286621622 podStartE2EDuration="1.286621622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:00.28375161 +0000 UTC m=+1169.033406919" watchObservedRunningTime="2026-10-30 01:01:00.286621622 +0000 UTC m=+1169.036276929"
Oct 30 01:01:00.476414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:00.476367    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:01:00.501241 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:00.501134    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:01:00.502914 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:00.502886    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ltz8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 01:01:00.503053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:00.502935    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j" podUID="dae18a2f-da25-474a-af36-6be757b43311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 01:01:01.507805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:01.507753    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ltz8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 01:01:01.508389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:01.507832    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j" podUID="dae18a2f-da25-474a-af36-6be757b43311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 01:01:02.539765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:02.539724    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ltz8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 01:01:02.540405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:02.540368    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j" podUID="dae18a2f-da25-474a-af36-6be757b43311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 01:01:03.500800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:03.500757    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ltz8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 01:01:03.501015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:03.500822    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j" podUID="dae18a2f-da25-474a-af36-6be757b43311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 01:01:04.500489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.500444    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ltz8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 01:01:04.500948 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.500510    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j" podUID="dae18a2f-da25-474a-af36-6be757b43311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 01:01:04.647036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.647004    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"]
Oct 30 01:01:04.668985 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.657300    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"
Oct 30 01:01:04.682020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.679339    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:01:04.683207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.683182    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:01:04.683391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.683377    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:01:04.683531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.683513    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:01:04.684944 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.684922    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"]
Oct 30 01:01:04.804535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.804487    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6617180-d8xzj\" (UID: \"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"
Oct 30 01:01:04.804744 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.804724    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6617180-d8xzj\" (UID: \"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"
Oct 30 01:01:04.804859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.804838    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z56h\" (UniqueName: \"kubernetes.io/projected/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-kube-api-access-5z56h\") pod \"maas-api-revoke-sub-a6617180-d8xzj\" (UID: \"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"
Oct 30 01:01:04.906008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.905973    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6617180-d8xzj\" (UID: \"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"
Oct 30 01:01:04.906229 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.906137    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6617180-d8xzj\" (UID: \"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"
Oct 30 01:01:04.906303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.906238    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z56h\" (UniqueName: \"kubernetes.io/projected/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-kube-api-access-5z56h\") pod \"maas-api-revoke-sub-a6617180-d8xzj\" (UID: \"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"
Oct 30 01:01:04.906819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.906791    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6617180-d8xzj\" (UID: \"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"
Oct 30 01:01:04.908445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.908424    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6617180-d8xzj\" (UID: \"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"
Oct 30 01:01:04.926807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.926780    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z56h\" (UniqueName: \"kubernetes.io/projected/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-kube-api-access-5z56h\") pod \"maas-api-revoke-sub-a6617180-d8xzj\" (UID: \"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"
Oct 30 01:01:04.989450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:04.989414    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"
Oct 30 01:01:05.236655 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.236620    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"]
Oct 30 01:01:05.288537 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.288504    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"
Oct 30 01:01:05.292392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.292358    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"]
Oct 30 01:01:05.307450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.305103    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-153746\""
Oct 30 01:01:05.307667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.307644    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"]
Oct 30 01:01:05.311408 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.311387    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"]
Oct 30 01:01:05.318292 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.318273    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"
Oct 30 01:01:05.332711 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.332684    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"]
Oct 30 01:01:05.334619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.334593    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-153746-dockercfg-qqffv\""
Oct 30 01:01:05.428178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.413355    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b29bb481-ba88-4108-855e-bd487d1ee296-plugins-config-volume\") pod \"payload-processing-e2e-api-b-153746-c5db87648-7hghn\" (UID: \"b29bb481-ba88-4108-855e-bd487d1ee296\") " pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"
Oct 30 01:01:05.428178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.413489    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4srgb\" (UniqueName: \"kubernetes.io/projected/b29bb481-ba88-4108-855e-bd487d1ee296-kube-api-access-4srgb\") pod \"payload-processing-e2e-api-b-153746-c5db87648-7hghn\" (UID: \"b29bb481-ba88-4108-855e-bd487d1ee296\") " pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"
Oct 30 01:01:05.428178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.413539    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f7zb\" (UniqueName: \"kubernetes.io/projected/03c01c12-309c-4260-b87c-374110f957fd-kube-api-access-9f7zb\") pod \"payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2\" (UID: \"03c01c12-309c-4260-b87c-374110f957fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"
Oct 30 01:01:05.428178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.413622    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c01c12-309c-4260-b87c-374110f957fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2\" (UID: \"03c01c12-309c-4260-b87c-374110f957fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"
Oct 30 01:01:05.512177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.504589    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ltz8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 01:01:05.512177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.504662    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j" podUID="dae18a2f-da25-474a-af36-6be757b43311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 01:01:05.515126 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.515086    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b29bb481-ba88-4108-855e-bd487d1ee296-plugins-config-volume\") pod \"payload-processing-e2e-api-b-153746-c5db87648-7hghn\" (UID: \"b29bb481-ba88-4108-855e-bd487d1ee296\") " pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"
Oct 30 01:01:05.516324 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.516303    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4srgb\" (UniqueName: \"kubernetes.io/projected/b29bb481-ba88-4108-855e-bd487d1ee296-kube-api-access-4srgb\") pod \"payload-processing-e2e-api-b-153746-c5db87648-7hghn\" (UID: \"b29bb481-ba88-4108-855e-bd487d1ee296\") " pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"
Oct 30 01:01:05.516521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.516484    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f7zb\" (UniqueName: \"kubernetes.io/projected/03c01c12-309c-4260-b87c-374110f957fd-kube-api-access-9f7zb\") pod \"payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2\" (UID: \"03c01c12-309c-4260-b87c-374110f957fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"
Oct 30 01:01:05.516699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.516684    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c01c12-309c-4260-b87c-374110f957fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2\" (UID: \"03c01c12-309c-4260-b87c-374110f957fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"
Oct 30 01:01:05.517785 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.517754    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c01c12-309c-4260-b87c-374110f957fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2\" (UID: \"03c01c12-309c-4260-b87c-374110f957fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"
Oct 30 01:01:05.517933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.516183    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b29bb481-ba88-4108-855e-bd487d1ee296-plugins-config-volume\") pod \"payload-processing-e2e-api-b-153746-c5db87648-7hghn\" (UID: \"b29bb481-ba88-4108-855e-bd487d1ee296\") " pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"
Oct 30 01:01:05.530978 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.530932    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj" event={"ID":"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680","Type":"ContainerStarted","Data":"a3abc73ab14153175b36dc68b36f4d8dc311a0d2fe995dcc1db47687bb9844a1"}
Oct 30 01:01:05.544946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.544910    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f7zb\" (UniqueName: \"kubernetes.io/projected/03c01c12-309c-4260-b87c-374110f957fd-kube-api-access-9f7zb\") pod \"payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2\" (UID: \"03c01c12-309c-4260-b87c-374110f957fd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"
Oct 30 01:01:05.546193 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.545591    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4srgb\" (UniqueName: \"kubernetes.io/projected/b29bb481-ba88-4108-855e-bd487d1ee296-kube-api-access-4srgb\") pod \"payload-processing-e2e-api-b-153746-c5db87648-7hghn\" (UID: \"b29bb481-ba88-4108-855e-bd487d1ee296\") " pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"
Oct 30 01:01:05.608746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.608718    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"
Oct 30 01:01:05.683813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:05.683439    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"
Oct 30 01:01:06.028672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:06.028644    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"]
Oct 30 01:01:06.029002 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:01:06.028977    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29bb481_ba88_4108_855e_bd487d1ee296.slice/crio-aac435b4cd1f08fd2492c68345c59a8ebdadec994bb92624cfddac2b860ab18c WatchSource:0}: Error finding container aac435b4cd1f08fd2492c68345c59a8ebdadec994bb92624cfddac2b860ab18c: Status 404 returned error can't find the container with id aac435b4cd1f08fd2492c68345c59a8ebdadec994bb92624cfddac2b860ab18c
Oct 30 01:01:06.087210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:06.087186    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"]
Oct 30 01:01:06.088125 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:01:06.088102    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c01c12_309c_4260_b87c_374110f957fd.slice/crio-8177f62bb01d8ed1f4e34c72c02e0bce6e2ea1e85217693685e89b8f54ce108f WatchSource:0}: Error finding container 8177f62bb01d8ed1f4e34c72c02e0bce6e2ea1e85217693685e89b8f54ce108f: Status 404 returned error can't find the container with id 8177f62bb01d8ed1f4e34c72c02e0bce6e2ea1e85217693685e89b8f54ce108f
Oct 30 01:01:06.501207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:06.501102    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ltz8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 01:01:06.501207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:06.501185    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j" podUID="dae18a2f-da25-474a-af36-6be757b43311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 01:01:06.540403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:06.540366    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn" event={"ID":"b29bb481-ba88-4108-855e-bd487d1ee296","Type":"ContainerStarted","Data":"b854e7588937671bcc68b656975b54123dc0609a7da05531b7c55abcf92910a3"}
Oct 30 01:01:06.540874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:06.540436    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"
Oct 30 01:01:06.540874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:06.540461    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn" event={"ID":"b29bb481-ba88-4108-855e-bd487d1ee296","Type":"ContainerStarted","Data":"aac435b4cd1f08fd2492c68345c59a8ebdadec994bb92624cfddac2b860ab18c"}
Oct 30 01:01:06.542346 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:06.542319    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj" event={"ID":"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680","Type":"ContainerStarted","Data":"29bc7cca1f5c7da995a73d5019c2e52c6bd20e62ac4d2d736dbb81d8fd2fb3ca"}
Oct 30 01:01:06.543985 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:06.543958    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2" event={"ID":"03c01c12-309c-4260-b87c-374110f957fd","Type":"ContainerStarted","Data":"bf51bbb0443530fc2f082aba6a6ace7b6b25eb87d40acfe79609d6d50db00b05"}
Oct 30 01:01:06.544074 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:06.544009    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2" event={"ID":"03c01c12-309c-4260-b87c-374110f957fd","Type":"ContainerStarted","Data":"8177f62bb01d8ed1f4e34c72c02e0bce6e2ea1e85217693685e89b8f54ce108f"}
Oct 30 01:01:06.544119 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:06.544089    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"
Oct 30 01:01:06.579079 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:06.578497    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn" podStartSLOduration=1.578480941 podStartE2EDuration="1.578480941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:06.575464639 +0000 UTC m=+1175.325119949" watchObservedRunningTime="2026-10-30 01:01:06.578480941 +0000 UTC m=+1175.328136251"
Oct 30 01:01:06.614419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:06.614363    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj" podStartSLOduration=2.614345335 podStartE2EDuration="2.614345335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:06.610638978 +0000 UTC m=+1175.360294287" watchObservedRunningTime="2026-10-30 01:01:06.614345335 +0000 UTC m=+1175.364000645"
Oct 30 01:01:06.651858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:06.651808    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2" podStartSLOduration=1.651788558 podStartE2EDuration="1.651788558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:06.647982731 +0000 UTC m=+1175.397638054" watchObservedRunningTime="2026-10-30 01:01:06.651788558 +0000 UTC m=+1175.401443866"
Oct 30 01:01:07.500902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:07.500864    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ltz8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 30 01:01:07.501066 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:07.500923    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j" podUID="dae18a2f-da25-474a-af36-6be757b43311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 30 01:01:07.565417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:07.565292    4425 generic.go:356] "Generic (PLEG): container finished" podID="2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680" containerID="29bc7cca1f5c7da995a73d5019c2e52c6bd20e62ac4d2d736dbb81d8fd2fb3ca" exitCode=0
Oct 30 01:01:07.568295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:07.566197    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj" event={"ID":"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680","Type":"ContainerDied","Data":"29bc7cca1f5c7da995a73d5019c2e52c6bd20e62ac4d2d736dbb81d8fd2fb3ca"}
Oct 30 01:01:07.572121 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:01:07.572091    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:01:08.504600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:08.504573    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:01:08.504879 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:08.504852    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:01:08.505620 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:08.505601    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:01:08.758622 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:08.758560    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"
Oct 30 01:01:08.866299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:08.866268    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-maas-api-openshift-ca\") pod \"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680\" (UID: \"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680\") "
Oct 30 01:01:08.866464 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:08.866333    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-kube-api-access-5z56h\" (UniqueName: \"kubernetes.io/projected/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-kube-api-access-5z56h\") pod \"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680\" (UID: \"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680\") "
Oct 30 01:01:08.866573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:08.866550    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-maas-api-xks-ca\") pod \"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680\" (UID: \"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680\") "
Oct 30 01:01:08.866678 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:08.866650    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-maas-api-openshift-ca" pod "2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680" (UID: "2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:08.867083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:08.867060    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:01:08.868381 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:08.868358    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-kube-api-access-5z56h" pod "2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680" (UID: "2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680"). InnerVolumeSpecName "kube-api-access-5z56h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:08.868482 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:08.868391    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-maas-api-xks-ca" pod "2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680" (UID: "2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:08.968144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:08.968118    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:01:08.968144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:08.968141    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z56h\" (UniqueName: \"kubernetes.io/projected/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680-kube-api-access-5z56h\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:01:09.580020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:09.579994    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"
Oct 30 01:01:09.580231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:09.580029    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj" event={"ID":"2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680","Type":"ContainerDied","Data":"a3abc73ab14153175b36dc68b36f4d8dc311a0d2fe995dcc1db47687bb9844a1"}
Oct 30 01:01:09.580231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:09.580073    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3abc73ab14153175b36dc68b36f4d8dc311a0d2fe995dcc1db47687bb9844a1"
Oct 30 01:01:17.202933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:17.202897    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"]
Oct 30 01:01:17.212471 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:17.212437    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1406e8-nc7gk"]
Oct 30 01:01:17.513409 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:17.512191    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7164c599-d510-4d81-96f9-c8d14ab111b9" path="/var/lib/kubelet/pods/7164c599-d510-4d81-96f9-c8d14ab111b9/volumes"
Oct 30 01:01:17.712425 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:17.712396    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"
Oct 30 01:01:17.801849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:17.801765    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"
Oct 30 01:01:18.924683 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:18.923648    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"]
Oct 30 01:01:18.925185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:18.925136    4425 cpu_manager.go:414] "containerMap: removing container" podUID="2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680" containerName="revoke-keys"
Oct 30 01:01:18.925185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:18.925175    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680" containerName="revoke-keys"
Oct 30 01:01:18.929174 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:18.925480    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680" containerName="revoke-keys"
Oct 30 01:01:18.931747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:18.931724    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"
Oct 30 01:01:18.948114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:18.948090    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"]
Oct 30 01:01:18.948491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:18.948379    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:01:18.948491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:18.948405    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:01:18.957051 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:18.957031    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:01:18.957215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:18.957121    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:01:19.013405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:19.013367    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6974c649-c853-4f80-a251-15cdb72b6de4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af37dd22-gblwq\" (UID: \"6974c649-c853-4f80-a251-15cdb72b6de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"
Oct 30 01:01:19.013562 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:19.013536    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58xvj\" (UniqueName: \"kubernetes.io/projected/6974c649-c853-4f80-a251-15cdb72b6de4-kube-api-access-58xvj\") pod \"maas-api-revoke-sub-af37dd22-gblwq\" (UID: \"6974c649-c853-4f80-a251-15cdb72b6de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"
Oct 30 01:01:19.013619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:19.013576    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6974c649-c853-4f80-a251-15cdb72b6de4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af37dd22-gblwq\" (UID: \"6974c649-c853-4f80-a251-15cdb72b6de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"
Oct 30 01:01:19.114665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:19.114633    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6974c649-c853-4f80-a251-15cdb72b6de4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af37dd22-gblwq\" (UID: \"6974c649-c853-4f80-a251-15cdb72b6de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"
Oct 30 01:01:19.114863 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:19.114809    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58xvj\" (UniqueName: \"kubernetes.io/projected/6974c649-c853-4f80-a251-15cdb72b6de4-kube-api-access-58xvj\") pod \"maas-api-revoke-sub-af37dd22-gblwq\" (UID: \"6974c649-c853-4f80-a251-15cdb72b6de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"
Oct 30 01:01:19.114927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:19.114869    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6974c649-c853-4f80-a251-15cdb72b6de4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af37dd22-gblwq\" (UID: \"6974c649-c853-4f80-a251-15cdb72b6de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"
Oct 30 01:01:19.115902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:19.115876    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6974c649-c853-4f80-a251-15cdb72b6de4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af37dd22-gblwq\" (UID: \"6974c649-c853-4f80-a251-15cdb72b6de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"
Oct 30 01:01:19.133433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:19.122597    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6974c649-c853-4f80-a251-15cdb72b6de4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af37dd22-gblwq\" (UID: \"6974c649-c853-4f80-a251-15cdb72b6de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"
Oct 30 01:01:19.136966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:19.136939    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58xvj\" (UniqueName: \"kubernetes.io/projected/6974c649-c853-4f80-a251-15cdb72b6de4-kube-api-access-58xvj\") pod \"maas-api-revoke-sub-af37dd22-gblwq\" (UID: \"6974c649-c853-4f80-a251-15cdb72b6de4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"
Oct 30 01:01:19.255907 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:19.255816    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"
Oct 30 01:01:19.566514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:19.566485    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"]
Oct 30 01:01:19.567624 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:01:19.567600    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6974c649_c853_4f80_a251_15cdb72b6de4.slice/crio-7d11eded2ffaba00996d9ac2a65215869118035443f96493ca213ff8894a97f8 WatchSource:0}: Error finding container 7d11eded2ffaba00996d9ac2a65215869118035443f96493ca213ff8894a97f8: Status 404 returned error can't find the container with id 7d11eded2ffaba00996d9ac2a65215869118035443f96493ca213ff8894a97f8
Oct 30 01:01:19.727961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:19.727923    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq" event={"ID":"6974c649-c853-4f80-a251-15cdb72b6de4","Type":"ContainerStarted","Data":"7d11eded2ffaba00996d9ac2a65215869118035443f96493ca213ff8894a97f8"}
Oct 30 01:01:20.738194 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:20.738068    4425 generic.go:356] "Generic (PLEG): container finished" podID="6974c649-c853-4f80-a251-15cdb72b6de4" containerID="492e0f7a7ae9ae22241308235fd80243ce4f2a055cee10a9cc6a2d3a32f95560" exitCode=0
Oct 30 01:01:20.738625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:20.738150    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq" event={"ID":"6974c649-c853-4f80-a251-15cdb72b6de4","Type":"ContainerDied","Data":"492e0f7a7ae9ae22241308235fd80243ce4f2a055cee10a9cc6a2d3a32f95560"}
Oct 30 01:01:21.902651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:21.902625    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"
Oct 30 01:01:22.055146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:22.055110    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6974c649-c853-4f80-a251-15cdb72b6de4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6974c649-c853-4f80-a251-15cdb72b6de4-maas-api-openshift-ca\") pod \"6974c649-c853-4f80-a251-15cdb72b6de4\" (UID: \"6974c649-c853-4f80-a251-15cdb72b6de4\") "
Oct 30 01:01:22.055355 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:22.055275    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6974c649-c853-4f80-a251-15cdb72b6de4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6974c649-c853-4f80-a251-15cdb72b6de4-maas-api-xks-ca\") pod \"6974c649-c853-4f80-a251-15cdb72b6de4\" (UID: \"6974c649-c853-4f80-a251-15cdb72b6de4\") "
Oct 30 01:01:22.055355 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:22.055316    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6974c649-c853-4f80-a251-15cdb72b6de4-kube-api-access-58xvj\" (UniqueName: \"kubernetes.io/projected/6974c649-c853-4f80-a251-15cdb72b6de4-kube-api-access-58xvj\") pod \"6974c649-c853-4f80-a251-15cdb72b6de4\" (UID: \"6974c649-c853-4f80-a251-15cdb72b6de4\") "
Oct 30 01:01:22.055545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:22.055518    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6974c649-c853-4f80-a251-15cdb72b6de4-maas-api-openshift-ca" pod "6974c649-c853-4f80-a251-15cdb72b6de4" (UID: "6974c649-c853-4f80-a251-15cdb72b6de4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:22.055889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:22.055866    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6974c649-c853-4f80-a251-15cdb72b6de4-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.057394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:22.057363    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6974c649-c853-4f80-a251-15cdb72b6de4-maas-api-xks-ca" pod "6974c649-c853-4f80-a251-15cdb72b6de4" (UID: "6974c649-c853-4f80-a251-15cdb72b6de4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:22.059318 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:22.059291    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6974c649-c853-4f80-a251-15cdb72b6de4-kube-api-access-58xvj" pod "6974c649-c853-4f80-a251-15cdb72b6de4" (UID: "6974c649-c853-4f80-a251-15cdb72b6de4"). InnerVolumeSpecName "kube-api-access-58xvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:22.156975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:22.156943    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6974c649-c853-4f80-a251-15cdb72b6de4-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.156975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:22.156971    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58xvj\" (UniqueName: \"kubernetes.io/projected/6974c649-c853-4f80-a251-15cdb72b6de4-kube-api-access-58xvj\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:01:22.762263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:22.762226    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"
Oct 30 01:01:22.762492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:22.762241    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq" event={"ID":"6974c649-c853-4f80-a251-15cdb72b6de4","Type":"ContainerDied","Data":"7d11eded2ffaba00996d9ac2a65215869118035443f96493ca213ff8894a97f8"}
Oct 30 01:01:22.762492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:22.762367    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d11eded2ffaba00996d9ac2a65215869118035443f96493ca213ff8894a97f8"
Oct 30 01:01:24.763690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.763657    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"]
Oct 30 01:01:24.764222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.764208    4425 cpu_manager.go:414] "containerMap: removing container" podUID="6974c649-c853-4f80-a251-15cdb72b6de4" containerName="revoke-keys"
Oct 30 01:01:24.764222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.764222    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6974c649-c853-4f80-a251-15cdb72b6de4" containerName="revoke-keys"
Oct 30 01:01:24.764361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.764352    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6974c649-c853-4f80-a251-15cdb72b6de4" containerName="revoke-keys"
Oct 30 01:01:24.768042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.768026    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"
Oct 30 01:01:24.789111 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.789081    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v75jc\" (UniqueName: \"kubernetes.io/projected/ddd3452d-e481-4d50-8c60-f102ecca1aa7-kube-api-access-v75jc\") pod \"payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf\" (UID: \"ddd3452d-e481-4d50-8c60-f102ecca1aa7\") " pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"
Oct 30 01:01:24.789283 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.789234    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd3452d-e481-4d50-8c60-f102ecca1aa7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf\" (UID: \"ddd3452d-e481-4d50-8c60-f102ecca1aa7\") " pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"
Oct 30 01:01:24.794022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.793998    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6482ae\""
Oct 30 01:01:24.797802 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.797782    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"]
Oct 30 01:01:24.802659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.802638    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"]
Oct 30 01:01:24.806803 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.806786    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"
Oct 30 01:01:24.823449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.823427    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6482ae-dockercfg-rwzhl\""
Oct 30 01:01:24.839225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.839199    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"]
Oct 30 01:01:24.890918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.890886    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd3452d-e481-4d50-8c60-f102ecca1aa7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf\" (UID: \"ddd3452d-e481-4d50-8c60-f102ecca1aa7\") " pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"
Oct 30 01:01:24.891267 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.891235    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr5k2\" (UniqueName: \"kubernetes.io/projected/de1980d7-8abc-483c-8448-5406fbc83631-kube-api-access-fr5k2\") pod \"payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh\" (UID: \"de1980d7-8abc-483c-8448-5406fbc83631\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"
Oct 30 01:01:24.891387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.891355    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1980d7-8abc-483c-8448-5406fbc83631-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh\" (UID: \"de1980d7-8abc-483c-8448-5406fbc83631\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"
Oct 30 01:01:24.891438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.891409    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v75jc\" (UniqueName: \"kubernetes.io/projected/ddd3452d-e481-4d50-8c60-f102ecca1aa7-kube-api-access-v75jc\") pod \"payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf\" (UID: \"ddd3452d-e481-4d50-8c60-f102ecca1aa7\") " pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"
Oct 30 01:01:24.891808 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.891784    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd3452d-e481-4d50-8c60-f102ecca1aa7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf\" (UID: \"ddd3452d-e481-4d50-8c60-f102ecca1aa7\") " pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"
Oct 30 01:01:24.914175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.914108    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v75jc\" (UniqueName: \"kubernetes.io/projected/ddd3452d-e481-4d50-8c60-f102ecca1aa7-kube-api-access-v75jc\") pod \"payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf\" (UID: \"ddd3452d-e481-4d50-8c60-f102ecca1aa7\") " pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"
Oct 30 01:01:24.992996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.992949    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr5k2\" (UniqueName: \"kubernetes.io/projected/de1980d7-8abc-483c-8448-5406fbc83631-kube-api-access-fr5k2\") pod \"payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh\" (UID: \"de1980d7-8abc-483c-8448-5406fbc83631\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"
Oct 30 01:01:24.993244 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.993049    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1980d7-8abc-483c-8448-5406fbc83631-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh\" (UID: \"de1980d7-8abc-483c-8448-5406fbc83631\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"
Oct 30 01:01:24.993861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:24.993838    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1980d7-8abc-483c-8448-5406fbc83631-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh\" (UID: \"de1980d7-8abc-483c-8448-5406fbc83631\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"
Oct 30 01:01:25.014951 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:25.014875    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr5k2\" (UniqueName: \"kubernetes.io/projected/de1980d7-8abc-483c-8448-5406fbc83631-kube-api-access-fr5k2\") pod \"payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh\" (UID: \"de1980d7-8abc-483c-8448-5406fbc83631\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"
Oct 30 01:01:25.082420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:25.082389    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"
Oct 30 01:01:25.121814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:25.121781    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"
Oct 30 01:01:25.393355 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:25.393328    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"]
Oct 30 01:01:25.394283 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:01:25.394256    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd3452d_e481_4d50_8c60_f102ecca1aa7.slice/crio-0c54ce33072fb656ae3c8ca3174a1cf80ef4d0a59a5b617b3df66af76e35f7b2 WatchSource:0}: Error finding container 0c54ce33072fb656ae3c8ca3174a1cf80ef4d0a59a5b617b3df66af76e35f7b2: Status 404 returned error can't find the container with id 0c54ce33072fb656ae3c8ca3174a1cf80ef4d0a59a5b617b3df66af76e35f7b2
Oct 30 01:01:25.479659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:25.479636    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"]
Oct 30 01:01:25.480231 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:01:25.480206    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1980d7_8abc_483c_8448_5406fbc83631.slice/crio-e07dec4731a28539ac0e43f0d36d66756bf7cc814ec37ba8c74df0f3e94a97a4 WatchSource:0}: Error finding container e07dec4731a28539ac0e43f0d36d66756bf7cc814ec37ba8c74df0f3e94a97a4: Status 404 returned error can't find the container with id e07dec4731a28539ac0e43f0d36d66756bf7cc814ec37ba8c74df0f3e94a97a4
Oct 30 01:01:25.782713 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:25.782673    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh" event={"ID":"de1980d7-8abc-483c-8448-5406fbc83631","Type":"ContainerStarted","Data":"0fe934a7ef4979e8a0a9206021d6712a6a9f555e4e3625fbc991fb68fb7f9e64"}
Oct 30 01:01:25.783195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:25.782732    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"
Oct 30 01:01:25.783195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:25.782756    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh" event={"ID":"de1980d7-8abc-483c-8448-5406fbc83631","Type":"ContainerStarted","Data":"e07dec4731a28539ac0e43f0d36d66756bf7cc814ec37ba8c74df0f3e94a97a4"}
Oct 30 01:01:25.784299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:25.784273    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf" event={"ID":"ddd3452d-e481-4d50-8c60-f102ecca1aa7","Type":"ContainerStarted","Data":"c525e16dd2362e13eef8a8d498299339c1683b9962037d4fec5461919c8e4691"}
Oct 30 01:01:25.784418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:25.784318    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf" event={"ID":"ddd3452d-e481-4d50-8c60-f102ecca1aa7","Type":"ContainerStarted","Data":"0c54ce33072fb656ae3c8ca3174a1cf80ef4d0a59a5b617b3df66af76e35f7b2"}
Oct 30 01:01:25.784418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:25.784360    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"
Oct 30 01:01:25.818150 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:25.817965    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh" podStartSLOduration=1.817952348 podStartE2EDuration="1.817952348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:25.817329956 +0000 UTC m=+1194.566985265" watchObservedRunningTime="2026-10-30 01:01:25.817952348 +0000 UTC m=+1194.567607728"
Oct 30 01:01:25.855016 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:25.854973    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf" podStartSLOduration=1.854960466 podStartE2EDuration="1.854960466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:25.852470439 +0000 UTC m=+1194.602125747" watchObservedRunningTime="2026-10-30 01:01:25.854960466 +0000 UTC m=+1194.604615774"
Oct 30 01:01:36.794821 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:36.794791    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"
Oct 30 01:01:36.795360 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:36.794965    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"
Oct 30 01:01:37.555120 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:01:37.555085    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:01:37.631945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:37.631915    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7294-8c5983"]
Oct 30 01:01:37.637253 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:37.637233    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7294-8c5983"
Oct 30 01:01:37.650937 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:37.650917    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7294-8c5983"]
Oct 30 01:01:37.655938 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:37.655917    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4z6vl\""
Oct 30 01:01:37.749758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:37.749729    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nztg8\" (UniqueName: \"kubernetes.io/projected/ff338750-d711-425c-9a44-7a28d98a6090-kube-api-access-nztg8\") pod \"test-curl-7294-8c5983\" (UID: \"ff338750-d711-425c-9a44-7a28d98a6090\") " pod="opendatahub/test-curl-7294-8c5983"
Oct 30 01:01:37.851463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:37.851393    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nztg8\" (UniqueName: \"kubernetes.io/projected/ff338750-d711-425c-9a44-7a28d98a6090-kube-api-access-nztg8\") pod \"test-curl-7294-8c5983\" (UID: \"ff338750-d711-425c-9a44-7a28d98a6090\") " pod="opendatahub/test-curl-7294-8c5983"
Oct 30 01:01:37.872527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:37.872502    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nztg8\" (UniqueName: \"kubernetes.io/projected/ff338750-d711-425c-9a44-7a28d98a6090-kube-api-access-nztg8\") pod \"test-curl-7294-8c5983\" (UID: \"ff338750-d711-425c-9a44-7a28d98a6090\") " pod="opendatahub/test-curl-7294-8c5983"
Oct 30 01:01:37.950706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:37.950683    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7294-8c5983"
Oct 30 01:01:38.252667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:38.252643    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7294-8c5983"]
Oct 30 01:01:38.254353 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:01:38.254326    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff338750_d711_425c_9a44_7a28d98a6090.slice/crio-fed998e38fe9f1905a0821fd57dc6fc71854e490499260813517b93d4185f702 WatchSource:0}: Error finding container fed998e38fe9f1905a0821fd57dc6fc71854e490499260813517b93d4185f702: Status 404 returned error can't find the container with id fed998e38fe9f1905a0821fd57dc6fc71854e490499260813517b93d4185f702
Oct 30 01:01:38.256208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:38.256189    4425 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:01:38.887742 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:38.887710    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7294-8c5983" event={"ID":"ff338750-d711-425c-9a44-7a28d98a6090","Type":"ContainerStarted","Data":"fed998e38fe9f1905a0821fd57dc6fc71854e490499260813517b93d4185f702"}
Oct 30 01:01:39.896375 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:39.896336    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7294-8c5983" event={"ID":"ff338750-d711-425c-9a44-7a28d98a6090","Type":"ContainerStarted","Data":"3888dece46855ac5c4d41c256b6505ceca13d91ae3de382bc3dbf3cca7a396f3"}
Oct 30 01:01:39.932860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:39.932810    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7294-8c5983" podStartSLOduration=2.217608506 podStartE2EDuration="2.932797758s" totalImagesPullingTime="715.189252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:39.929530903 +0000 UTC m=+1208.679186213" watchObservedRunningTime="2026-10-30 01:01:39.932797758 +0000 UTC m=+1208.682453068"
Oct 30 01:01:40.826766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:40.825909    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7294-8c5983"]
Oct 30 01:01:40.830599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:40.830572    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7294-8c5983"]
Oct 30 01:01:41.511440 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.511409    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"]
Oct 30 01:01:41.512207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.512185    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ff338750-d711-425c-9a44-7a28d98a6090" containerName="test-curl-7294-8c5983"
Oct 30 01:01:41.512207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.512207    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff338750-d711-425c-9a44-7a28d98a6090" containerName="test-curl-7294-8c5983"
Oct 30 01:01:41.512415 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.512400    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff338750-d711-425c-9a44-7a28d98a6090" containerName="test-curl-7294-8c5983"
Oct 30 01:01:41.515985 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.515965    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"
Oct 30 01:01:41.533606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.533581    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"]
Oct 30 01:01:41.539181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.538540    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:01:41.539181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.538558    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:01:41.539181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.538601    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:01:41.539181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.538642    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:01:41.599918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.599894    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aac47da-0db9-4277-845c-c167c6244317-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-4rs24\" (UID: \"6aac47da-0db9-4277-845c-c167c6244317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"
Oct 30 01:01:41.600032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.599929    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aac47da-0db9-4277-845c-c167c6244317-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-4rs24\" (UID: \"6aac47da-0db9-4277-845c-c167c6244317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"
Oct 30 01:01:41.600075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.600045    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5pkx\" (UniqueName: \"kubernetes.io/projected/6aac47da-0db9-4277-845c-c167c6244317-kube-api-access-q5pkx\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-4rs24\" (UID: \"6aac47da-0db9-4277-845c-c167c6244317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"
Oct 30 01:01:41.701601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.701571    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aac47da-0db9-4277-845c-c167c6244317-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-4rs24\" (UID: \"6aac47da-0db9-4277-845c-c167c6244317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"
Oct 30 01:01:41.701731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.701616    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aac47da-0db9-4277-845c-c167c6244317-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-4rs24\" (UID: \"6aac47da-0db9-4277-845c-c167c6244317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"
Oct 30 01:01:41.701731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.701697    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5pkx\" (UniqueName: \"kubernetes.io/projected/6aac47da-0db9-4277-845c-c167c6244317-kube-api-access-q5pkx\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-4rs24\" (UID: \"6aac47da-0db9-4277-845c-c167c6244317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"
Oct 30 01:01:41.702567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.702538    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aac47da-0db9-4277-845c-c167c6244317-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-4rs24\" (UID: \"6aac47da-0db9-4277-845c-c167c6244317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"
Oct 30 01:01:41.706275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.706250    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aac47da-0db9-4277-845c-c167c6244317-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-4rs24\" (UID: \"6aac47da-0db9-4277-845c-c167c6244317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"
Oct 30 01:01:41.722715 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.722685    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5pkx\" (UniqueName: \"kubernetes.io/projected/6aac47da-0db9-4277-845c-c167c6244317-kube-api-access-q5pkx\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-4rs24\" (UID: \"6aac47da-0db9-4277-845c-c167c6244317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"
Oct 30 01:01:41.830018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.829952    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"
Oct 30 01:01:41.921491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:41.921449    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7294-8c5983" podUID="ff338750-d711-425c-9a44-7a28d98a6090" containerName="test-curl-7294-8c5983" containerID="cri-o://3888dece46855ac5c4d41c256b6505ceca13d91ae3de382bc3dbf3cca7a396f3" gracePeriod=2
Oct 30 01:01:42.135807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:42.135777    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"]
Oct 30 01:01:42.136836 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:01:42.136811    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aac47da_0db9_4277_845c_c167c6244317.slice/crio-1171389195d445b26b8b04e721233f407cad88ba478ee025637aedec302b5d28 WatchSource:0}: Error finding container 1171389195d445b26b8b04e721233f407cad88ba478ee025637aedec302b5d28: Status 404 returned error can't find the container with id 1171389195d445b26b8b04e721233f407cad88ba478ee025637aedec302b5d28
Oct 30 01:01:42.928752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:42.928719    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24" event={"ID":"6aac47da-0db9-4277-845c-c167c6244317","Type":"ContainerStarted","Data":"31d7ff2cd851ad05da1b708d85698bcc168dd8086fedacb415cf73bc0108c412"}
Oct 30 01:01:42.929127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:42.928774    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24" event={"ID":"6aac47da-0db9-4277-845c-c167c6244317","Type":"ContainerStarted","Data":"1171389195d445b26b8b04e721233f407cad88ba478ee025637aedec302b5d28"}
Oct 30 01:01:42.966210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:42.966144    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24" podStartSLOduration=1.966128346 podStartE2EDuration="1.966128346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:42.965450253 +0000 UTC m=+1211.715105565" watchObservedRunningTime="2026-10-30 01:01:42.966128346 +0000 UTC m=+1211.715783656"
Oct 30 01:01:43.054176 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:43.053945    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:01:43.942795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:43.942766    4425 generic.go:356] "Generic (PLEG): container finished" podID="ff338750-d711-425c-9a44-7a28d98a6090" containerID="3888dece46855ac5c4d41c256b6505ceca13d91ae3de382bc3dbf3cca7a396f3" exitCode=137
Oct 30 01:01:43.944589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:43.944569    4425 generic.go:356] "Generic (PLEG): container finished" podID="6aac47da-0db9-4277-845c-c167c6244317" containerID="31d7ff2cd851ad05da1b708d85698bcc168dd8086fedacb415cf73bc0108c412" exitCode=0
Oct 30 01:01:43.944689 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:43.944623    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24" event={"ID":"6aac47da-0db9-4277-845c-c167c6244317","Type":"ContainerDied","Data":"31d7ff2cd851ad05da1b708d85698bcc168dd8086fedacb415cf73bc0108c412"}
Oct 30 01:01:44.217164 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:44.217132    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7294-8c5983"
Oct 30 01:01:44.353933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:44.353898    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff338750-d711-425c-9a44-7a28d98a6090-kube-api-access-nztg8\" (UniqueName: \"kubernetes.io/projected/ff338750-d711-425c-9a44-7a28d98a6090-kube-api-access-nztg8\") pod \"ff338750-d711-425c-9a44-7a28d98a6090\" (UID: \"ff338750-d711-425c-9a44-7a28d98a6090\") "
Oct 30 01:01:44.356519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:44.356487    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff338750-d711-425c-9a44-7a28d98a6090-kube-api-access-nztg8" pod "ff338750-d711-425c-9a44-7a28d98a6090" (UID: "ff338750-d711-425c-9a44-7a28d98a6090"). InnerVolumeSpecName "kube-api-access-nztg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:44.456371 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:44.456273    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nztg8\" (UniqueName: \"kubernetes.io/projected/ff338750-d711-425c-9a44-7a28d98a6090-kube-api-access-nztg8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:01:44.974025 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:44.973989    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7294-8c5983"
Oct 30 01:01:44.975519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:44.975496    4425 scope.go:122] "RemoveContainer" containerID="3888dece46855ac5c4d41c256b6505ceca13d91ae3de382bc3dbf3cca7a396f3"
Oct 30 01:01:45.009789 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:01:45.009710    4425 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7294-8c5983\" is forbidden: User \"system:node:ip-10-0-133-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-32.ec2.internal' and this object" podUID="ff338750-d711-425c-9a44-7a28d98a6090" pod="opendatahub/test-curl-7294-8c5983"
Oct 30 01:01:45.258200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:45.252307    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"
Oct 30 01:01:45.272177 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:01:45.263961    4425 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7294-8c5983\" is forbidden: User \"system:node:ip-10-0-133-32.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-32.ec2.internal' and this object" podUID="ff338750-d711-425c-9a44-7a28d98a6090" pod="opendatahub/test-curl-7294-8c5983"
Oct 30 01:01:45.342374 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:45.342332    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aac47da-0db9-4277-845c-c167c6244317-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aac47da-0db9-4277-845c-c167c6244317-maas-api-xks-ca\") pod \"6aac47da-0db9-4277-845c-c167c6244317\" (UID: \"6aac47da-0db9-4277-845c-c167c6244317\") "
Oct 30 01:01:45.342565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:45.342509    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aac47da-0db9-4277-845c-c167c6244317-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aac47da-0db9-4277-845c-c167c6244317-maas-api-openshift-ca\") pod \"6aac47da-0db9-4277-845c-c167c6244317\" (UID: \"6aac47da-0db9-4277-845c-c167c6244317\") "
Oct 30 01:01:45.342648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:45.342575    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aac47da-0db9-4277-845c-c167c6244317-kube-api-access-q5pkx\" (UniqueName: \"kubernetes.io/projected/6aac47da-0db9-4277-845c-c167c6244317-kube-api-access-q5pkx\") pod \"6aac47da-0db9-4277-845c-c167c6244317\" (UID: \"6aac47da-0db9-4277-845c-c167c6244317\") "
Oct 30 01:01:45.344802 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:45.344750    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aac47da-0db9-4277-845c-c167c6244317-maas-api-openshift-ca" pod "6aac47da-0db9-4277-845c-c167c6244317" (UID: "6aac47da-0db9-4277-845c-c167c6244317"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:45.349088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:45.349048    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aac47da-0db9-4277-845c-c167c6244317-maas-api-xks-ca" pod "6aac47da-0db9-4277-845c-c167c6244317" (UID: "6aac47da-0db9-4277-845c-c167c6244317"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:45.361242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:45.353602    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aac47da-0db9-4277-845c-c167c6244317-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:01:45.361242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:45.353632    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aac47da-0db9-4277-845c-c167c6244317-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:01:45.397386 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:45.390015    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aac47da-0db9-4277-845c-c167c6244317-kube-api-access-q5pkx" pod "6aac47da-0db9-4277-845c-c167c6244317" (UID: "6aac47da-0db9-4277-845c-c167c6244317"). InnerVolumeSpecName "kube-api-access-q5pkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:45.505254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:45.447930    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff338750-d711-425c-9a44-7a28d98a6090" path="/var/lib/kubelet/pods/ff338750-d711-425c-9a44-7a28d98a6090/volumes"
Oct 30 01:01:45.505254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:45.476032    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5pkx\" (UniqueName: \"kubernetes.io/projected/6aac47da-0db9-4277-845c-c167c6244317-kube-api-access-q5pkx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:01:46.025706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:46.025534    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24" event={"ID":"6aac47da-0db9-4277-845c-c167c6244317","Type":"ContainerDied","Data":"1171389195d445b26b8b04e721233f407cad88ba478ee025637aedec302b5d28"}
Oct 30 01:01:46.025706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:46.025623    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"
Oct 30 01:01:46.025706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:46.025648    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1171389195d445b26b8b04e721233f407cad88ba478ee025637aedec302b5d28"
Oct 30 01:01:51.069345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.069313    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"]
Oct 30 01:01:51.070115 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.070092    4425 cpu_manager.go:414] "containerMap: removing container" podUID="6aac47da-0db9-4277-845c-c167c6244317" containerName="revoke-keys"
Oct 30 01:01:51.070256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.070116    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aac47da-0db9-4277-845c-c167c6244317" containerName="revoke-keys"
Oct 30 01:01:51.070349 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.070335    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aac47da-0db9-4277-845c-c167c6244317" containerName="revoke-keys"
Oct 30 01:01:51.073826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.073807    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"
Oct 30 01:01:51.096266 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.096237    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df03d21-8093-4916-a005-b0448deee17b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-52b947-zjv4q\" (UID: \"7df03d21-8093-4916-a005-b0448deee17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"
Oct 30 01:01:51.096559 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.096532    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df03d21-8093-4916-a005-b0448deee17b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-52b947-zjv4q\" (UID: \"7df03d21-8093-4916-a005-b0448deee17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"
Oct 30 01:01:51.096624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.096599    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl82q\" (UniqueName: \"kubernetes.io/projected/7df03d21-8093-4916-a005-b0448deee17b-kube-api-access-vl82q\") pod \"maas-api-revoke-keys-e2e-api-a-52b947-zjv4q\" (UID: \"7df03d21-8093-4916-a005-b0448deee17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"
Oct 30 01:01:51.097313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.097295    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:01:51.097397 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.097350    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:01:51.097397 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.097351    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:01:51.097609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.097579    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:01:51.102056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.102019    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"]
Oct 30 01:01:51.197604 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.197578    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df03d21-8093-4916-a005-b0448deee17b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-52b947-zjv4q\" (UID: \"7df03d21-8093-4916-a005-b0448deee17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"
Oct 30 01:01:51.197755 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.197738    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df03d21-8093-4916-a005-b0448deee17b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-52b947-zjv4q\" (UID: \"7df03d21-8093-4916-a005-b0448deee17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"
Oct 30 01:01:51.197799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.197777    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl82q\" (UniqueName: \"kubernetes.io/projected/7df03d21-8093-4916-a005-b0448deee17b-kube-api-access-vl82q\") pod \"maas-api-revoke-keys-e2e-api-a-52b947-zjv4q\" (UID: \"7df03d21-8093-4916-a005-b0448deee17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"
Oct 30 01:01:51.198383 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.198362    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df03d21-8093-4916-a005-b0448deee17b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-52b947-zjv4q\" (UID: \"7df03d21-8093-4916-a005-b0448deee17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"
Oct 30 01:01:51.199884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.199863    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df03d21-8093-4916-a005-b0448deee17b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-52b947-zjv4q\" (UID: \"7df03d21-8093-4916-a005-b0448deee17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"
Oct 30 01:01:51.218296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.218268    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl82q\" (UniqueName: \"kubernetes.io/projected/7df03d21-8093-4916-a005-b0448deee17b-kube-api-access-vl82q\") pod \"maas-api-revoke-keys-e2e-api-a-52b947-zjv4q\" (UID: \"7df03d21-8093-4916-a005-b0448deee17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"
Oct 30 01:01:51.387530 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.387468    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"
Oct 30 01:01:51.689904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:51.689874    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"]
Oct 30 01:01:51.690022 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:01:51.689986    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df03d21_8093_4916_a005_b0448deee17b.slice/crio-89029050d601bf709fb01c1d1c0547bcf7fc19830db13481019ed349cc39541c WatchSource:0}: Error finding container 89029050d601bf709fb01c1d1c0547bcf7fc19830db13481019ed349cc39541c: Status 404 returned error can't find the container with id 89029050d601bf709fb01c1d1c0547bcf7fc19830db13481019ed349cc39541c
Oct 30 01:01:52.075727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:52.075680    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q" event={"ID":"7df03d21-8093-4916-a005-b0448deee17b","Type":"ContainerStarted","Data":"3577ce76554f50e2ddf897fe0053971732505b95d0087fe5e81257859f4cdb40"}
Oct 30 01:01:52.076226 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:52.075744    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q" event={"ID":"7df03d21-8093-4916-a005-b0448deee17b","Type":"ContainerStarted","Data":"89029050d601bf709fb01c1d1c0547bcf7fc19830db13481019ed349cc39541c"}
Oct 30 01:01:52.113289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:52.113246    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q" podStartSLOduration=1.113233307 podStartE2EDuration="1.113233307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:52.110455619 +0000 UTC m=+1220.860110927" watchObservedRunningTime="2026-10-30 01:01:52.113233307 +0000 UTC m=+1220.862888647"
Oct 30 01:01:53.083277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:53.083246    4425 generic.go:356] "Generic (PLEG): container finished" podID="7df03d21-8093-4916-a005-b0448deee17b" containerID="3577ce76554f50e2ddf897fe0053971732505b95d0087fe5e81257859f4cdb40" exitCode=0
Oct 30 01:01:53.083683 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:53.083333    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q" event={"ID":"7df03d21-8093-4916-a005-b0448deee17b","Type":"ContainerDied","Data":"3577ce76554f50e2ddf897fe0053971732505b95d0087fe5e81257859f4cdb40"}
Oct 30 01:01:54.246745 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:54.246712    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"
Oct 30 01:01:54.348296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:54.348221    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7df03d21-8093-4916-a005-b0448deee17b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df03d21-8093-4916-a005-b0448deee17b-maas-api-openshift-ca\") pod \"7df03d21-8093-4916-a005-b0448deee17b\" (UID: \"7df03d21-8093-4916-a005-b0448deee17b\") "
Oct 30 01:01:54.348296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:54.348279    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7df03d21-8093-4916-a005-b0448deee17b-kube-api-access-vl82q\" (UniqueName: \"kubernetes.io/projected/7df03d21-8093-4916-a005-b0448deee17b-kube-api-access-vl82q\") pod \"7df03d21-8093-4916-a005-b0448deee17b\" (UID: \"7df03d21-8093-4916-a005-b0448deee17b\") "
Oct 30 01:01:54.348586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:54.348565    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7df03d21-8093-4916-a005-b0448deee17b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df03d21-8093-4916-a005-b0448deee17b-maas-api-xks-ca\") pod \"7df03d21-8093-4916-a005-b0448deee17b\" (UID: \"7df03d21-8093-4916-a005-b0448deee17b\") "
Oct 30 01:01:54.348657 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:54.348613    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df03d21-8093-4916-a005-b0448deee17b-maas-api-openshift-ca" pod "7df03d21-8093-4916-a005-b0448deee17b" (UID: "7df03d21-8093-4916-a005-b0448deee17b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:01:54.349145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:54.349112    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df03d21-8093-4916-a005-b0448deee17b-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:01:54.350448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:54.350418    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df03d21-8093-4916-a005-b0448deee17b-kube-api-access-vl82q" pod "7df03d21-8093-4916-a005-b0448deee17b" (UID: "7df03d21-8093-4916-a005-b0448deee17b"). InnerVolumeSpecName "kube-api-access-vl82q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:54.350541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:54.350501    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df03d21-8093-4916-a005-b0448deee17b-maas-api-xks-ca" pod "7df03d21-8093-4916-a005-b0448deee17b" (UID: "7df03d21-8093-4916-a005-b0448deee17b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:54.450816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:54.450772    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl82q\" (UniqueName: \"kubernetes.io/projected/7df03d21-8093-4916-a005-b0448deee17b-kube-api-access-vl82q\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:01:54.450816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:54.450813    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df03d21-8093-4916-a005-b0448deee17b-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:01:55.098451 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:55.098419    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"
Oct 30 01:01:55.098639 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:55.098413    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q" event={"ID":"7df03d21-8093-4916-a005-b0448deee17b","Type":"ContainerDied","Data":"89029050d601bf709fb01c1d1c0547bcf7fc19830db13481019ed349cc39541c"}
Oct 30 01:01:55.098639 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:01:55.098536    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89029050d601bf709fb01c1d1c0547bcf7fc19830db13481019ed349cc39541c"
Oct 30 01:02:06.291207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.291170    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"]
Oct 30 01:02:06.291870 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.291853    4425 cpu_manager.go:414] "containerMap: removing container" podUID="7df03d21-8093-4916-a005-b0448deee17b" containerName="revoke-keys"
Oct 30 01:02:06.291932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.291872    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7df03d21-8093-4916-a005-b0448deee17b" containerName="revoke-keys"
Oct 30 01:02:06.292004 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.291994    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7df03d21-8093-4916-a005-b0448deee17b" containerName="revoke-keys"
Oct 30 01:02:06.297062 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.297038    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"
Oct 30 01:02:06.321348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.321318    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:02:06.321491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.321372    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:02:06.321491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.321384    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:02:06.321673 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.321652    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:02:06.327055 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.327034    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"]
Oct 30 01:02:06.404696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.404665    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71736696-7d27-45c2-9768-23978dc9e3aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc74a2d0-wkj8z\" (UID: \"71736696-7d27-45c2-9768-23978dc9e3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"
Oct 30 01:02:06.404862 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.404716    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxbd6\" (UniqueName: \"kubernetes.io/projected/71736696-7d27-45c2-9768-23978dc9e3aa-kube-api-access-rxbd6\") pod \"maas-api-revoke-sub-cc74a2d0-wkj8z\" (UID: \"71736696-7d27-45c2-9768-23978dc9e3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"
Oct 30 01:02:06.405009 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.404986    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71736696-7d27-45c2-9768-23978dc9e3aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc74a2d0-wkj8z\" (UID: \"71736696-7d27-45c2-9768-23978dc9e3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"
Oct 30 01:02:06.505921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.505895    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71736696-7d27-45c2-9768-23978dc9e3aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc74a2d0-wkj8z\" (UID: \"71736696-7d27-45c2-9768-23978dc9e3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"
Oct 30 01:02:06.506086 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.505935    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbd6\" (UniqueName: \"kubernetes.io/projected/71736696-7d27-45c2-9768-23978dc9e3aa-kube-api-access-rxbd6\") pod \"maas-api-revoke-sub-cc74a2d0-wkj8z\" (UID: \"71736696-7d27-45c2-9768-23978dc9e3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"
Oct 30 01:02:06.506195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.506174    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71736696-7d27-45c2-9768-23978dc9e3aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc74a2d0-wkj8z\" (UID: \"71736696-7d27-45c2-9768-23978dc9e3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"
Oct 30 01:02:06.506763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.506740    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71736696-7d27-45c2-9768-23978dc9e3aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc74a2d0-wkj8z\" (UID: \"71736696-7d27-45c2-9768-23978dc9e3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"
Oct 30 01:02:06.508324 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.508304    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71736696-7d27-45c2-9768-23978dc9e3aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc74a2d0-wkj8z\" (UID: \"71736696-7d27-45c2-9768-23978dc9e3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"
Oct 30 01:02:06.526724 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.526701    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxbd6\" (UniqueName: \"kubernetes.io/projected/71736696-7d27-45c2-9768-23978dc9e3aa-kube-api-access-rxbd6\") pod \"maas-api-revoke-sub-cc74a2d0-wkj8z\" (UID: \"71736696-7d27-45c2-9768-23978dc9e3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"
Oct 30 01:02:06.610976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.610920    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"
Oct 30 01:02:06.915554 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:06.915526    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"]
Oct 30 01:02:06.917367 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:02:06.917338    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71736696_7d27_45c2_9768_23978dc9e3aa.slice/crio-9fdf5a9e6a3bae0dc6ab31cb6158539763c4713a9a240fb20e9af014f8fde2d0 WatchSource:0}: Error finding container 9fdf5a9e6a3bae0dc6ab31cb6158539763c4713a9a240fb20e9af014f8fde2d0: Status 404 returned error can't find the container with id 9fdf5a9e6a3bae0dc6ab31cb6158539763c4713a9a240fb20e9af014f8fde2d0
Oct 30 01:02:07.183998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:07.183967    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z" event={"ID":"71736696-7d27-45c2-9768-23978dc9e3aa","Type":"ContainerStarted","Data":"9fdf5a9e6a3bae0dc6ab31cb6158539763c4713a9a240fb20e9af014f8fde2d0"}
Oct 30 01:02:07.553782 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:02:07.553753    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:02:08.192371 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:08.192329    4425 generic.go:356] "Generic (PLEG): container finished" podID="71736696-7d27-45c2-9768-23978dc9e3aa" containerID="fd289de2cf8c2cbc92e2a09aab996ceb5c0cc80bbd0bb491658d13c4b50a01e1" exitCode=0
Oct 30 01:02:08.192554 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:08.192378    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z" event={"ID":"71736696-7d27-45c2-9768-23978dc9e3aa","Type":"ContainerDied","Data":"fd289de2cf8c2cbc92e2a09aab996ceb5c0cc80bbd0bb491658d13c4b50a01e1"}
Oct 30 01:02:09.345170 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:09.345127    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"
Oct 30 01:02:09.443189 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:09.443146    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71736696-7d27-45c2-9768-23978dc9e3aa-kube-api-access-rxbd6\" (UniqueName: \"kubernetes.io/projected/71736696-7d27-45c2-9768-23978dc9e3aa-kube-api-access-rxbd6\") pod \"71736696-7d27-45c2-9768-23978dc9e3aa\" (UID: \"71736696-7d27-45c2-9768-23978dc9e3aa\") "
Oct 30 01:02:09.443350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:09.443209    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71736696-7d27-45c2-9768-23978dc9e3aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71736696-7d27-45c2-9768-23978dc9e3aa-maas-api-xks-ca\") pod \"71736696-7d27-45c2-9768-23978dc9e3aa\" (UID: \"71736696-7d27-45c2-9768-23978dc9e3aa\") "
Oct 30 01:02:09.443350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:09.443260    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71736696-7d27-45c2-9768-23978dc9e3aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71736696-7d27-45c2-9768-23978dc9e3aa-maas-api-openshift-ca\") pod \"71736696-7d27-45c2-9768-23978dc9e3aa\" (UID: \"71736696-7d27-45c2-9768-23978dc9e3aa\") "
Oct 30 01:02:09.443629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:09.443601    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71736696-7d27-45c2-9768-23978dc9e3aa-maas-api-openshift-ca" pod "71736696-7d27-45c2-9768-23978dc9e3aa" (UID: "71736696-7d27-45c2-9768-23978dc9e3aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:09.445133 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:09.445104    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71736696-7d27-45c2-9768-23978dc9e3aa-kube-api-access-rxbd6" pod "71736696-7d27-45c2-9768-23978dc9e3aa" (UID: "71736696-7d27-45c2-9768-23978dc9e3aa"). InnerVolumeSpecName "kube-api-access-rxbd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:09.445275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:09.445186    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71736696-7d27-45c2-9768-23978dc9e3aa-maas-api-xks-ca" pod "71736696-7d27-45c2-9768-23978dc9e3aa" (UID: "71736696-7d27-45c2-9768-23978dc9e3aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:02:09.544666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:09.544636    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71736696-7d27-45c2-9768-23978dc9e3aa-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:02:09.544666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:09.544664    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxbd6\" (UniqueName: \"kubernetes.io/projected/71736696-7d27-45c2-9768-23978dc9e3aa-kube-api-access-rxbd6\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:02:09.544666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:09.544674    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71736696-7d27-45c2-9768-23978dc9e3aa-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:02:10.207135 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:10.207105    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"
Oct 30 01:02:10.207317 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:10.207105    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z" event={"ID":"71736696-7d27-45c2-9768-23978dc9e3aa","Type":"ContainerDied","Data":"9fdf5a9e6a3bae0dc6ab31cb6158539763c4713a9a240fb20e9af014f8fde2d0"}
Oct 30 01:02:10.207317 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:10.207234    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fdf5a9e6a3bae0dc6ab31cb6158539763c4713a9a240fb20e9af014f8fde2d0"
Oct 30 01:02:13.496781 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.496748    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"]
Oct 30 01:02:13.497373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.497359    4425 cpu_manager.go:414] "containerMap: removing container" podUID="71736696-7d27-45c2-9768-23978dc9e3aa" containerName="revoke-keys"
Oct 30 01:02:13.497373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.497375    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71736696-7d27-45c2-9768-23978dc9e3aa" containerName="revoke-keys"
Oct 30 01:02:13.497473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.497467    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="71736696-7d27-45c2-9768-23978dc9e3aa" containerName="revoke-keys"
Oct 30 01:02:13.501392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.501371    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"
Oct 30 01:02:13.514850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.514829    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:02:13.524584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.524552    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:02:13.524702 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.524632    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:02:13.524702 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.524681    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:02:13.530859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.530837    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"]
Oct 30 01:02:13.594650 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.594621    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a1a121-pv2fd\" (UID: \"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"
Oct 30 01:02:13.594772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.594661    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnkjx\" (UniqueName: \"kubernetes.io/projected/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-kube-api-access-lnkjx\") pod \"maas-api-revoke-sub-c2a1a121-pv2fd\" (UID: \"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"
Oct 30 01:02:13.594907 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.594885    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a1a121-pv2fd\" (UID: \"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"
Oct 30 01:02:13.695752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.695728    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a1a121-pv2fd\" (UID: \"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"
Oct 30 01:02:13.695851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.695764    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnkjx\" (UniqueName: \"kubernetes.io/projected/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-kube-api-access-lnkjx\") pod \"maas-api-revoke-sub-c2a1a121-pv2fd\" (UID: \"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"
Oct 30 01:02:13.695919 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.695910    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a1a121-pv2fd\" (UID: \"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"
Oct 30 01:02:13.696542 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.696519    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a1a121-pv2fd\" (UID: \"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"
Oct 30 01:02:13.698057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.698030    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a1a121-pv2fd\" (UID: \"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"
Oct 30 01:02:13.716267 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.716247    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnkjx\" (UniqueName: \"kubernetes.io/projected/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-kube-api-access-lnkjx\") pod \"maas-api-revoke-sub-c2a1a121-pv2fd\" (UID: \"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"
Oct 30 01:02:13.814610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:13.814583    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"
Oct 30 01:02:14.129392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:14.129364    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"]
Oct 30 01:02:14.130031 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:02:14.130006    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd68b0f_cf8f_44bd_8f79_2e247fc43ec2.slice/crio-c1b4e016a7825a89d011c190b2aa889d7d7078b270005c72574554e13d9665a5 WatchSource:0}: Error finding container c1b4e016a7825a89d011c190b2aa889d7d7078b270005c72574554e13d9665a5: Status 404 returned error can't find the container with id c1b4e016a7825a89d011c190b2aa889d7d7078b270005c72574554e13d9665a5
Oct 30 01:02:14.239162 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:14.239131    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd" event={"ID":"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2","Type":"ContainerStarted","Data":"c1b4e016a7825a89d011c190b2aa889d7d7078b270005c72574554e13d9665a5"}
Oct 30 01:02:15.247275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:15.247193    4425 generic.go:356] "Generic (PLEG): container finished" podID="9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2" containerID="2dbf7d545fda21d84d2363e599a99b89fc8a5d5907d4cb1baafaba83d5a84b95" exitCode=0
Oct 30 01:02:15.247629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:15.247280    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd" event={"ID":"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2","Type":"ContainerDied","Data":"2dbf7d545fda21d84d2363e599a99b89fc8a5d5907d4cb1baafaba83d5a84b95"}
Oct 30 01:02:16.439124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:16.439105    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"
Oct 30 01:02:16.536145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:16.536116    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-kube-api-access-lnkjx\" (UniqueName: \"kubernetes.io/projected/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-kube-api-access-lnkjx\") pod \"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2\" (UID: \"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2\") "
Oct 30 01:02:16.536345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:16.536189    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-maas-api-openshift-ca\") pod \"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2\" (UID: \"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2\") "
Oct 30 01:02:16.536345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:16.536288    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-maas-api-xks-ca\") pod \"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2\" (UID: \"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2\") "
Oct 30 01:02:16.536549 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:16.536522    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-maas-api-openshift-ca" pod "9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2" (UID: "9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:16.536968 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:16.536948    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:02:16.538343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:16.538318    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-kube-api-access-lnkjx" pod "9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2" (UID: "9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2"). InnerVolumeSpecName "kube-api-access-lnkjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:16.538430 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:16.538342    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-maas-api-xks-ca" pod "9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2" (UID: "9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:02:16.637701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:16.637678    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnkjx\" (UniqueName: \"kubernetes.io/projected/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-kube-api-access-lnkjx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:02:16.637701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:16.637699    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:02:17.263340 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:17.263307    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd" event={"ID":"9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2","Type":"ContainerDied","Data":"c1b4e016a7825a89d011c190b2aa889d7d7078b270005c72574554e13d9665a5"}
Oct 30 01:02:17.263518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:17.263350    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"
Oct 30 01:02:17.263518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:17.263356    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1b4e016a7825a89d011c190b2aa889d7d7078b270005c72574554e13d9665a5"
Oct 30 01:02:25.753311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:25.753248    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"]
Oct 30 01:02:25.753959 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:25.753854    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2" containerName="revoke-keys"
Oct 30 01:02:25.753959 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:25.753867    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2" containerName="revoke-keys"
Oct 30 01:02:25.754082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:25.753973    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2" containerName="revoke-keys"
Oct 30 01:02:25.757624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:25.757600    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"
Oct 30 01:02:25.775180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:25.775130    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"]
Oct 30 01:02:25.780150 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:25.780125    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9a8d85\""
Oct 30 01:02:25.780290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:25.780211    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9a8d85-dockercfg-5cvt6\""
Oct 30 01:02:25.839178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:25.838607    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"]
Oct 30 01:02:25.878031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:25.874440    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"
Oct 30 01:02:25.878031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:25.876014    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfzbk\" (UniqueName: \"kubernetes.io/projected/8368346f-c6ee-4efa-bf21-31affa8e8b35-kube-api-access-zfzbk\") pod \"payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7\" (UID: \"8368346f-c6ee-4efa-bf21-31affa8e8b35\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"
Oct 30 01:02:25.898211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:25.898143    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"]
Oct 30 01:02:25.932746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:25.932710    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8368346f-c6ee-4efa-bf21-31affa8e8b35-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7\" (UID: \"8368346f-c6ee-4efa-bf21-31affa8e8b35\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"
Oct 30 01:02:26.034362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:26.034330    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v5sd\" (UniqueName: \"kubernetes.io/projected/bcb580cf-e9bd-4098-9ad4-6c72b709f5d4-kube-api-access-7v5sd\") pod \"payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz\" (UID: \"bcb580cf-e9bd-4098-9ad4-6c72b709f5d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"
Oct 30 01:02:26.034536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:26.034476    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8368346f-c6ee-4efa-bf21-31affa8e8b35-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7\" (UID: \"8368346f-c6ee-4efa-bf21-31affa8e8b35\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"
Oct 30 01:02:26.034536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:26.034514    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcb580cf-e9bd-4098-9ad4-6c72b709f5d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz\" (UID: \"bcb580cf-e9bd-4098-9ad4-6c72b709f5d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"
Oct 30 01:02:26.034643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:26.034620    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzbk\" (UniqueName: \"kubernetes.io/projected/8368346f-c6ee-4efa-bf21-31affa8e8b35-kube-api-access-zfzbk\") pod \"payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7\" (UID: \"8368346f-c6ee-4efa-bf21-31affa8e8b35\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"
Oct 30 01:02:26.035262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:26.035237    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8368346f-c6ee-4efa-bf21-31affa8e8b35-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7\" (UID: \"8368346f-c6ee-4efa-bf21-31affa8e8b35\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"
Oct 30 01:02:26.055901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:26.055876    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfzbk\" (UniqueName: \"kubernetes.io/projected/8368346f-c6ee-4efa-bf21-31affa8e8b35-kube-api-access-zfzbk\") pod \"payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7\" (UID: \"8368346f-c6ee-4efa-bf21-31affa8e8b35\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"
Oct 30 01:02:26.071081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:26.071045    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"
Oct 30 01:02:26.138799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:26.135946    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v5sd\" (UniqueName: \"kubernetes.io/projected/bcb580cf-e9bd-4098-9ad4-6c72b709f5d4-kube-api-access-7v5sd\") pod \"payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz\" (UID: \"bcb580cf-e9bd-4098-9ad4-6c72b709f5d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"
Oct 30 01:02:26.138799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:26.136123    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcb580cf-e9bd-4098-9ad4-6c72b709f5d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz\" (UID: \"bcb580cf-e9bd-4098-9ad4-6c72b709f5d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"
Oct 30 01:02:26.138799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:26.137144    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcb580cf-e9bd-4098-9ad4-6c72b709f5d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz\" (UID: \"bcb580cf-e9bd-4098-9ad4-6c72b709f5d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"
Oct 30 01:02:26.158710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:26.158635    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v5sd\" (UniqueName: \"kubernetes.io/projected/bcb580cf-e9bd-4098-9ad4-6c72b709f5d4-kube-api-access-7v5sd\") pod \"payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz\" (UID: \"bcb580cf-e9bd-4098-9ad4-6c72b709f5d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"
Oct 30 01:02:26.304107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:26.304022    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"
Oct 30 01:02:26.389017 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:02:26.388980    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8368346f_c6ee_4efa_bf21_31affa8e8b35.slice/crio-00e8a408f35d3ca79d48a0f69989ed0a4961fac616ad33d7652e862e81e33ee8 WatchSource:0}: Error finding container 00e8a408f35d3ca79d48a0f69989ed0a4961fac616ad33d7652e862e81e33ee8: Status 404 returned error can't find the container with id 00e8a408f35d3ca79d48a0f69989ed0a4961fac616ad33d7652e862e81e33ee8
Oct 30 01:02:26.393877 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:26.393423    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"]
Oct 30 01:02:26.663227 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:26.660829    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"]
Oct 30 01:02:26.677238 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:02:26.677207    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb580cf_e9bd_4098_9ad4_6c72b709f5d4.slice/crio-f22ebf246d4277d08cb6fda16f42de4a6e639504eba99e83591d9ae1d8518de9 WatchSource:0}: Error finding container f22ebf246d4277d08cb6fda16f42de4a6e639504eba99e83591d9ae1d8518de9: Status 404 returned error can't find the container with id f22ebf246d4277d08cb6fda16f42de4a6e639504eba99e83591d9ae1d8518de9
Oct 30 01:02:27.158357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:27.158315    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"]
Oct 30 01:02:27.162937 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:27.162900    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-4rs24"]
Oct 30 01:02:27.332778 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:27.332737    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz" event={"ID":"bcb580cf-e9bd-4098-9ad4-6c72b709f5d4","Type":"ContainerStarted","Data":"96b5c689544744dbc4bc65aff1b653e4f1738f77e88c4ad6b541caeeb39723b2"}
Oct 30 01:02:27.333001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:27.332838    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz" event={"ID":"bcb580cf-e9bd-4098-9ad4-6c72b709f5d4","Type":"ContainerStarted","Data":"f22ebf246d4277d08cb6fda16f42de4a6e639504eba99e83591d9ae1d8518de9"}
Oct 30 01:02:27.333001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:27.332981    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"
Oct 30 01:02:27.334481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:27.334459    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7" event={"ID":"8368346f-c6ee-4efa-bf21-31affa8e8b35","Type":"ContainerStarted","Data":"04885a0b12da457cd99c53c27c6366af7e5351546094b52c1c90b69acb360fb6"}
Oct 30 01:02:27.334572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:27.334495    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7" event={"ID":"8368346f-c6ee-4efa-bf21-31affa8e8b35","Type":"ContainerStarted","Data":"00e8a408f35d3ca79d48a0f69989ed0a4961fac616ad33d7652e862e81e33ee8"}
Oct 30 01:02:27.334572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:27.334531    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"
Oct 30 01:02:27.376035 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:27.375989    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz" podStartSLOduration=2.3759756579999998 podStartE2EDuration="2.375975658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:27.371438929 +0000 UTC m=+1256.121094238" watchObservedRunningTime="2026-10-30 01:02:27.375975658 +0000 UTC m=+1256.125630997"
Oct 30 01:02:27.408516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:27.408422    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7" podStartSLOduration=2.408408511 podStartE2EDuration="2.408408511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:27.405983487 +0000 UTC m=+1256.155638797" watchObservedRunningTime="2026-10-30 01:02:27.408408511 +0000 UTC m=+1256.158063820"
Oct 30 01:02:27.425612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:27.425578    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aac47da-0db9-4277-845c-c167c6244317" path="/var/lib/kubelet/pods/6aac47da-0db9-4277-845c-c167c6244317/volumes"
Oct 30 01:02:28.490861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:28.490831    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:02:31.934571 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:31.934538    4425 scope.go:122] "RemoveContainer" containerID="69bc971b8277b0c9ba4856752eeda5eccdccd5be51e176867002322c67f2a53e"
Oct 30 01:02:37.563597 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:02:37.563554    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:02:38.299213 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:38.299139    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"]
Oct 30 01:02:38.307382 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:38.307352    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52b947-zjv4q"]
Oct 30 01:02:38.344880 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:38.344846    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"
Oct 30 01:02:38.345283 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:38.345232    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"
Oct 30 01:02:39.422924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:39.422888    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7df03d21-8093-4916-a005-b0448deee17b" path="/var/lib/kubelet/pods/7df03d21-8093-4916-a005-b0448deee17b/volumes"
Oct 30 01:02:40.554030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.554000    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"]
Oct 30 01:02:40.559929 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.559911    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"
Oct 30 01:02:40.574364 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.574344    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9b2c3f\""
Oct 30 01:02:40.587984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.587961    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"]
Oct 30 01:02:40.634351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.634321    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9b2c3f-dockercfg-kdt2n\""
Oct 30 01:02:40.665679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.665646    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"]
Oct 30 01:02:40.671704 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.671674    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gph6k\" (UniqueName: \"kubernetes.io/projected/c6580527-dd10-4e58-9330-58460660831a-kube-api-access-gph6k\") pod \"payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47\" (UID: \"c6580527-dd10-4e58-9330-58460660831a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"
Oct 30 01:02:40.672935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.672909    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6580527-dd10-4e58-9330-58460660831a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47\" (UID: \"c6580527-dd10-4e58-9330-58460660831a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"
Oct 30 01:02:40.677515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.677495    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"
Oct 30 01:02:40.705917 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.705122    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"]
Oct 30 01:02:40.776585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.776555    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gph6k\" (UniqueName: \"kubernetes.io/projected/c6580527-dd10-4e58-9330-58460660831a-kube-api-access-gph6k\") pod \"payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47\" (UID: \"c6580527-dd10-4e58-9330-58460660831a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"
Oct 30 01:02:40.776781 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.776655    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlt9b\" (UniqueName: \"kubernetes.io/projected/e23c4d7c-5787-4c5b-ad47-9a6715c1d93d-kube-api-access-qlt9b\") pod \"payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2\" (UID: \"e23c4d7c-5787-4c5b-ad47-9a6715c1d93d\") " pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"
Oct 30 01:02:40.776781 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.776698    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6580527-dd10-4e58-9330-58460660831a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47\" (UID: \"c6580527-dd10-4e58-9330-58460660831a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"
Oct 30 01:02:40.776964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.776836    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e23c4d7c-5787-4c5b-ad47-9a6715c1d93d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2\" (UID: \"e23c4d7c-5787-4c5b-ad47-9a6715c1d93d\") " pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"
Oct 30 01:02:40.777467 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.777445    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6580527-dd10-4e58-9330-58460660831a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47\" (UID: \"c6580527-dd10-4e58-9330-58460660831a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"
Oct 30 01:02:40.797286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.797258    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gph6k\" (UniqueName: \"kubernetes.io/projected/c6580527-dd10-4e58-9330-58460660831a-kube-api-access-gph6k\") pod \"payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47\" (UID: \"c6580527-dd10-4e58-9330-58460660831a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"
Oct 30 01:02:40.873934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.873823    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"
Oct 30 01:02:40.878668 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.878636    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlt9b\" (UniqueName: \"kubernetes.io/projected/e23c4d7c-5787-4c5b-ad47-9a6715c1d93d-kube-api-access-qlt9b\") pod \"payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2\" (UID: \"e23c4d7c-5787-4c5b-ad47-9a6715c1d93d\") " pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"
Oct 30 01:02:40.878794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.878746    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e23c4d7c-5787-4c5b-ad47-9a6715c1d93d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2\" (UID: \"e23c4d7c-5787-4c5b-ad47-9a6715c1d93d\") " pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"
Oct 30 01:02:40.879450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.879432    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e23c4d7c-5787-4c5b-ad47-9a6715c1d93d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2\" (UID: \"e23c4d7c-5787-4c5b-ad47-9a6715c1d93d\") " pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"
Oct 30 01:02:40.899091 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:40.899069    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlt9b\" (UniqueName: \"kubernetes.io/projected/e23c4d7c-5787-4c5b-ad47-9a6715c1d93d-kube-api-access-qlt9b\") pod \"payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2\" (UID: \"e23c4d7c-5787-4c5b-ad47-9a6715c1d93d\") " pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"
Oct 30 01:02:41.039358 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:41.039329    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"
Oct 30 01:02:41.185818 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:41.185788    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"]
Oct 30 01:02:41.187566 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:02:41.187538    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6580527_dd10_4e58_9330_58460660831a.slice/crio-3104ec02f84c63bf878154ea236c929934a5e5eb9f2aceecd687d29da1560ace WatchSource:0}: Error finding container 3104ec02f84c63bf878154ea236c929934a5e5eb9f2aceecd687d29da1560ace: Status 404 returned error can't find the container with id 3104ec02f84c63bf878154ea236c929934a5e5eb9f2aceecd687d29da1560ace
Oct 30 01:02:41.348405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:41.348377    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"]
Oct 30 01:02:41.351491 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:02:41.351454    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23c4d7c_5787_4c5b_ad47_9a6715c1d93d.slice/crio-2c9461df60e229228f345e567ab6d77d96340fd90cfbe6c69706d9acaf517e11 WatchSource:0}: Error finding container 2c9461df60e229228f345e567ab6d77d96340fd90cfbe6c69706d9acaf517e11: Status 404 returned error can't find the container with id 2c9461df60e229228f345e567ab6d77d96340fd90cfbe6c69706d9acaf517e11
Oct 30 01:02:41.447341 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:41.447272    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47" event={"ID":"c6580527-dd10-4e58-9330-58460660831a","Type":"ContainerStarted","Data":"83872a30949313ddd9ac74031bb7051c0fc80984bb2028ccc126e2f67bdad437"}
Oct 30 01:02:41.447341 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:41.447320    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47" event={"ID":"c6580527-dd10-4e58-9330-58460660831a","Type":"ContainerStarted","Data":"3104ec02f84c63bf878154ea236c929934a5e5eb9f2aceecd687d29da1560ace"}
Oct 30 01:02:41.447548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:41.447360    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"
Oct 30 01:02:41.448944 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:41.448922    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2" event={"ID":"e23c4d7c-5787-4c5b-ad47-9a6715c1d93d","Type":"ContainerStarted","Data":"8b055146cffca4b70ef0856379a6d32800bdb01f5f83a27dd7f08887e02f1c86"}
Oct 30 01:02:41.449064 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:41.448956    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2" event={"ID":"e23c4d7c-5787-4c5b-ad47-9a6715c1d93d","Type":"ContainerStarted","Data":"2c9461df60e229228f345e567ab6d77d96340fd90cfbe6c69706d9acaf517e11"}
Oct 30 01:02:41.449064 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:41.448995    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"
Oct 30 01:02:41.485963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:41.485915    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47" podStartSLOduration=1.485902819 podStartE2EDuration="1.485902819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:41.482564736 +0000 UTC m=+1270.232220045" watchObservedRunningTime="2026-10-30 01:02:41.485902819 +0000 UTC m=+1270.235558127"
Oct 30 01:02:41.521558 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:41.521508    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2" podStartSLOduration=1.521494894 podStartE2EDuration="1.521494894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:02:41.518116028 +0000 UTC m=+1270.267771336" watchObservedRunningTime="2026-10-30 01:02:41.521494894 +0000 UTC m=+1270.271150203"
Oct 30 01:02:52.489185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:52.487381    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"
Oct 30 01:02:52.489185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:52.487974    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"
Oct 30 01:02:54.557179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:54.554400    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ltz8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:02:54.557179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:54.554475    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j" podUID="dae18a2f-da25-474a-af36-6be757b43311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:54.883893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:54.883795    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8z8gn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:02:54.883893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:54.883866    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn" podUID="bfee5645-34a5-4aec-a646-1742da2db219" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:02:56.747141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:56.747110    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"]
Oct 30 01:02:56.751270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:56.751249    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"
Oct 30 01:02:56.767446 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:56.767417    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:02:56.775139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:56.775111    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:02:56.779437 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:56.779410    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:02:56.779571 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:56.779550    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:02:56.780308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:56.780284    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"]
Oct 30 01:02:56.843200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:56.843146    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b2lf\" (UniqueName: \"kubernetes.io/projected/d47f37b6-499b-446d-b417-f087d97b372d-kube-api-access-7b2lf\") pod \"maas-api-revoke-keys-e2e-api-b-153746-mv52c\" (UID: \"d47f37b6-499b-446d-b417-f087d97b372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"
Oct 30 01:02:56.843401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:56.843382    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47f37b6-499b-446d-b417-f087d97b372d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-153746-mv52c\" (UID: \"d47f37b6-499b-446d-b417-f087d97b372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"
Oct 30 01:02:56.843453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:56.843441    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47f37b6-499b-446d-b417-f087d97b372d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-153746-mv52c\" (UID: \"d47f37b6-499b-446d-b417-f087d97b372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"
Oct 30 01:02:56.944606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:56.944512    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47f37b6-499b-446d-b417-f087d97b372d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-153746-mv52c\" (UID: \"d47f37b6-499b-446d-b417-f087d97b372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"
Oct 30 01:02:56.944606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:56.944597    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47f37b6-499b-446d-b417-f087d97b372d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-153746-mv52c\" (UID: \"d47f37b6-499b-446d-b417-f087d97b372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"
Oct 30 01:02:56.944881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:56.944679    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b2lf\" (UniqueName: \"kubernetes.io/projected/d47f37b6-499b-446d-b417-f087d97b372d-kube-api-access-7b2lf\") pod \"maas-api-revoke-keys-e2e-api-b-153746-mv52c\" (UID: \"d47f37b6-499b-446d-b417-f087d97b372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"
Oct 30 01:02:56.945808 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:56.945774    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47f37b6-499b-446d-b417-f087d97b372d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-153746-mv52c\" (UID: \"d47f37b6-499b-446d-b417-f087d97b372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"
Oct 30 01:02:56.947667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:56.947631    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47f37b6-499b-446d-b417-f087d97b372d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-153746-mv52c\" (UID: \"d47f37b6-499b-446d-b417-f087d97b372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"
Oct 30 01:02:56.965357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:56.965333    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b2lf\" (UniqueName: \"kubernetes.io/projected/d47f37b6-499b-446d-b417-f087d97b372d-kube-api-access-7b2lf\") pod \"maas-api-revoke-keys-e2e-api-b-153746-mv52c\" (UID: \"d47f37b6-499b-446d-b417-f087d97b372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"
Oct 30 01:02:57.072571 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:57.072542    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"
Oct 30 01:02:57.381405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:57.381377    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"]
Oct 30 01:02:57.382237 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:02:57.382204    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47f37b6_499b_446d_b417_f087d97b372d.slice/crio-75b686f16714f3c2bfe964ce0f996aa0f78394cedd07e3ccb4ace68d25bafbca WatchSource:0}: Error finding container 75b686f16714f3c2bfe964ce0f996aa0f78394cedd07e3ccb4ace68d25bafbca: Status 404 returned error can't find the container with id 75b686f16714f3c2bfe964ce0f996aa0f78394cedd07e3ccb4ace68d25bafbca
Oct 30 01:02:57.576952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:57.576916    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c" event={"ID":"d47f37b6-499b-446d-b417-f087d97b372d","Type":"ContainerStarted","Data":"75b686f16714f3c2bfe964ce0f996aa0f78394cedd07e3ccb4ace68d25bafbca"}
Oct 30 01:02:58.587795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:58.587700    4425 generic.go:356] "Generic (PLEG): container finished" podID="d47f37b6-499b-446d-b417-f087d97b372d" containerID="5109c1e4e5264d48048d711267f17436d1606f1844eb0bc6de27e3fc590b4167" exitCode=0
Oct 30 01:02:58.588271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:58.587893    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c" event={"ID":"d47f37b6-499b-446d-b417-f087d97b372d","Type":"ContainerDied","Data":"5109c1e4e5264d48048d711267f17436d1606f1844eb0bc6de27e3fc590b4167"}
Oct 30 01:02:59.750076 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:59.750046    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"
Oct 30 01:02:59.883092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:59.883025    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d47f37b6-499b-446d-b417-f087d97b372d-kube-api-access-7b2lf\" (UniqueName: \"kubernetes.io/projected/d47f37b6-499b-446d-b417-f087d97b372d-kube-api-access-7b2lf\") pod \"d47f37b6-499b-446d-b417-f087d97b372d\" (UID: \"d47f37b6-499b-446d-b417-f087d97b372d\") "
Oct 30 01:02:59.883248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:59.883094    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d47f37b6-499b-446d-b417-f087d97b372d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47f37b6-499b-446d-b417-f087d97b372d-maas-api-xks-ca\") pod \"d47f37b6-499b-446d-b417-f087d97b372d\" (UID: \"d47f37b6-499b-446d-b417-f087d97b372d\") "
Oct 30 01:02:59.883248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:59.883186    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d47f37b6-499b-446d-b417-f087d97b372d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47f37b6-499b-446d-b417-f087d97b372d-maas-api-openshift-ca\") pod \"d47f37b6-499b-446d-b417-f087d97b372d\" (UID: \"d47f37b6-499b-446d-b417-f087d97b372d\") "
Oct 30 01:02:59.883558 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:59.883525    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d47f37b6-499b-446d-b417-f087d97b372d-maas-api-openshift-ca" pod "d47f37b6-499b-446d-b417-f087d97b372d" (UID: "d47f37b6-499b-446d-b417-f087d97b372d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:59.883690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:59.883673    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d47f37b6-499b-446d-b417-f087d97b372d-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:02:59.885281 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:59.885252    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47f37b6-499b-446d-b417-f087d97b372d-kube-api-access-7b2lf" pod "d47f37b6-499b-446d-b417-f087d97b372d" (UID: "d47f37b6-499b-446d-b417-f087d97b372d"). InnerVolumeSpecName "kube-api-access-7b2lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:59.885281 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:59.885268    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47f37b6-499b-446d-b417-f087d97b372d-maas-api-xks-ca" pod "d47f37b6-499b-446d-b417-f087d97b372d" (UID: "d47f37b6-499b-446d-b417-f087d97b372d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:02:59.984890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:59.984868    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d47f37b6-499b-446d-b417-f087d97b372d-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:02:59.984890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:02:59.984890    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b2lf\" (UniqueName: \"kubernetes.io/projected/d47f37b6-499b-446d-b417-f087d97b372d-kube-api-access-7b2lf\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:03:00.602315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:00.602288    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"
Oct 30 01:03:00.602493 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:00.602283    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c" event={"ID":"d47f37b6-499b-446d-b417-f087d97b372d","Type":"ContainerDied","Data":"75b686f16714f3c2bfe964ce0f996aa0f78394cedd07e3ccb4ace68d25bafbca"}
Oct 30 01:03:00.602493 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:00.602417    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75b686f16714f3c2bfe964ce0f996aa0f78394cedd07e3ccb4ace68d25bafbca"
Oct 30 01:03:04.781099 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:04.781063    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"]
Oct 30 01:03:04.781755 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:04.781734    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d47f37b6-499b-446d-b417-f087d97b372d" containerName="revoke-keys"
Oct 30 01:03:04.781849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:04.781757    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d47f37b6-499b-446d-b417-f087d97b372d" containerName="revoke-keys"
Oct 30 01:03:04.781912 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:04.781902    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d47f37b6-499b-446d-b417-f087d97b372d" containerName="revoke-keys"
Oct 30 01:03:04.785615 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:04.785596    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:04.812945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:04.812918    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"]
Oct 30 01:03:04.886493 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:04.886463    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9a8d85-openshift-default-dockercfg-x8rls\""
Oct 30 01:03:04.948875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:04.948843    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istiod-ca-cert\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:04.948998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:04.948903    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-token\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:04.948998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:04.948952    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shlpj\" (UniqueName: \"kubernetes.io/projected/4fa39d8a-a4c2-4e0b-b504-feb83231a183-kube-api-access-shlpj\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:04.949121 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:04.949005    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-data\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:04.949121 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:04.949045    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-workload-socket\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:04.949259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:04.949187    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-envoy\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:04.949259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:04.949237    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-podinfo\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:04.949332 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:04.949285    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-credential-socket\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:04.949368 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:04.949359    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-workload-certs\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.050112 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.050041    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-envoy\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.050112 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.050100    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-podinfo\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.050317 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.050258    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-credential-socket\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.050378 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.050323    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-workload-certs\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.050437 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.050402    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istiod-ca-cert\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.050495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.050461    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-token\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.050495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.050488    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shlpj\" (UniqueName: \"kubernetes.io/projected/4fa39d8a-a4c2-4e0b-b504-feb83231a183-kube-api-access-shlpj\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.050612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.050525    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-data\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.050612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.050575    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-workload-socket\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.050715 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.050630    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-credential-socket\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.051071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.051045    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-workload-socket\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.051259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.051227    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-data\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.051560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.051537    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istiod-ca-cert\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.051835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.051817    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-workload-certs\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.052784 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.052760    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-envoy\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.053088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.053067    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-podinfo\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.071236 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.071211    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-token\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.071539 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.071516    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shlpj\" (UniqueName: \"kubernetes.io/projected/4fa39d8a-a4c2-4e0b-b504-feb83231a183-kube-api-access-shlpj\") pod \"e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") " pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.100874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.100849    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:05.423807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.423778    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"]
Oct 30 01:03:05.424224 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:03:05.424195    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa39d8a_a4c2_4e0b_b504_feb83231a183.slice/crio-32d10e780339699754760e5af9f964f600b73bd46a5b4ce1834aec2a16e7a834 WatchSource:0}: Error finding container 32d10e780339699754760e5af9f964f600b73bd46a5b4ce1834aec2a16e7a834: Status 404 returned error can't find the container with id 32d10e780339699754760e5af9f964f600b73bd46a5b4ce1834aec2a16e7a834
Oct 30 01:03:05.426517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.426485    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:03:05.426604 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.426546    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:03:05.426604 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.426575    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:03:05.639328 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.639297    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp" event={"ID":"4fa39d8a-a4c2-4e0b-b504-feb83231a183","Type":"ContainerStarted","Data":"eccfc8ad1888fb4d216db977fb250443ce9385d5aa13e349d39a816877b0d4ce"}
Oct 30 01:03:05.639508 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.639351    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp" event={"ID":"4fa39d8a-a4c2-4e0b-b504-feb83231a183","Type":"ContainerStarted","Data":"32d10e780339699754760e5af9f964f600b73bd46a5b4ce1834aec2a16e7a834"}
Oct 30 01:03:05.683716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:05.683616    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp" podStartSLOduration=1.6835961529999999 podStartE2EDuration="1.683596153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:05.679366021 +0000 UTC m=+1294.429021329" watchObservedRunningTime="2026-10-30 01:03:05.683596153 +0000 UTC m=+1294.433251462"
Oct 30 01:03:06.101946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:06.101188    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:06.102487 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:06.102465    4425 patch_prober.go:29] interesting pod/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 01:03:06.102567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:06.102512    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp" podUID="4fa39d8a-a4c2-4e0b-b504-feb83231a183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 01:03:07.101484 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:07.101448    4425 patch_prober.go:29] interesting pod/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 01:03:07.101653 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:07.101507    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp" podUID="4fa39d8a-a4c2-4e0b-b504-feb83231a183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 01:03:07.189169 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:07.189123    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"]
Oct 30 01:03:07.223417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:07.223383    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"]
Oct 30 01:03:07.581098 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:03:07.580918    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:03:08.101407 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:08.101378    4425 patch_prober.go:29] interesting pod/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 01:03:08.101585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:08.101432    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp" podUID="4fa39d8a-a4c2-4e0b-b504-feb83231a183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 01:03:09.101627 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:09.101589    4425 patch_prober.go:29] interesting pod/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 01:03:09.102095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:09.101654    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp" podUID="4fa39d8a-a4c2-4e0b-b504-feb83231a183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 01:03:09.910053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:09.910018    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"]
Oct 30 01:03:09.914415 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:09.914395    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"
Oct 30 01:03:09.930091 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:09.930075    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8363a40b\""
Oct 30 01:03:09.931978 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:09.931957    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f3634c9-1768-46e5-9979-9738b4486e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24\" (UID: \"9f3634c9-1768-46e5-9979-9738b4486e10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"
Oct 30 01:03:09.932104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:09.932017    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fn47\" (UniqueName: \"kubernetes.io/projected/9f3634c9-1768-46e5-9979-9738b4486e10-kube-api-access-8fn47\") pod \"payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24\" (UID: \"9f3634c9-1768-46e5-9979-9738b4486e10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"
Oct 30 01:03:09.946198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:09.946175    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"]
Oct 30 01:03:10.136842 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.136645    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8363a40b-dockercfg-hnbp6\""
Oct 30 01:03:10.178630 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.178486    4425 patch_prober.go:29] interesting pod/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 01:03:10.178901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.178838    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp" podUID="4fa39d8a-a4c2-4e0b-b504-feb83231a183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 01:03:10.237752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.237720    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"]
Oct 30 01:03:10.252178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.252033    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f3634c9-1768-46e5-9979-9738b4486e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24\" (UID: \"9f3634c9-1768-46e5-9979-9738b4486e10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"
Oct 30 01:03:10.252178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.252110    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fn47\" (UniqueName: \"kubernetes.io/projected/9f3634c9-1768-46e5-9979-9738b4486e10-kube-api-access-8fn47\") pod \"payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24\" (UID: \"9f3634c9-1768-46e5-9979-9738b4486e10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"
Oct 30 01:03:10.253992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.253965    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f3634c9-1768-46e5-9979-9738b4486e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24\" (UID: \"9f3634c9-1768-46e5-9979-9738b4486e10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"
Oct 30 01:03:10.274746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.272344    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"]
Oct 30 01:03:10.274746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.272471    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"
Oct 30 01:03:10.297174 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.293791    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fn47\" (UniqueName: \"kubernetes.io/projected/9f3634c9-1768-46e5-9979-9738b4486e10-kube-api-access-8fn47\") pod \"payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24\" (UID: \"9f3634c9-1768-46e5-9979-9738b4486e10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"
Oct 30 01:03:10.355016 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.354981    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w6bx\" (UniqueName: \"kubernetes.io/projected/c963677e-9e3a-4708-be7d-0fa8202a70ef-kube-api-access-6w6bx\") pod \"payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq\" (UID: \"c963677e-9e3a-4708-be7d-0fa8202a70ef\") " pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"
Oct 30 01:03:10.368110 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.355343    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c963677e-9e3a-4708-be7d-0fa8202a70ef-plugins-config-volume\") pod \"payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq\" (UID: \"c963677e-9e3a-4708-be7d-0fa8202a70ef\") " pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"
Oct 30 01:03:10.460230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.459135    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w6bx\" (UniqueName: \"kubernetes.io/projected/c963677e-9e3a-4708-be7d-0fa8202a70ef-kube-api-access-6w6bx\") pod \"payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq\" (UID: \"c963677e-9e3a-4708-be7d-0fa8202a70ef\") " pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"
Oct 30 01:03:10.460230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.459267    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c963677e-9e3a-4708-be7d-0fa8202a70ef-plugins-config-volume\") pod \"payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq\" (UID: \"c963677e-9e3a-4708-be7d-0fa8202a70ef\") " pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"
Oct 30 01:03:10.460230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.460215    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c963677e-9e3a-4708-be7d-0fa8202a70ef-plugins-config-volume\") pod \"payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq\" (UID: \"c963677e-9e3a-4708-be7d-0fa8202a70ef\") " pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"
Oct 30 01:03:10.489993 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.489950    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w6bx\" (UniqueName: \"kubernetes.io/projected/c963677e-9e3a-4708-be7d-0fa8202a70ef-kube-api-access-6w6bx\") pod \"payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq\" (UID: \"c963677e-9e3a-4708-be7d-0fa8202a70ef\") " pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"
Oct 30 01:03:10.557189 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.556589    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"
Oct 30 01:03:10.630529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.630496    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"
Oct 30 01:03:10.795909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.795874    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"]
Oct 30 01:03:10.803504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.802577    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"
Oct 30 01:03:10.817289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.817260    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:03:10.817289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.817260    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:03:10.817506 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.817410    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:03:10.817771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.817748    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:03:10.825265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.825245    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"]
Oct 30 01:03:10.921245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.908657    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2\" (UID: \"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"
Oct 30 01:03:10.921245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.908779    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2mlg\" (UniqueName: \"kubernetes.io/projected/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-kube-api-access-m2mlg\") pod \"maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2\" (UID: \"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"
Oct 30 01:03:10.921245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:10.908829    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2\" (UID: \"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"
Oct 30 01:03:11.015466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.013793    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2mlg\" (UniqueName: \"kubernetes.io/projected/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-kube-api-access-m2mlg\") pod \"maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2\" (UID: \"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"
Oct 30 01:03:11.015466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.013859    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2\" (UID: \"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"
Oct 30 01:03:11.015466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.013981    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2\" (UID: \"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"
Oct 30 01:03:11.015466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.015411    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2\" (UID: \"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"
Oct 30 01:03:11.018370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.018313    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2\" (UID: \"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"
Oct 30 01:03:11.038637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.036149    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2mlg\" (UniqueName: \"kubernetes.io/projected/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-kube-api-access-m2mlg\") pod \"maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2\" (UID: \"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"
Oct 30 01:03:11.111925 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.108531    4425 patch_prober.go:29] interesting pod/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 01:03:11.111925 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.108596    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp" podUID="4fa39d8a-a4c2-4e0b-b504-feb83231a183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 01:03:11.124373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.124319    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"
Oct 30 01:03:11.192262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.191081    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"]
Oct 30 01:03:11.196860 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:03:11.196773    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc963677e_9e3a_4708_be7d_0fa8202a70ef.slice/crio-7a0e4a5870b01d7222c1d795ec4c6e19ffc1256ca5ec9a32b6f3d60f9b5ae5c5 WatchSource:0}: Error finding container 7a0e4a5870b01d7222c1d795ec4c6e19ffc1256ca5ec9a32b6f3d60f9b5ae5c5: Status 404 returned error can't find the container with id 7a0e4a5870b01d7222c1d795ec4c6e19ffc1256ca5ec9a32b6f3d60f9b5ae5c5
Oct 30 01:03:11.391306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.389335    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"]
Oct 30 01:03:11.534672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.534014    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"]
Oct 30 01:03:11.536262 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:03:11.536229    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5ed5a4_fb1d_44c6_9ab9_3b5a1fd3b198.slice/crio-9f88d6c24bf6c11d0fbbab0a51f38df784dd4f4f7dd54722239b5d44de03bf05 WatchSource:0}: Error finding container 9f88d6c24bf6c11d0fbbab0a51f38df784dd4f4f7dd54722239b5d44de03bf05: Status 404 returned error can't find the container with id 9f88d6c24bf6c11d0fbbab0a51f38df784dd4f4f7dd54722239b5d44de03bf05
Oct 30 01:03:11.770292 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.770257    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24" event={"ID":"9f3634c9-1768-46e5-9979-9738b4486e10","Type":"ContainerStarted","Data":"ece0cc0ff69a41aa2e74cb2f0cbbbefec743ea204c1bbfc28e3fc867cf75807c"}
Oct 30 01:03:11.770454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.770324    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24" event={"ID":"9f3634c9-1768-46e5-9979-9738b4486e10","Type":"ContainerStarted","Data":"e2801e5c44489eb590b84110d36b3dbde38ce3f4f86af539c358bac474408dae"}
Oct 30 01:03:11.770454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.770357    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"
Oct 30 01:03:11.771723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.771697    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2" event={"ID":"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198","Type":"ContainerStarted","Data":"9f88d6c24bf6c11d0fbbab0a51f38df784dd4f4f7dd54722239b5d44de03bf05"}
Oct 30 01:03:11.773457 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.773433    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq" event={"ID":"c963677e-9e3a-4708-be7d-0fa8202a70ef","Type":"ContainerStarted","Data":"0c1043c6e254c10a60c82435ad8793971f33d8dff46b0a004df22874b9bf0c54"}
Oct 30 01:03:11.773556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.773475    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq" event={"ID":"c963677e-9e3a-4708-be7d-0fa8202a70ef","Type":"ContainerStarted","Data":"7a0e4a5870b01d7222c1d795ec4c6e19ffc1256ca5ec9a32b6f3d60f9b5ae5c5"}
Oct 30 01:03:11.773556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.773512    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"
Oct 30 01:03:11.808348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.808303    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24" podStartSLOduration=2.8082895949999997 podStartE2EDuration="2.808289595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:11.805261459 +0000 UTC m=+1300.554916767" watchObservedRunningTime="2026-10-30 01:03:11.808289595 +0000 UTC m=+1300.557944902"
Oct 30 01:03:11.843461 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:11.843421    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq" podStartSLOduration=2.8434082099999998 podStartE2EDuration="2.84340821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:11.839594162 +0000 UTC m=+1300.589249469" watchObservedRunningTime="2026-10-30 01:03:11.84340821 +0000 UTC m=+1300.593063578"
Oct 30 01:03:12.106785 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.106714    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:12.107209 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.107187    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:12.108057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.108037    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:03:12.189854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.189812    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn" podUID="b29bb481-ba88-4108-855e-bd487d1ee296" containerName="payload-processing" containerID="cri-o://b854e7588937671bcc68b656975b54123dc0609a7da05531b7c55abcf92910a3" gracePeriod=25
Oct 30 01:03:12.227612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.226776    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2" podUID="03c01c12-309c-4260-b87c-374110f957fd" containerName="payload-pre-processing" containerID="cri-o://bf51bbb0443530fc2f082aba6a6ace7b6b25eb87d40acfe79609d6d50db00b05" gracePeriod=25
Oct 30 01:03:12.519896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.519866    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"
Oct 30 01:03:12.632891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.632867    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"
Oct 30 01:03:12.674843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.674649    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b29bb481-ba88-4108-855e-bd487d1ee296-kube-api-access-4srgb\" (UniqueName: \"kubernetes.io/projected/b29bb481-ba88-4108-855e-bd487d1ee296-kube-api-access-4srgb\") pod \"b29bb481-ba88-4108-855e-bd487d1ee296\" (UID: \"b29bb481-ba88-4108-855e-bd487d1ee296\") "
Oct 30 01:03:12.675406 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.675381    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b29bb481-ba88-4108-855e-bd487d1ee296-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b29bb481-ba88-4108-855e-bd487d1ee296-plugins-config-volume\") pod \"b29bb481-ba88-4108-855e-bd487d1ee296\" (UID: \"b29bb481-ba88-4108-855e-bd487d1ee296\") "
Oct 30 01:03:12.676705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.676680    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b29bb481-ba88-4108-855e-bd487d1ee296-plugins-config-volume" pod "b29bb481-ba88-4108-855e-bd487d1ee296" (UID: "b29bb481-ba88-4108-855e-bd487d1ee296"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:12.677746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.677723    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b29bb481-ba88-4108-855e-bd487d1ee296-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:03:12.679936 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.679914    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29bb481-ba88-4108-855e-bd487d1ee296-kube-api-access-4srgb" pod "b29bb481-ba88-4108-855e-bd487d1ee296" (UID: "b29bb481-ba88-4108-855e-bd487d1ee296"). InnerVolumeSpecName "kube-api-access-4srgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:12.778602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.778551    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c01c12-309c-4260-b87c-374110f957fd-kube-api-access-9f7zb\" (UniqueName: \"kubernetes.io/projected/03c01c12-309c-4260-b87c-374110f957fd-kube-api-access-9f7zb\") pod \"03c01c12-309c-4260-b87c-374110f957fd\" (UID: \"03c01c12-309c-4260-b87c-374110f957fd\") "
Oct 30 01:03:12.778777 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.778737    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03c01c12-309c-4260-b87c-374110f957fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c01c12-309c-4260-b87c-374110f957fd-plugins-config-volume\") pod \"03c01c12-309c-4260-b87c-374110f957fd\" (UID: \"03c01c12-309c-4260-b87c-374110f957fd\") "
Oct 30 01:03:12.779077 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.779054    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c01c12-309c-4260-b87c-374110f957fd-plugins-config-volume" pod "03c01c12-309c-4260-b87c-374110f957fd" (UID: "03c01c12-309c-4260-b87c-374110f957fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:12.779399 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.779383    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4srgb\" (UniqueName: \"kubernetes.io/projected/b29bb481-ba88-4108-855e-bd487d1ee296-kube-api-access-4srgb\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:03:12.779466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.779401    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03c01c12-309c-4260-b87c-374110f957fd-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:03:12.780790 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.780770    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c01c12-309c-4260-b87c-374110f957fd-kube-api-access-9f7zb" pod "03c01c12-309c-4260-b87c-374110f957fd" (UID: "03c01c12-309c-4260-b87c-374110f957fd"). InnerVolumeSpecName "kube-api-access-9f7zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:12.783434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.783404    4425 generic.go:356] "Generic (PLEG): container finished" podID="de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198" containerID="ac0722eae50aa47ab9ed124f1d57875d0d1386c3979a27faaafc0f69d19f5c41" exitCode=0
Oct 30 01:03:12.783540 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.783505    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2" event={"ID":"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198","Type":"ContainerDied","Data":"ac0722eae50aa47ab9ed124f1d57875d0d1386c3979a27faaafc0f69d19f5c41"}
Oct 30 01:03:12.786908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.786884    4425 generic.go:356] "Generic (PLEG): container finished" podID="b29bb481-ba88-4108-855e-bd487d1ee296" containerID="b854e7588937671bcc68b656975b54123dc0609a7da05531b7c55abcf92910a3" exitCode=0
Oct 30 01:03:12.787032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.786949    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"
Oct 30 01:03:12.787032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.786966    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn" event={"ID":"b29bb481-ba88-4108-855e-bd487d1ee296","Type":"ContainerDied","Data":"b854e7588937671bcc68b656975b54123dc0609a7da05531b7c55abcf92910a3"}
Oct 30 01:03:12.787032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.787021    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn" event={"ID":"b29bb481-ba88-4108-855e-bd487d1ee296","Type":"ContainerDied","Data":"aac435b4cd1f08fd2492c68345c59a8ebdadec994bb92624cfddac2b860ab18c"}
Oct 30 01:03:12.787196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.787049    4425 scope.go:122] "RemoveContainer" containerID="b854e7588937671bcc68b656975b54123dc0609a7da05531b7c55abcf92910a3"
Oct 30 01:03:12.788601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.788578    4425 generic.go:356] "Generic (PLEG): container finished" podID="03c01c12-309c-4260-b87c-374110f957fd" containerID="bf51bbb0443530fc2f082aba6a6ace7b6b25eb87d40acfe79609d6d50db00b05" exitCode=0
Oct 30 01:03:12.788710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.788628    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"
Oct 30 01:03:12.788710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.788647    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2" event={"ID":"03c01c12-309c-4260-b87c-374110f957fd","Type":"ContainerDied","Data":"bf51bbb0443530fc2f082aba6a6ace7b6b25eb87d40acfe79609d6d50db00b05"}
Oct 30 01:03:12.788710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.788687    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2" event={"ID":"03c01c12-309c-4260-b87c-374110f957fd","Type":"ContainerDied","Data":"8177f62bb01d8ed1f4e34c72c02e0bce6e2ea1e85217693685e89b8f54ce108f"}
Oct 30 01:03:12.799127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.799110    4425 scope.go:122] "RemoveContainer" containerID="b854e7588937671bcc68b656975b54123dc0609a7da05531b7c55abcf92910a3"
Oct 30 01:03:12.799436 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:03:12.799417    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b854e7588937671bcc68b656975b54123dc0609a7da05531b7c55abcf92910a3\": container with ID starting with b854e7588937671bcc68b656975b54123dc0609a7da05531b7c55abcf92910a3 not found: ID does not exist" containerID="b854e7588937671bcc68b656975b54123dc0609a7da05531b7c55abcf92910a3"
Oct 30 01:03:12.799486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.799443    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b854e7588937671bcc68b656975b54123dc0609a7da05531b7c55abcf92910a3"} err="failed to get container status \"b854e7588937671bcc68b656975b54123dc0609a7da05531b7c55abcf92910a3\": rpc error: code = NotFound desc = could not find container \"b854e7588937671bcc68b656975b54123dc0609a7da05531b7c55abcf92910a3\": container with ID starting with b854e7588937671bcc68b656975b54123dc0609a7da05531b7c55abcf92910a3 not found: ID does not exist"
Oct 30 01:03:12.799486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.799458    4425 scope.go:122] "RemoveContainer" containerID="bf51bbb0443530fc2f082aba6a6ace7b6b25eb87d40acfe79609d6d50db00b05"
Oct 30 01:03:12.810024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.809838    4425 scope.go:122] "RemoveContainer" containerID="bf51bbb0443530fc2f082aba6a6ace7b6b25eb87d40acfe79609d6d50db00b05"
Oct 30 01:03:12.814237 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:03:12.813348    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf51bbb0443530fc2f082aba6a6ace7b6b25eb87d40acfe79609d6d50db00b05\": container with ID starting with bf51bbb0443530fc2f082aba6a6ace7b6b25eb87d40acfe79609d6d50db00b05 not found: ID does not exist" containerID="bf51bbb0443530fc2f082aba6a6ace7b6b25eb87d40acfe79609d6d50db00b05"
Oct 30 01:03:12.814237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.813402    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf51bbb0443530fc2f082aba6a6ace7b6b25eb87d40acfe79609d6d50db00b05"} err="failed to get container status \"bf51bbb0443530fc2f082aba6a6ace7b6b25eb87d40acfe79609d6d50db00b05\": rpc error: code = NotFound desc = could not find container \"bf51bbb0443530fc2f082aba6a6ace7b6b25eb87d40acfe79609d6d50db00b05\": container with ID starting with bf51bbb0443530fc2f082aba6a6ace7b6b25eb87d40acfe79609d6d50db00b05 not found: ID does not exist"
Oct 30 01:03:12.857704 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.857672    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"]
Oct 30 01:03:12.862739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.862709    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-153746-6f5649557f-sbzv2"]
Oct 30 01:03:12.881041 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.881006    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f7zb\" (UniqueName: \"kubernetes.io/projected/03c01c12-309c-4260-b87c-374110f957fd-kube-api-access-9f7zb\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:03:12.892526 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.892497    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"]
Oct 30 01:03:12.898779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:12.898757    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-153746-c5db87648-7hghn"]
Oct 30 01:03:13.425321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:13.425287    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03c01c12-309c-4260-b87c-374110f957fd" path="/var/lib/kubelet/pods/03c01c12-309c-4260-b87c-374110f957fd/volumes"
Oct 30 01:03:13.425777 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:13.425763    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b29bb481-ba88-4108-855e-bd487d1ee296" path="/var/lib/kubelet/pods/b29bb481-ba88-4108-855e-bd487d1ee296/volumes"
Oct 30 01:03:13.953919 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:13.953897    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"
Oct 30 01:03:13.997101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:13.997076    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-kube-api-access-m2mlg\" (UniqueName: \"kubernetes.io/projected/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-kube-api-access-m2mlg\") pod \"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198\" (UID: \"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198\") "
Oct 30 01:03:13.997304 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:13.997290    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-maas-api-openshift-ca\") pod \"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198\" (UID: \"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198\") "
Oct 30 01:03:13.997460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:13.997439    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-maas-api-xks-ca\") pod \"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198\" (UID: \"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198\") "
Oct 30 01:03:13.997691 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:13.997669    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-maas-api-openshift-ca" pod "de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198" (UID: "de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:13.997896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:13.997875    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:03:13.999389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:13.999347    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-kube-api-access-m2mlg" pod "de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198" (UID: "de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198"). InnerVolumeSpecName "kube-api-access-m2mlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:13.999628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:13.999599    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-maas-api-xks-ca" pod "de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198" (UID: "de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:14.098849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:14.098765    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2mlg\" (UniqueName: \"kubernetes.io/projected/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-kube-api-access-m2mlg\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:03:14.098849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:14.098798    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:03:14.807265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:14.807236    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"
Oct 30 01:03:14.807635 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:14.807235    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2" event={"ID":"de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198","Type":"ContainerDied","Data":"9f88d6c24bf6c11d0fbbab0a51f38df784dd4f4f7dd54722239b5d44de03bf05"}
Oct 30 01:03:14.807635 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:14.807359    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f88d6c24bf6c11d0fbbab0a51f38df784dd4f4f7dd54722239b5d44de03bf05"
Oct 30 01:03:17.782874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:17.781944    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:03:18.457596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.457555    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"]
Oct 30 01:03:18.458213 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.458193    4425 cpu_manager.go:414] "containerMap: removing container" podUID="03c01c12-309c-4260-b87c-374110f957fd" containerName="payload-pre-processing"
Oct 30 01:03:18.458334 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.458214    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c01c12-309c-4260-b87c-374110f957fd" containerName="payload-pre-processing"
Oct 30 01:03:18.458334 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.458270    4425 cpu_manager.go:414] "containerMap: removing container" podUID="de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198" containerName="revoke-keys"
Oct 30 01:03:18.458334 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.458279    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198" containerName="revoke-keys"
Oct 30 01:03:18.458334 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.458312    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b29bb481-ba88-4108-855e-bd487d1ee296" containerName="payload-processing"
Oct 30 01:03:18.458334 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.458319    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b29bb481-ba88-4108-855e-bd487d1ee296" containerName="payload-processing"
Oct 30 01:03:18.458573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.458479    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198" containerName="revoke-keys"
Oct 30 01:03:18.458573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.458499    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b29bb481-ba88-4108-855e-bd487d1ee296" containerName="payload-processing"
Oct 30 01:03:18.458573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.458512    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c01c12-309c-4260-b87c-374110f957fd" containerName="payload-pre-processing"
Oct 30 01:03:18.462451 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.462431    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"
Oct 30 01:03:18.480497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.480476    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-645126-dockercfg-9pdl4\""
Oct 30 01:03:18.484563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.484533    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"]
Oct 30 01:03:18.489670 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.489653    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-645126\""
Oct 30 01:03:18.623203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.587950    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k89hw\" (UniqueName: \"kubernetes.io/projected/0eabc38b-35ba-4133-a05d-bba79769f477-kube-api-access-k89hw\") pod \"payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g\" (UID: \"0eabc38b-35ba-4133-a05d-bba79769f477\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"
Oct 30 01:03:18.623203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.588177    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eabc38b-35ba-4133-a05d-bba79769f477-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g\" (UID: \"0eabc38b-35ba-4133-a05d-bba79769f477\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"
Oct 30 01:03:18.623203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.599470    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"]
Oct 30 01:03:18.623203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.608142    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"
Oct 30 01:03:18.623203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.609236    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"]
Oct 30 01:03:18.689135 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.689092    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6hzs\" (UniqueName: \"kubernetes.io/projected/2fb77042-ce3f-4913-b677-394cd6520fc0-kube-api-access-g6hzs\") pod \"payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b\" (UID: \"2fb77042-ce3f-4913-b677-394cd6520fc0\") " pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"
Oct 30 01:03:18.689364 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.689209    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k89hw\" (UniqueName: \"kubernetes.io/projected/0eabc38b-35ba-4133-a05d-bba79769f477-kube-api-access-k89hw\") pod \"payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g\" (UID: \"0eabc38b-35ba-4133-a05d-bba79769f477\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"
Oct 30 01:03:18.689426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.689412    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eabc38b-35ba-4133-a05d-bba79769f477-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g\" (UID: \"0eabc38b-35ba-4133-a05d-bba79769f477\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"
Oct 30 01:03:18.689584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.689566    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb77042-ce3f-4913-b677-394cd6520fc0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b\" (UID: \"2fb77042-ce3f-4913-b677-394cd6520fc0\") " pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"
Oct 30 01:03:18.690367 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.690343    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eabc38b-35ba-4133-a05d-bba79769f477-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g\" (UID: \"0eabc38b-35ba-4133-a05d-bba79769f477\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"
Oct 30 01:03:18.710092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.710030    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k89hw\" (UniqueName: \"kubernetes.io/projected/0eabc38b-35ba-4133-a05d-bba79769f477-kube-api-access-k89hw\") pod \"payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g\" (UID: \"0eabc38b-35ba-4133-a05d-bba79769f477\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"
Oct 30 01:03:18.776223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.776193    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"
Oct 30 01:03:18.791243 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.791213    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb77042-ce3f-4913-b677-394cd6520fc0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b\" (UID: \"2fb77042-ce3f-4913-b677-394cd6520fc0\") " pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"
Oct 30 01:03:18.791601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.791309    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6hzs\" (UniqueName: \"kubernetes.io/projected/2fb77042-ce3f-4913-b677-394cd6520fc0-kube-api-access-g6hzs\") pod \"payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b\" (UID: \"2fb77042-ce3f-4913-b677-394cd6520fc0\") " pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"
Oct 30 01:03:18.791868 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.791850    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb77042-ce3f-4913-b677-394cd6520fc0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b\" (UID: \"2fb77042-ce3f-4913-b677-394cd6520fc0\") " pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"
Oct 30 01:03:18.814040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.814015    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6hzs\" (UniqueName: \"kubernetes.io/projected/2fb77042-ce3f-4913-b677-394cd6520fc0-kube-api-access-g6hzs\") pod \"payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b\" (UID: \"2fb77042-ce3f-4913-b677-394cd6520fc0\") " pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"
Oct 30 01:03:18.962191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:18.962093    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"
Oct 30 01:03:19.088120 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:03:19.088089    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eabc38b_35ba_4133_a05d_bba79769f477.slice/crio-5b5a4a75acc412741ee789dd0f9a82caea92e7dbc8b3d7c36d1e45a602bcc240 WatchSource:0}: Error finding container 5b5a4a75acc412741ee789dd0f9a82caea92e7dbc8b3d7c36d1e45a602bcc240: Status 404 returned error can't find the container with id 5b5a4a75acc412741ee789dd0f9a82caea92e7dbc8b3d7c36d1e45a602bcc240
Oct 30 01:03:19.090483 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:19.090455    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"]
Oct 30 01:03:19.313290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:19.313256    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"]
Oct 30 01:03:19.314019 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:03:19.313994    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb77042_ce3f_4913_b677_394cd6520fc0.slice/crio-c2af9daedbbf72633407c6328ae1ad1ca41e62f76de72e4ceeb1b464b70d507e WatchSource:0}: Error finding container c2af9daedbbf72633407c6328ae1ad1ca41e62f76de72e4ceeb1b464b70d507e: Status 404 returned error can't find the container with id c2af9daedbbf72633407c6328ae1ad1ca41e62f76de72e4ceeb1b464b70d507e
Oct 30 01:03:19.881017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:19.880978    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b" event={"ID":"2fb77042-ce3f-4913-b677-394cd6520fc0","Type":"ContainerStarted","Data":"16fe9b18ec8b6d8ddbc33c023976f83956241a2bc5b1163066ba85333b85778e"}
Oct 30 01:03:19.881510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:19.881043    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b" event={"ID":"2fb77042-ce3f-4913-b677-394cd6520fc0","Type":"ContainerStarted","Data":"c2af9daedbbf72633407c6328ae1ad1ca41e62f76de72e4ceeb1b464b70d507e"}
Oct 30 01:03:19.882020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:19.882002    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"
Oct 30 01:03:19.884168 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:19.884122    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g" event={"ID":"0eabc38b-35ba-4133-a05d-bba79769f477","Type":"ContainerStarted","Data":"c8f0ff0ec5e0c1225203aef2f50d4ef9f0443b5d6352b4caa62eadec061dcb1c"}
Oct 30 01:03:19.884314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:19.884191    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g" event={"ID":"0eabc38b-35ba-4133-a05d-bba79769f477","Type":"ContainerStarted","Data":"5b5a4a75acc412741ee789dd0f9a82caea92e7dbc8b3d7c36d1e45a602bcc240"}
Oct 30 01:03:19.884314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:19.884223    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"
Oct 30 01:03:19.920211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:19.920133    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b" podStartSLOduration=1.92011405 podStartE2EDuration="1.92011405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:19.918010924 +0000 UTC m=+1308.667666232" watchObservedRunningTime="2026-10-30 01:03:19.92011405 +0000 UTC m=+1308.669769361"
Oct 30 01:03:19.953634 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:19.953585    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g" podStartSLOduration=1.953571704 podStartE2EDuration="1.953571704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:19.953147497 +0000 UTC m=+1308.702802807" watchObservedRunningTime="2026-10-30 01:03:19.953571704 +0000 UTC m=+1308.703227038"
Oct 30 01:03:21.246646 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:21.246609    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"]
Oct 30 01:03:21.279590 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:21.279556    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"]
Oct 30 01:03:22.792259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:22.792227    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"
Oct 30 01:03:22.792671 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:22.792344    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"
Oct 30 01:03:26.238458 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.238425    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"]
Oct 30 01:03:26.243032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.243010    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"
Oct 30 01:03:26.247519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.247474    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf" podUID="ddd3452d-e481-4d50-8c60-f102ecca1aa7" containerName="payload-processing" containerID="cri-o://c525e16dd2362e13eef8a8d498299339c1683b9962037d4fec5461919c8e4691" gracePeriod=25
Oct 30 01:03:26.260095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.260074    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1c78b8\""
Oct 30 01:03:26.264571 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.264544    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"]
Oct 30 01:03:26.280838 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.280807    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh" podUID="de1980d7-8abc-483c-8448-5406fbc83631" containerName="payload-pre-processing" containerID="cri-o://0fe934a7ef4979e8a0a9206021d6712a6a9f555e4e3625fbc991fb68fb7f9e64" gracePeriod=25
Oct 30 01:03:26.316853 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.316831    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"]
Oct 30 01:03:26.321254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.321235    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"
Oct 30 01:03:26.321919 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.321899    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtkc6\" (UniqueName: \"kubernetes.io/projected/7cba143d-77d0-467f-af24-07888c71e960-kube-api-access-mtkc6\") pod \"payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj\" (UID: \"7cba143d-77d0-467f-af24-07888c71e960\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"
Oct 30 01:03:26.321988 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.321976    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cba143d-77d0-467f-af24-07888c71e960-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj\" (UID: \"7cba143d-77d0-467f-af24-07888c71e960\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"
Oct 30 01:03:26.343825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.343803    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1c78b8-dockercfg-tzvtg\""
Oct 30 01:03:26.347215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.347191    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"]
Oct 30 01:03:26.426814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.425252    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ebc6cf-1479-4394-842c-26395091cedd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f\" (UID: \"94ebc6cf-1479-4394-842c-26395091cedd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"
Oct 30 01:03:26.426814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.425313    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtkc6\" (UniqueName: \"kubernetes.io/projected/7cba143d-77d0-467f-af24-07888c71e960-kube-api-access-mtkc6\") pod \"payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj\" (UID: \"7cba143d-77d0-467f-af24-07888c71e960\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"
Oct 30 01:03:26.426814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.425407    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cba143d-77d0-467f-af24-07888c71e960-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj\" (UID: \"7cba143d-77d0-467f-af24-07888c71e960\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"
Oct 30 01:03:26.426814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.425538    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qmth\" (UniqueName: \"kubernetes.io/projected/94ebc6cf-1479-4394-842c-26395091cedd-kube-api-access-5qmth\") pod \"payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f\" (UID: \"94ebc6cf-1479-4394-842c-26395091cedd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"
Oct 30 01:03:26.426814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.426762    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cba143d-77d0-467f-af24-07888c71e960-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj\" (UID: \"7cba143d-77d0-467f-af24-07888c71e960\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"
Oct 30 01:03:26.465214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.463119    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtkc6\" (UniqueName: \"kubernetes.io/projected/7cba143d-77d0-467f-af24-07888c71e960-kube-api-access-mtkc6\") pod \"payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj\" (UID: \"7cba143d-77d0-467f-af24-07888c71e960\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"
Oct 30 01:03:26.531186 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.529580    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ebc6cf-1479-4394-842c-26395091cedd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f\" (UID: \"94ebc6cf-1479-4394-842c-26395091cedd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"
Oct 30 01:03:26.531186 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.529831    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qmth\" (UniqueName: \"kubernetes.io/projected/94ebc6cf-1479-4394-842c-26395091cedd-kube-api-access-5qmth\") pod \"payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f\" (UID: \"94ebc6cf-1479-4394-842c-26395091cedd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"
Oct 30 01:03:26.531186 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.530751    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ebc6cf-1479-4394-842c-26395091cedd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f\" (UID: \"94ebc6cf-1479-4394-842c-26395091cedd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"
Oct 30 01:03:26.565202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.559459    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"
Oct 30 01:03:26.565202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.563145    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qmth\" (UniqueName: \"kubernetes.io/projected/94ebc6cf-1479-4394-842c-26395091cedd-kube-api-access-5qmth\") pod \"payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f\" (UID: \"94ebc6cf-1479-4394-842c-26395091cedd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"
Oct 30 01:03:26.631007 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.630790    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"
Oct 30 01:03:26.680931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.680681    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"
Oct 30 01:03:26.719805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.719485    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"
Oct 30 01:03:26.769262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.764698    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd3452d-e481-4d50-8c60-f102ecca1aa7-kube-api-access-v75jc\" (UniqueName: \"kubernetes.io/projected/ddd3452d-e481-4d50-8c60-f102ecca1aa7-kube-api-access-v75jc\") pod \"ddd3452d-e481-4d50-8c60-f102ecca1aa7\" (UID: \"ddd3452d-e481-4d50-8c60-f102ecca1aa7\") "
Oct 30 01:03:26.769262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.764796    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd3452d-e481-4d50-8c60-f102ecca1aa7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd3452d-e481-4d50-8c60-f102ecca1aa7-plugins-config-volume\") pod \"ddd3452d-e481-4d50-8c60-f102ecca1aa7\" (UID: \"ddd3452d-e481-4d50-8c60-f102ecca1aa7\") "
Oct 30 01:03:26.769262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.765912    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd3452d-e481-4d50-8c60-f102ecca1aa7-plugins-config-volume" pod "ddd3452d-e481-4d50-8c60-f102ecca1aa7" (UID: "ddd3452d-e481-4d50-8c60-f102ecca1aa7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:26.782619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.782584    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd3452d-e481-4d50-8c60-f102ecca1aa7-kube-api-access-v75jc" pod "ddd3452d-e481-4d50-8c60-f102ecca1aa7" (UID: "ddd3452d-e481-4d50-8c60-f102ecca1aa7"). InnerVolumeSpecName "kube-api-access-v75jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:26.867435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.867399    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1980d7-8abc-483c-8448-5406fbc83631-kube-api-access-fr5k2\" (UniqueName: \"kubernetes.io/projected/de1980d7-8abc-483c-8448-5406fbc83631-kube-api-access-fr5k2\") pod \"de1980d7-8abc-483c-8448-5406fbc83631\" (UID: \"de1980d7-8abc-483c-8448-5406fbc83631\") "
Oct 30 01:03:26.867730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.867710    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de1980d7-8abc-483c-8448-5406fbc83631-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1980d7-8abc-483c-8448-5406fbc83631-plugins-config-volume\") pod \"de1980d7-8abc-483c-8448-5406fbc83631\" (UID: \"de1980d7-8abc-483c-8448-5406fbc83631\") "
Oct 30 01:03:26.868231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.868201    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de1980d7-8abc-483c-8448-5406fbc83631-plugins-config-volume" pod "de1980d7-8abc-483c-8448-5406fbc83631" (UID: "de1980d7-8abc-483c-8448-5406fbc83631"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:26.868405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.868384    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v75jc\" (UniqueName: \"kubernetes.io/projected/ddd3452d-e481-4d50-8c60-f102ecca1aa7-kube-api-access-v75jc\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:03:26.868452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.868413    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de1980d7-8abc-483c-8448-5406fbc83631-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:03:26.868452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.868427    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd3452d-e481-4d50-8c60-f102ecca1aa7-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:03:26.869609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.869585    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1980d7-8abc-483c-8448-5406fbc83631-kube-api-access-fr5k2" pod "de1980d7-8abc-483c-8448-5406fbc83631" (UID: "de1980d7-8abc-483c-8448-5406fbc83631"). InnerVolumeSpecName "kube-api-access-fr5k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:26.893974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.893953    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"]
Oct 30 01:03:26.894943 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:03:26.894919    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cba143d_77d0_467f_af24_07888c71e960.slice/crio-a47de4455372e8b83db01bd7b19046b80466f3460027ba96527551cdf603bbe8 WatchSource:0}: Error finding container a47de4455372e8b83db01bd7b19046b80466f3460027ba96527551cdf603bbe8: Status 404 returned error can't find the container with id a47de4455372e8b83db01bd7b19046b80466f3460027ba96527551cdf603bbe8
Oct 30 01:03:26.957341 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.957254    4425 generic.go:356] "Generic (PLEG): container finished" podID="ddd3452d-e481-4d50-8c60-f102ecca1aa7" containerID="c525e16dd2362e13eef8a8d498299339c1683b9962037d4fec5461919c8e4691" exitCode=0
Oct 30 01:03:26.957341 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.957291    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"
Oct 30 01:03:26.957473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.957336    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf" event={"ID":"ddd3452d-e481-4d50-8c60-f102ecca1aa7","Type":"ContainerDied","Data":"c525e16dd2362e13eef8a8d498299339c1683b9962037d4fec5461919c8e4691"}
Oct 30 01:03:26.957473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.957382    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf" event={"ID":"ddd3452d-e481-4d50-8c60-f102ecca1aa7","Type":"ContainerDied","Data":"0c54ce33072fb656ae3c8ca3174a1cf80ef4d0a59a5b617b3df66af76e35f7b2"}
Oct 30 01:03:26.957473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.957404    4425 scope.go:122] "RemoveContainer" containerID="c525e16dd2362e13eef8a8d498299339c1683b9962037d4fec5461919c8e4691"
Oct 30 01:03:26.959470 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.959443    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj" event={"ID":"7cba143d-77d0-467f-af24-07888c71e960","Type":"ContainerStarted","Data":"a47de4455372e8b83db01bd7b19046b80466f3460027ba96527551cdf603bbe8"}
Oct 30 01:03:26.960899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.960879    4425 generic.go:356] "Generic (PLEG): container finished" podID="de1980d7-8abc-483c-8448-5406fbc83631" containerID="0fe934a7ef4979e8a0a9206021d6712a6a9f555e4e3625fbc991fb68fb7f9e64" exitCode=0
Oct 30 01:03:26.960968 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.960939    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"
Oct 30 01:03:26.961027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.960973    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh" event={"ID":"de1980d7-8abc-483c-8448-5406fbc83631","Type":"ContainerDied","Data":"0fe934a7ef4979e8a0a9206021d6712a6a9f555e4e3625fbc991fb68fb7f9e64"}
Oct 30 01:03:26.961027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.961015    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh" event={"ID":"de1980d7-8abc-483c-8448-5406fbc83631","Type":"ContainerDied","Data":"e07dec4731a28539ac0e43f0d36d66756bf7cc814ec37ba8c74df0f3e94a97a4"}
Oct 30 01:03:26.968096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.968020    4425 scope.go:122] "RemoveContainer" containerID="c525e16dd2362e13eef8a8d498299339c1683b9962037d4fec5461919c8e4691"
Oct 30 01:03:26.968363 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:03:26.968321    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c525e16dd2362e13eef8a8d498299339c1683b9962037d4fec5461919c8e4691\": container with ID starting with c525e16dd2362e13eef8a8d498299339c1683b9962037d4fec5461919c8e4691 not found: ID does not exist" containerID="c525e16dd2362e13eef8a8d498299339c1683b9962037d4fec5461919c8e4691"
Oct 30 01:03:26.968363 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.968352    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c525e16dd2362e13eef8a8d498299339c1683b9962037d4fec5461919c8e4691"} err="failed to get container status \"c525e16dd2362e13eef8a8d498299339c1683b9962037d4fec5461919c8e4691\": rpc error: code = NotFound desc = could not find container \"c525e16dd2362e13eef8a8d498299339c1683b9962037d4fec5461919c8e4691\": container with ID starting with c525e16dd2362e13eef8a8d498299339c1683b9962037d4fec5461919c8e4691 not found: ID does not exist"
Oct 30 01:03:26.968592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.968370    4425 scope.go:122] "RemoveContainer" containerID="0fe934a7ef4979e8a0a9206021d6712a6a9f555e4e3625fbc991fb68fb7f9e64"
Oct 30 01:03:26.969583 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.969564    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr5k2\" (UniqueName: \"kubernetes.io/projected/de1980d7-8abc-483c-8448-5406fbc83631-kube-api-access-fr5k2\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:03:26.977360 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.977345    4425 scope.go:122] "RemoveContainer" containerID="0fe934a7ef4979e8a0a9206021d6712a6a9f555e4e3625fbc991fb68fb7f9e64"
Oct 30 01:03:26.977628 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:03:26.977609    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe934a7ef4979e8a0a9206021d6712a6a9f555e4e3625fbc991fb68fb7f9e64\": container with ID starting with 0fe934a7ef4979e8a0a9206021d6712a6a9f555e4e3625fbc991fb68fb7f9e64 not found: ID does not exist" containerID="0fe934a7ef4979e8a0a9206021d6712a6a9f555e4e3625fbc991fb68fb7f9e64"
Oct 30 01:03:26.977682 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:26.977634    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe934a7ef4979e8a0a9206021d6712a6a9f555e4e3625fbc991fb68fb7f9e64"} err="failed to get container status \"0fe934a7ef4979e8a0a9206021d6712a6a9f555e4e3625fbc991fb68fb7f9e64\": rpc error: code = NotFound desc = could not find container \"0fe934a7ef4979e8a0a9206021d6712a6a9f555e4e3625fbc991fb68fb7f9e64\": container with ID starting with 0fe934a7ef4979e8a0a9206021d6712a6a9f555e4e3625fbc991fb68fb7f9e64 not found: ID does not exist"
Oct 30 01:03:27.004378 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:27.004355    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"]
Oct 30 01:03:27.008420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:27.008397    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6482ae-75574d87c4-fhgjf"]
Oct 30 01:03:27.040009 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:27.039985    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"]
Oct 30 01:03:27.041197 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:03:27.041173    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ebc6cf_1479_4394_842c_26395091cedd.slice/crio-2d7cdf2a8fcd9cf4eec11f77ed31218c49f8ef999070b1d54fbad326e8b423cb WatchSource:0}: Error finding container 2d7cdf2a8fcd9cf4eec11f77ed31218c49f8ef999070b1d54fbad326e8b423cb: Status 404 returned error can't find the container with id 2d7cdf2a8fcd9cf4eec11f77ed31218c49f8ef999070b1d54fbad326e8b423cb
Oct 30 01:03:27.051140 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:27.051116    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"]
Oct 30 01:03:27.057123 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:27.057099    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6482ae-6b4675bd95-7zgxh"]
Oct 30 01:03:27.423033 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:27.422961    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd3452d-e481-4d50-8c60-f102ecca1aa7" path="/var/lib/kubelet/pods/ddd3452d-e481-4d50-8c60-f102ecca1aa7/volumes"
Oct 30 01:03:27.423460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:27.423446    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de1980d7-8abc-483c-8448-5406fbc83631" path="/var/lib/kubelet/pods/de1980d7-8abc-483c-8448-5406fbc83631/volumes"
Oct 30 01:03:27.971337 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:27.971306    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f" event={"ID":"94ebc6cf-1479-4394-842c-26395091cedd","Type":"ContainerStarted","Data":"2fb91750a75440b03719751fac5dd7debee2be899e70306e7fef1162c94fe2d0"}
Oct 30 01:03:27.971659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:27.971354    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f" event={"ID":"94ebc6cf-1479-4394-842c-26395091cedd","Type":"ContainerStarted","Data":"2d7cdf2a8fcd9cf4eec11f77ed31218c49f8ef999070b1d54fbad326e8b423cb"}
Oct 30 01:03:27.971659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:27.971506    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"
Oct 30 01:03:27.974264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:27.974236    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj" event={"ID":"7cba143d-77d0-467f-af24-07888c71e960","Type":"ContainerStarted","Data":"44d6dca66d56db84e3fc702b056bd51c1a4fdd7d472b238a27f1b021ec3f5887"}
Oct 30 01:03:27.974421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:27.974291    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"
Oct 30 01:03:28.008686 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:28.008639    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f" podStartSLOduration=2.008627172 podStartE2EDuration="2.008627172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:28.008049265 +0000 UTC m=+1316.757704570" watchObservedRunningTime="2026-10-30 01:03:28.008627172 +0000 UTC m=+1316.758282479"
Oct 30 01:03:28.047424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:28.047379    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj" podStartSLOduration=2.047364632 podStartE2EDuration="2.047364632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:28.042986034 +0000 UTC m=+1316.792641341" watchObservedRunningTime="2026-10-30 01:03:28.047364632 +0000 UTC m=+1316.797019939"
Oct 30 01:03:30.896200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:30.896091    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"
Oct 30 01:03:31.925851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:31.925826    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"
Oct 30 01:03:35.033825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:35.033791    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"]
Oct 30 01:03:35.043515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:35.043489    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a6e97fb-mj6b6"]
Oct 30 01:03:35.424824 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:35.424752    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9752cf4b-2544-4b25-ae84-1b47b48c6ccb" path="/var/lib/kubelet/pods/9752cf4b-2544-4b25-ae84-1b47b48c6ccb/volumes"
Oct 30 01:03:37.555257 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:03:37.555222    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:03:38.789819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.789784    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"]
Oct 30 01:03:38.790754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.790724    4425 cpu_manager.go:414] "containerMap: removing container" podUID="de1980d7-8abc-483c-8448-5406fbc83631" containerName="payload-pre-processing"
Oct 30 01:03:38.790754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.790754    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1980d7-8abc-483c-8448-5406fbc83631" containerName="payload-pre-processing"
Oct 30 01:03:38.790923 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.790831    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ddd3452d-e481-4d50-8c60-f102ecca1aa7" containerName="payload-processing"
Oct 30 01:03:38.790923 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.790838    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd3452d-e481-4d50-8c60-f102ecca1aa7" containerName="payload-processing"
Oct 30 01:03:38.791000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.790972    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd3452d-e481-4d50-8c60-f102ecca1aa7" containerName="payload-processing"
Oct 30 01:03:38.791000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.790986    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1980d7-8abc-483c-8448-5406fbc83631" containerName="payload-pre-processing"
Oct 30 01:03:38.796572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.796549    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.810945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.810920    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-htf9g\""
Oct 30 01:03:38.812056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.811194    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 01:03:38.812056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.811887    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 01:03:38.812056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.811978    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 01:03:38.833036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.828140    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.833036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.828221    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.833036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.828332    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b917a55-0178-422d-919a-c7499e12aff3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.833036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.828443    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b917a55-0178-422d-919a-c7499e12aff3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.833036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.828486    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b917a55-0178-422d-919a-c7499e12aff3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.833036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.828546    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6fsl\" (UniqueName: \"kubernetes.io/projected/7b917a55-0178-422d-919a-c7499e12aff3-kube-api-access-g6fsl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.833036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.828587    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.833036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.832404    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"]
Oct 30 01:03:38.929417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.929379    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b917a55-0178-422d-919a-c7499e12aff3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.929605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.929455    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6fsl\" (UniqueName: \"kubernetes.io/projected/7b917a55-0178-422d-919a-c7499e12aff3-kube-api-access-g6fsl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.929605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.929544    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.929682 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.929621    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.929716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.929679    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.929829 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.929810    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b917a55-0178-422d-919a-c7499e12aff3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.929972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.929955    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b917a55-0178-422d-919a-c7499e12aff3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.930702 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.930675    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b917a55-0178-422d-919a-c7499e12aff3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.932359 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.932337    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b917a55-0178-422d-919a-c7499e12aff3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.932487 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.932373    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.932605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.932577    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.932776 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.932755    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.950611 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.950583    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6fsl\" (UniqueName: \"kubernetes.io/projected/7b917a55-0178-422d-919a-c7499e12aff3-kube-api-access-g6fsl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.950820 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.950798    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b917a55-0178-422d-919a-c7499e12aff3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-krk8f\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:38.986064 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.986024    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"
Oct 30 01:03:38.986299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:38.986099    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"
Oct 30 01:03:39.125469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:39.125381    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:39.455008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:39.454978    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"]
Oct 30 01:03:39.456024 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:03:39.455996    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b917a55_0178_422d_919a_c7499e12aff3.slice/crio-dd042bd683991308ae4d166c572f6c8033703fe7990a81db788ec56e82d01489 WatchSource:0}: Error finding container dd042bd683991308ae4d166c572f6c8033703fe7990a81db788ec56e82d01489: Status 404 returned error can't find the container with id dd042bd683991308ae4d166c572f6c8033703fe7990a81db788ec56e82d01489
Oct 30 01:03:39.539115 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:39.539086    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"]
Oct 30 01:03:39.543433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:39.543400    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-153746-mv52c"]
Oct 30 01:03:40.084470 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:40.084433    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f" event={"ID":"7b917a55-0178-422d-919a-c7499e12aff3","Type":"ContainerStarted","Data":"dd042bd683991308ae4d166c572f6c8033703fe7990a81db788ec56e82d01489"}
Oct 30 01:03:41.429507 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:41.429470    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d47f37b6-499b-446d-b417-f087d97b372d" path="/var/lib/kubelet/pods/d47f37b6-499b-446d-b417-f087d97b372d/volumes"
Oct 30 01:03:41.939977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:41.939940    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:03:41.940114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:41.940008    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:03:41.954076 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:41.953938    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"]
Oct 30 01:03:41.960432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:41.960404    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6482ae-qxmq2"]
Oct 30 01:03:42.030646 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:42.030621    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"]
Oct 30 01:03:42.036148 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:42.036124    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c01ba8f6-4p422"]
Oct 30 01:03:42.122459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:42.122418    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f" event={"ID":"7b917a55-0178-422d-919a-c7499e12aff3","Type":"ContainerStarted","Data":"4d82b3e069898bfac687f9baab0e391c4d1171835d0bf858cfc06b93177c3190"}
Oct 30 01:03:42.122658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:42.122568    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:42.166111 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:42.166046    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f" podStartSLOduration=1.685876774 podStartE2EDuration="4.166027495s" totalImagesPullingTime="2.480150721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:42.16169351 +0000 UTC m=+1330.911348819" watchObservedRunningTime="2026-10-30 01:03:42.166027495 +0000 UTC m=+1330.915682804"
Oct 30 01:03:43.131509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:43.131480    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:03:43.423880 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:43.423806    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80196e9-ead5-4ce0-b6f1-90506f67d2eb" path="/var/lib/kubelet/pods/c80196e9-ead5-4ce0-b6f1-90506f67d2eb/volumes"
Oct 30 01:03:43.424310 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:43.424290    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198" path="/var/lib/kubelet/pods/de5ed5a4-fb1d-44c6-9ab9-3b5a1fd3b198/volumes"
Oct 30 01:03:49.793524 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:49.793481    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"]
Oct 30 01:03:49.799301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:49.799281    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"
Oct 30 01:03:49.822177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:49.819401    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"]
Oct 30 01:03:49.823893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:49.823863    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:03:49.823994 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:49.823913    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:03:49.833202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:49.832039    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:03:49.833202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:49.832117    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:03:49.927779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:49.927744    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c520da-45a6-4afa-b265-1b227e0a5b36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw\" (UID: \"02c520da-45a6-4afa-b265-1b227e0a5b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"
Oct 30 01:03:49.927779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:49.927780    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktzrn\" (UniqueName: \"kubernetes.io/projected/02c520da-45a6-4afa-b265-1b227e0a5b36-kube-api-access-ktzrn\") pod \"maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw\" (UID: \"02c520da-45a6-4afa-b265-1b227e0a5b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"
Oct 30 01:03:49.927977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:49.927917    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c520da-45a6-4afa-b265-1b227e0a5b36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw\" (UID: \"02c520da-45a6-4afa-b265-1b227e0a5b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"
Oct 30 01:03:50.029771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:50.029738    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c520da-45a6-4afa-b265-1b227e0a5b36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw\" (UID: \"02c520da-45a6-4afa-b265-1b227e0a5b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"
Oct 30 01:03:50.029771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:50.029778    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktzrn\" (UniqueName: \"kubernetes.io/projected/02c520da-45a6-4afa-b265-1b227e0a5b36-kube-api-access-ktzrn\") pod \"maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw\" (UID: \"02c520da-45a6-4afa-b265-1b227e0a5b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"
Oct 30 01:03:50.030041 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:50.029916    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c520da-45a6-4afa-b265-1b227e0a5b36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw\" (UID: \"02c520da-45a6-4afa-b265-1b227e0a5b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"
Oct 30 01:03:50.030585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:50.030556    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c520da-45a6-4afa-b265-1b227e0a5b36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw\" (UID: \"02c520da-45a6-4afa-b265-1b227e0a5b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"
Oct 30 01:03:50.032370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:50.032346    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c520da-45a6-4afa-b265-1b227e0a5b36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw\" (UID: \"02c520da-45a6-4afa-b265-1b227e0a5b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"
Oct 30 01:03:50.051132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:50.051081    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktzrn\" (UniqueName: \"kubernetes.io/projected/02c520da-45a6-4afa-b265-1b227e0a5b36-kube-api-access-ktzrn\") pod \"maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw\" (UID: \"02c520da-45a6-4afa-b265-1b227e0a5b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"
Oct 30 01:03:50.113290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:50.113262    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"
Oct 30 01:03:50.418542 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:50.418514    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"]
Oct 30 01:03:50.418723 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:03:50.418701    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c520da_45a6_4afa_b265_1b227e0a5b36.slice/crio-2a7766f130a7c3a3cadb54d74806c10eb7e3fb1ce254ad7cf9d4c82c9acc3342 WatchSource:0}: Error finding container 2a7766f130a7c3a3cadb54d74806c10eb7e3fb1ce254ad7cf9d4c82c9acc3342: Status 404 returned error can't find the container with id 2a7766f130a7c3a3cadb54d74806c10eb7e3fb1ce254ad7cf9d4c82c9acc3342
Oct 30 01:03:51.201589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:51.201554    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw" event={"ID":"02c520da-45a6-4afa-b265-1b227e0a5b36","Type":"ContainerStarted","Data":"2f77a27ed2147c56d0511c5c93d97f0c5c9bbb01c39e78e5fb8799f877feb9db"}
Oct 30 01:03:51.202005 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:51.201614    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw" event={"ID":"02c520da-45a6-4afa-b265-1b227e0a5b36","Type":"ContainerStarted","Data":"2a7766f130a7c3a3cadb54d74806c10eb7e3fb1ce254ad7cf9d4c82c9acc3342"}
Oct 30 01:03:51.241370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:51.241315    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw" podStartSLOduration=2.241300836 podStartE2EDuration="2.241300836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:51.237124127 +0000 UTC m=+1339.986779437" watchObservedRunningTime="2026-10-30 01:03:51.241300836 +0000 UTC m=+1339.990956143"
Oct 30 01:03:52.209985 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:52.209948    4425 generic.go:356] "Generic (PLEG): container finished" podID="02c520da-45a6-4afa-b265-1b227e0a5b36" containerID="2f77a27ed2147c56d0511c5c93d97f0c5c9bbb01c39e78e5fb8799f877feb9db" exitCode=0
Oct 30 01:03:52.210454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:52.210001    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw" event={"ID":"02c520da-45a6-4afa-b265-1b227e0a5b36","Type":"ContainerDied","Data":"2f77a27ed2147c56d0511c5c93d97f0c5c9bbb01c39e78e5fb8799f877feb9db"}
Oct 30 01:03:53.404437 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:53.404411    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"
Oct 30 01:03:53.581251 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:53.581213    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c520da-45a6-4afa-b265-1b227e0a5b36-kube-api-access-ktzrn\" (UniqueName: \"kubernetes.io/projected/02c520da-45a6-4afa-b265-1b227e0a5b36-kube-api-access-ktzrn\") pod \"02c520da-45a6-4afa-b265-1b227e0a5b36\" (UID: \"02c520da-45a6-4afa-b265-1b227e0a5b36\") "
Oct 30 01:03:53.581251 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:53.581256    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02c520da-45a6-4afa-b265-1b227e0a5b36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c520da-45a6-4afa-b265-1b227e0a5b36-maas-api-xks-ca\") pod \"02c520da-45a6-4afa-b265-1b227e0a5b36\" (UID: \"02c520da-45a6-4afa-b265-1b227e0a5b36\") "
Oct 30 01:03:53.581524 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:53.581460    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c520da-45a6-4afa-b265-1b227e0a5b36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c520da-45a6-4afa-b265-1b227e0a5b36-maas-api-openshift-ca\") pod \"02c520da-45a6-4afa-b265-1b227e0a5b36\" (UID: \"02c520da-45a6-4afa-b265-1b227e0a5b36\") "
Oct 30 01:03:53.581817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:53.581791    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c520da-45a6-4afa-b265-1b227e0a5b36-maas-api-openshift-ca" pod "02c520da-45a6-4afa-b265-1b227e0a5b36" (UID: "02c520da-45a6-4afa-b265-1b227e0a5b36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:53.582143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:53.582121    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c520da-45a6-4afa-b265-1b227e0a5b36-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:03:53.583456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:53.583433    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c520da-45a6-4afa-b265-1b227e0a5b36-maas-api-xks-ca" pod "02c520da-45a6-4afa-b265-1b227e0a5b36" (UID: "02c520da-45a6-4afa-b265-1b227e0a5b36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:53.583556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:53.583524    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c520da-45a6-4afa-b265-1b227e0a5b36-kube-api-access-ktzrn" pod "02c520da-45a6-4afa-b265-1b227e0a5b36" (UID: "02c520da-45a6-4afa-b265-1b227e0a5b36"). InnerVolumeSpecName "kube-api-access-ktzrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:53.683834 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:53.683799    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktzrn\" (UniqueName: \"kubernetes.io/projected/02c520da-45a6-4afa-b265-1b227e0a5b36-kube-api-access-ktzrn\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:03:53.683834 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:53.683830    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c520da-45a6-4afa-b265-1b227e0a5b36-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:03:54.227607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:54.227572    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw" event={"ID":"02c520da-45a6-4afa-b265-1b227e0a5b36","Type":"ContainerDied","Data":"2a7766f130a7c3a3cadb54d74806c10eb7e3fb1ce254ad7cf9d4c82c9acc3342"}
Oct 30 01:03:54.227770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:54.227611    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"
Oct 30 01:03:54.227770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:54.227669    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a7766f130a7c3a3cadb54d74806c10eb7e3fb1ce254ad7cf9d4c82c9acc3342"
Oct 30 01:03:57.536604 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.536567    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"]
Oct 30 01:03:57.537601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.537580    4425 cpu_manager.go:414] "containerMap: removing container" podUID="02c520da-45a6-4afa-b265-1b227e0a5b36" containerName="revoke-keys"
Oct 30 01:03:57.537670 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.537605    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c520da-45a6-4afa-b265-1b227e0a5b36" containerName="revoke-keys"
Oct 30 01:03:57.537779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.537767    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c520da-45a6-4afa-b265-1b227e0a5b36" containerName="revoke-keys"
Oct 30 01:03:57.542140 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.542117    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"
Oct 30 01:03:57.559861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.559828    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"]
Oct 30 01:03:57.564754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.564725    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:03:57.564754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.564765    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:03:57.564945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.564811    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:03:57.564945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.564891    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:03:57.634592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.634560    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4733821-6e60-4bd4-bb19-7b16fdb1171d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-flggq\" (UID: \"b4733821-6e60-4bd4-bb19-7b16fdb1171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"
Oct 30 01:03:57.634755 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.634625    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prr9p\" (UniqueName: \"kubernetes.io/projected/b4733821-6e60-4bd4-bb19-7b16fdb1171d-kube-api-access-prr9p\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-flggq\" (UID: \"b4733821-6e60-4bd4-bb19-7b16fdb1171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"
Oct 30 01:03:57.634755 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.634732    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4733821-6e60-4bd4-bb19-7b16fdb1171d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-flggq\" (UID: \"b4733821-6e60-4bd4-bb19-7b16fdb1171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"
Oct 30 01:03:57.736344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.736312    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prr9p\" (UniqueName: \"kubernetes.io/projected/b4733821-6e60-4bd4-bb19-7b16fdb1171d-kube-api-access-prr9p\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-flggq\" (UID: \"b4733821-6e60-4bd4-bb19-7b16fdb1171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"
Oct 30 01:03:57.736504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.736393    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4733821-6e60-4bd4-bb19-7b16fdb1171d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-flggq\" (UID: \"b4733821-6e60-4bd4-bb19-7b16fdb1171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"
Oct 30 01:03:57.736561 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.736547    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4733821-6e60-4bd4-bb19-7b16fdb1171d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-flggq\" (UID: \"b4733821-6e60-4bd4-bb19-7b16fdb1171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"
Oct 30 01:03:57.737145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.737117    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4733821-6e60-4bd4-bb19-7b16fdb1171d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-flggq\" (UID: \"b4733821-6e60-4bd4-bb19-7b16fdb1171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"
Oct 30 01:03:57.738733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.738712    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4733821-6e60-4bd4-bb19-7b16fdb1171d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-flggq\" (UID: \"b4733821-6e60-4bd4-bb19-7b16fdb1171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"
Oct 30 01:03:57.757068 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.757046    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prr9p\" (UniqueName: \"kubernetes.io/projected/b4733821-6e60-4bd4-bb19-7b16fdb1171d-kube-api-access-prr9p\") pod \"maas-api-revoke-keys-e2e-mt-8363a40b-flggq\" (UID: \"b4733821-6e60-4bd4-bb19-7b16fdb1171d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"
Oct 30 01:03:57.857372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:57.857301    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"
Oct 30 01:03:58.162539 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:58.162506    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"]
Oct 30 01:03:58.164327 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:03:58.164300    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4733821_6e60_4bd4_bb19_7b16fdb1171d.slice/crio-e2e5d619d2c0dc98581614df8edd5928cb02005c0185ee80a2d81dc357df6207 WatchSource:0}: Error finding container e2e5d619d2c0dc98581614df8edd5928cb02005c0185ee80a2d81dc357df6207: Status 404 returned error can't find the container with id e2e5d619d2c0dc98581614df8edd5928cb02005c0185ee80a2d81dc357df6207
Oct 30 01:03:58.264723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:58.264690    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq" event={"ID":"b4733821-6e60-4bd4-bb19-7b16fdb1171d","Type":"ContainerStarted","Data":"e2e5d619d2c0dc98581614df8edd5928cb02005c0185ee80a2d81dc357df6207"}
Oct 30 01:03:59.181752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:59.181718    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:03:59.294519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:59.294481    4425 generic.go:356] "Generic (PLEG): container finished" podID="b4733821-6e60-4bd4-bb19-7b16fdb1171d" containerID="33cb77879e33a468e610c077647e648bfa92fe4a846862c2cb66def0b9bed699" exitCode=0
Oct 30 01:03:59.294708 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:03:59.294588    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq" event={"ID":"b4733821-6e60-4bd4-bb19-7b16fdb1171d","Type":"ContainerDied","Data":"33cb77879e33a468e610c077647e648bfa92fe4a846862c2cb66def0b9bed699"}
Oct 30 01:04:00.238434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:00.238395    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"]
Oct 30 01:04:00.290870 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:00.290194    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"]
Oct 30 01:04:00.613734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:00.613708    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"
Oct 30 01:04:00.695231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:00.695196    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4733821-6e60-4bd4-bb19-7b16fdb1171d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4733821-6e60-4bd4-bb19-7b16fdb1171d-maas-api-openshift-ca\") pod \"b4733821-6e60-4bd4-bb19-7b16fdb1171d\" (UID: \"b4733821-6e60-4bd4-bb19-7b16fdb1171d\") "
Oct 30 01:04:00.695411 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:00.695340    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4733821-6e60-4bd4-bb19-7b16fdb1171d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4733821-6e60-4bd4-bb19-7b16fdb1171d-maas-api-xks-ca\") pod \"b4733821-6e60-4bd4-bb19-7b16fdb1171d\" (UID: \"b4733821-6e60-4bd4-bb19-7b16fdb1171d\") "
Oct 30 01:04:00.695527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:00.695504    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4733821-6e60-4bd4-bb19-7b16fdb1171d-kube-api-access-prr9p\" (UniqueName: \"kubernetes.io/projected/b4733821-6e60-4bd4-bb19-7b16fdb1171d-kube-api-access-prr9p\") pod \"b4733821-6e60-4bd4-bb19-7b16fdb1171d\" (UID: \"b4733821-6e60-4bd4-bb19-7b16fdb1171d\") "
Oct 30 01:04:00.696549 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:00.695641    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4733821-6e60-4bd4-bb19-7b16fdb1171d-maas-api-openshift-ca" pod "b4733821-6e60-4bd4-bb19-7b16fdb1171d" (UID: "b4733821-6e60-4bd4-bb19-7b16fdb1171d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:00.696549 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:00.696268    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4733821-6e60-4bd4-bb19-7b16fdb1171d-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:00.698015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:00.697988    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4733821-6e60-4bd4-bb19-7b16fdb1171d-kube-api-access-prr9p" pod "b4733821-6e60-4bd4-bb19-7b16fdb1171d" (UID: "b4733821-6e60-4bd4-bb19-7b16fdb1171d"). InnerVolumeSpecName "kube-api-access-prr9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:00.698216 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:00.698028    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4733821-6e60-4bd4-bb19-7b16fdb1171d-maas-api-xks-ca" pod "b4733821-6e60-4bd4-bb19-7b16fdb1171d" (UID: "b4733821-6e60-4bd4-bb19-7b16fdb1171d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:00.802142 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:00.798856    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4733821-6e60-4bd4-bb19-7b16fdb1171d-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:00.803226 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:00.803195    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prr9p\" (UniqueName: \"kubernetes.io/projected/b4733821-6e60-4bd4-bb19-7b16fdb1171d-kube-api-access-prr9p\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:01.314382 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:01.314353    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"
Oct 30 01:04:01.314876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:01.314382    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq" event={"ID":"b4733821-6e60-4bd4-bb19-7b16fdb1171d","Type":"ContainerDied","Data":"e2e5d619d2c0dc98581614df8edd5928cb02005c0185ee80a2d81dc357df6207"}
Oct 30 01:04:01.314876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:01.314473    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2e5d619d2c0dc98581614df8edd5928cb02005c0185ee80a2d81dc357df6207"
Oct 30 01:04:05.089745 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.089710    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"]
Oct 30 01:04:05.098788 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.098758    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c903a9d4-xcn9n"]
Oct 30 01:04:05.239088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.239036    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz" podUID="bcb580cf-e9bd-4098-9ad4-6c72b709f5d4" containerName="payload-processing" containerID="cri-o://96b5c689544744dbc4bc65aff1b653e4f1738f77e88c4ad6b541caeeb39723b2" gracePeriod=25
Oct 30 01:04:05.291260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.291216    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7" podUID="8368346f-c6ee-4efa-bf21-31affa8e8b35" containerName="payload-pre-processing" containerID="cri-o://04885a0b12da457cd99c53c27c6366af7e5351546094b52c1c90b69acb360fb6" gracePeriod=25
Oct 30 01:04:05.428204 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.426690    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad" path="/var/lib/kubelet/pods/f9c9c70b-b6c8-43e7-b276-43b1b5ac20ad/volumes"
Oct 30 01:04:05.633117 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.633085    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"
Oct 30 01:04:05.671120 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.671095    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"
Oct 30 01:04:05.689810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.689771    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb580cf-e9bd-4098-9ad4-6c72b709f5d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcb580cf-e9bd-4098-9ad4-6c72b709f5d4-plugins-config-volume\") pod \"bcb580cf-e9bd-4098-9ad4-6c72b709f5d4\" (UID: \"bcb580cf-e9bd-4098-9ad4-6c72b709f5d4\") "
Oct 30 01:04:05.689976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.689839    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb580cf-e9bd-4098-9ad4-6c72b709f5d4-kube-api-access-7v5sd\" (UniqueName: \"kubernetes.io/projected/bcb580cf-e9bd-4098-9ad4-6c72b709f5d4-kube-api-access-7v5sd\") pod \"bcb580cf-e9bd-4098-9ad4-6c72b709f5d4\" (UID: \"bcb580cf-e9bd-4098-9ad4-6c72b709f5d4\") "
Oct 30 01:04:05.690198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.690172    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb580cf-e9bd-4098-9ad4-6c72b709f5d4-plugins-config-volume" pod "bcb580cf-e9bd-4098-9ad4-6c72b709f5d4" (UID: "bcb580cf-e9bd-4098-9ad4-6c72b709f5d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:05.690736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.690707    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcb580cf-e9bd-4098-9ad4-6c72b709f5d4-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:05.692146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.692118    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb580cf-e9bd-4098-9ad4-6c72b709f5d4-kube-api-access-7v5sd" pod "bcb580cf-e9bd-4098-9ad4-6c72b709f5d4" (UID: "bcb580cf-e9bd-4098-9ad4-6c72b709f5d4"). InnerVolumeSpecName "kube-api-access-7v5sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:05.791463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.791421    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8368346f-c6ee-4efa-bf21-31affa8e8b35-kube-api-access-zfzbk\" (UniqueName: \"kubernetes.io/projected/8368346f-c6ee-4efa-bf21-31affa8e8b35-kube-api-access-zfzbk\") pod \"8368346f-c6ee-4efa-bf21-31affa8e8b35\" (UID: \"8368346f-c6ee-4efa-bf21-31affa8e8b35\") "
Oct 30 01:04:05.791637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.791492    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8368346f-c6ee-4efa-bf21-31affa8e8b35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8368346f-c6ee-4efa-bf21-31affa8e8b35-plugins-config-volume\") pod \"8368346f-c6ee-4efa-bf21-31affa8e8b35\" (UID: \"8368346f-c6ee-4efa-bf21-31affa8e8b35\") "
Oct 30 01:04:05.791946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.791916    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8368346f-c6ee-4efa-bf21-31affa8e8b35-plugins-config-volume" pod "8368346f-c6ee-4efa-bf21-31affa8e8b35" (UID: "8368346f-c6ee-4efa-bf21-31affa8e8b35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:05.792135 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.792120    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8368346f-c6ee-4efa-bf21-31affa8e8b35-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:05.792205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.792137    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v5sd\" (UniqueName: \"kubernetes.io/projected/bcb580cf-e9bd-4098-9ad4-6c72b709f5d4-kube-api-access-7v5sd\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:05.793619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.793592    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8368346f-c6ee-4efa-bf21-31affa8e8b35-kube-api-access-zfzbk" pod "8368346f-c6ee-4efa-bf21-31affa8e8b35" (UID: "8368346f-c6ee-4efa-bf21-31affa8e8b35"). InnerVolumeSpecName "kube-api-access-zfzbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:05.893582 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:05.893550    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfzbk\" (UniqueName: \"kubernetes.io/projected/8368346f-c6ee-4efa-bf21-31affa8e8b35-kube-api-access-zfzbk\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:06.358747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.358716    4425 generic.go:356] "Generic (PLEG): container finished" podID="bcb580cf-e9bd-4098-9ad4-6c72b709f5d4" containerID="96b5c689544744dbc4bc65aff1b653e4f1738f77e88c4ad6b541caeeb39723b2" exitCode=0
Oct 30 01:04:06.359225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.358770    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"
Oct 30 01:04:06.359225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.358792    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz" event={"ID":"bcb580cf-e9bd-4098-9ad4-6c72b709f5d4","Type":"ContainerDied","Data":"96b5c689544744dbc4bc65aff1b653e4f1738f77e88c4ad6b541caeeb39723b2"}
Oct 30 01:04:06.359225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.358871    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz" event={"ID":"bcb580cf-e9bd-4098-9ad4-6c72b709f5d4","Type":"ContainerDied","Data":"f22ebf246d4277d08cb6fda16f42de4a6e639504eba99e83591d9ae1d8518de9"}
Oct 30 01:04:06.359225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.358903    4425 scope.go:122] "RemoveContainer" containerID="96b5c689544744dbc4bc65aff1b653e4f1738f77e88c4ad6b541caeeb39723b2"
Oct 30 01:04:06.360368 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.360350    4425 generic.go:356] "Generic (PLEG): container finished" podID="8368346f-c6ee-4efa-bf21-31affa8e8b35" containerID="04885a0b12da457cd99c53c27c6366af7e5351546094b52c1c90b69acb360fb6" exitCode=0
Oct 30 01:04:06.360463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.360447    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"
Oct 30 01:04:06.360546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.360443    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7" event={"ID":"8368346f-c6ee-4efa-bf21-31affa8e8b35","Type":"ContainerDied","Data":"04885a0b12da457cd99c53c27c6366af7e5351546094b52c1c90b69acb360fb6"}
Oct 30 01:04:06.360616 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.360570    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7" event={"ID":"8368346f-c6ee-4efa-bf21-31affa8e8b35","Type":"ContainerDied","Data":"00e8a408f35d3ca79d48a0f69989ed0a4961fac616ad33d7652e862e81e33ee8"}
Oct 30 01:04:06.376435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.372376    4425 scope.go:122] "RemoveContainer" containerID="96b5c689544744dbc4bc65aff1b653e4f1738f77e88c4ad6b541caeeb39723b2"
Oct 30 01:04:06.376435 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:04:06.372681    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b5c689544744dbc4bc65aff1b653e4f1738f77e88c4ad6b541caeeb39723b2\": container with ID starting with 96b5c689544744dbc4bc65aff1b653e4f1738f77e88c4ad6b541caeeb39723b2 not found: ID does not exist" containerID="96b5c689544744dbc4bc65aff1b653e4f1738f77e88c4ad6b541caeeb39723b2"
Oct 30 01:04:06.376435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.372721    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b5c689544744dbc4bc65aff1b653e4f1738f77e88c4ad6b541caeeb39723b2"} err="failed to get container status \"96b5c689544744dbc4bc65aff1b653e4f1738f77e88c4ad6b541caeeb39723b2\": rpc error: code = NotFound desc = could not find container \"96b5c689544744dbc4bc65aff1b653e4f1738f77e88c4ad6b541caeeb39723b2\": container with ID starting with 96b5c689544744dbc4bc65aff1b653e4f1738f77e88c4ad6b541caeeb39723b2 not found: ID does not exist"
Oct 30 01:04:06.376435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.372739    4425 scope.go:122] "RemoveContainer" containerID="04885a0b12da457cd99c53c27c6366af7e5351546094b52c1c90b69acb360fb6"
Oct 30 01:04:06.387374 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.387337    4425 scope.go:122] "RemoveContainer" containerID="04885a0b12da457cd99c53c27c6366af7e5351546094b52c1c90b69acb360fb6"
Oct 30 01:04:06.387860 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:04:06.387800    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04885a0b12da457cd99c53c27c6366af7e5351546094b52c1c90b69acb360fb6\": container with ID starting with 04885a0b12da457cd99c53c27c6366af7e5351546094b52c1c90b69acb360fb6 not found: ID does not exist" containerID="04885a0b12da457cd99c53c27c6366af7e5351546094b52c1c90b69acb360fb6"
Oct 30 01:04:06.387860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.387840    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04885a0b12da457cd99c53c27c6366af7e5351546094b52c1c90b69acb360fb6"} err="failed to get container status \"04885a0b12da457cd99c53c27c6366af7e5351546094b52c1c90b69acb360fb6\": rpc error: code = NotFound desc = could not find container \"04885a0b12da457cd99c53c27c6366af7e5351546094b52c1c90b69acb360fb6\": container with ID starting with 04885a0b12da457cd99c53c27c6366af7e5351546094b52c1c90b69acb360fb6 not found: ID does not exist"
Oct 30 01:04:06.415279 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.415227    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"]
Oct 30 01:04:06.428858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.428821    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9a8d85-6bb6b9bb87-5krt7"]
Oct 30 01:04:06.450668 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.450631    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"]
Oct 30 01:04:06.456191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:06.456128    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9a8d85-b84dc7694-m2ksz"]
Oct 30 01:04:07.423220 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:07.423191    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8368346f-c6ee-4efa-bf21-31affa8e8b35" path="/var/lib/kubelet/pods/8368346f-c6ee-4efa-bf21-31affa8e8b35/volumes"
Oct 30 01:04:07.423637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:07.423623    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb580cf-e9bd-4098-9ad4-6c72b709f5d4" path="/var/lib/kubelet/pods/bcb580cf-e9bd-4098-9ad4-6c72b709f5d4/volumes"
Oct 30 01:04:07.555285 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:04:07.555246    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:04:07.956394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:07.956365    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"]
Oct 30 01:04:07.983002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:07.982972    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"]
Oct 30 01:04:10.759496 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.759462    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"]
Oct 30 01:04:10.760078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.760064    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8368346f-c6ee-4efa-bf21-31affa8e8b35" containerName="payload-pre-processing"
Oct 30 01:04:10.760124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.760079    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8368346f-c6ee-4efa-bf21-31affa8e8b35" containerName="payload-pre-processing"
Oct 30 01:04:10.760124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.760089    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b4733821-6e60-4bd4-bb19-7b16fdb1171d" containerName="revoke-keys"
Oct 30 01:04:10.760124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.760094    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4733821-6e60-4bd4-bb19-7b16fdb1171d" containerName="revoke-keys"
Oct 30 01:04:10.760124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.760117    4425 cpu_manager.go:414] "containerMap: removing container" podUID="bcb580cf-e9bd-4098-9ad4-6c72b709f5d4" containerName="payload-processing"
Oct 30 01:04:10.760124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.760122    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb580cf-e9bd-4098-9ad4-6c72b709f5d4" containerName="payload-processing"
Oct 30 01:04:10.760294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.760267    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb580cf-e9bd-4098-9ad4-6c72b709f5d4" containerName="payload-processing"
Oct 30 01:04:10.760294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.760288    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8368346f-c6ee-4efa-bf21-31affa8e8b35" containerName="payload-pre-processing"
Oct 30 01:04:10.760294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.760294    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4733821-6e60-4bd4-bb19-7b16fdb1171d" containerName="revoke-keys"
Oct 30 01:04:10.765242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.765218    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"
Oct 30 01:04:10.780841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.780819    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:04:10.785843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.785817    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"]
Oct 30 01:04:10.790657 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.790633    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:04:10.790657 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.790633    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:04:10.790825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.790661    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:04:10.864932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.864898    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6nw\" (UniqueName: \"kubernetes.io/projected/278cc048-1556-407a-9e61-98799928615d-kube-api-access-ck6nw\") pod \"maas-api-revoke-sub-f8a6bc3f-nhr4h\" (UID: \"278cc048-1556-407a-9e61-98799928615d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"
Oct 30 01:04:10.865184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.865040    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/278cc048-1556-407a-9e61-98799928615d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8a6bc3f-nhr4h\" (UID: \"278cc048-1556-407a-9e61-98799928615d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"
Oct 30 01:04:10.865295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.865274    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/278cc048-1556-407a-9e61-98799928615d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8a6bc3f-nhr4h\" (UID: \"278cc048-1556-407a-9e61-98799928615d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"
Oct 30 01:04:10.966327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.966291    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/278cc048-1556-407a-9e61-98799928615d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8a6bc3f-nhr4h\" (UID: \"278cc048-1556-407a-9e61-98799928615d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"
Oct 30 01:04:10.966550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.966390    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6nw\" (UniqueName: \"kubernetes.io/projected/278cc048-1556-407a-9e61-98799928615d-kube-api-access-ck6nw\") pod \"maas-api-revoke-sub-f8a6bc3f-nhr4h\" (UID: \"278cc048-1556-407a-9e61-98799928615d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"
Oct 30 01:04:10.966550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.966491    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/278cc048-1556-407a-9e61-98799928615d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8a6bc3f-nhr4h\" (UID: \"278cc048-1556-407a-9e61-98799928615d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"
Oct 30 01:04:10.966966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.966944    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/278cc048-1556-407a-9e61-98799928615d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8a6bc3f-nhr4h\" (UID: \"278cc048-1556-407a-9e61-98799928615d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"
Oct 30 01:04:10.968966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.968944    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/278cc048-1556-407a-9e61-98799928615d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8a6bc3f-nhr4h\" (UID: \"278cc048-1556-407a-9e61-98799928615d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"
Oct 30 01:04:10.987514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:10.987489    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6nw\" (UniqueName: \"kubernetes.io/projected/278cc048-1556-407a-9e61-98799928615d-kube-api-access-ck6nw\") pod \"maas-api-revoke-sub-f8a6bc3f-nhr4h\" (UID: \"278cc048-1556-407a-9e61-98799928615d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"
Oct 30 01:04:11.041305 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:11.041235    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"]
Oct 30 01:04:11.045893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:11.045875    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"
Oct 30 01:04:11.064950 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:11.064925    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"]
Oct 30 01:04:11.080455 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:11.080428    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"
Oct 30 01:04:11.169694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:11.169662    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2zj7\" (UniqueName: \"kubernetes.io/projected/78237fc0-d05e-4377-8727-0aa79b414b05-kube-api-access-v2zj7\") pod \"maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl\" (UID: \"78237fc0-d05e-4377-8727-0aa79b414b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"
Oct 30 01:04:11.169845 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:11.169795    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78237fc0-d05e-4377-8727-0aa79b414b05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl\" (UID: \"78237fc0-d05e-4377-8727-0aa79b414b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"
Oct 30 01:04:11.169845 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:11.169832    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78237fc0-d05e-4377-8727-0aa79b414b05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl\" (UID: \"78237fc0-d05e-4377-8727-0aa79b414b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"
Oct 30 01:04:11.271291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:11.271260    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78237fc0-d05e-4377-8727-0aa79b414b05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl\" (UID: \"78237fc0-d05e-4377-8727-0aa79b414b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"
Oct 30 01:04:11.271470 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:11.271310    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78237fc0-d05e-4377-8727-0aa79b414b05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl\" (UID: \"78237fc0-d05e-4377-8727-0aa79b414b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"
Oct 30 01:04:11.271470 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:11.271417    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2zj7\" (UniqueName: \"kubernetes.io/projected/78237fc0-d05e-4377-8727-0aa79b414b05-kube-api-access-v2zj7\") pod \"maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl\" (UID: \"78237fc0-d05e-4377-8727-0aa79b414b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"
Oct 30 01:04:11.272089 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:11.272056    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78237fc0-d05e-4377-8727-0aa79b414b05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl\" (UID: \"78237fc0-d05e-4377-8727-0aa79b414b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"
Oct 30 01:04:11.273564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:11.273541    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78237fc0-d05e-4377-8727-0aa79b414b05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl\" (UID: \"78237fc0-d05e-4377-8727-0aa79b414b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"
Oct 30 01:04:11.291861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:11.291808    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2zj7\" (UniqueName: \"kubernetes.io/projected/78237fc0-d05e-4377-8727-0aa79b414b05-kube-api-access-v2zj7\") pod \"maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl\" (UID: \"78237fc0-d05e-4377-8727-0aa79b414b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"
Oct 30 01:04:11.363057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:11.363032    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"
Oct 30 01:04:11.395462 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:04:11.395433    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278cc048_1556_407a_9e61_98799928615d.slice/crio-4c873bce9586df6a995c1b3da3f4735d10f9fff58bbc92a4c587d57daf107970 WatchSource:0}: Error finding container 4c873bce9586df6a995c1b3da3f4735d10f9fff58bbc92a4c587d57daf107970: Status 404 returned error can't find the container with id 4c873bce9586df6a995c1b3da3f4735d10f9fff58bbc92a4c587d57daf107970
Oct 30 01:04:11.395579 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:11.395557    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"]
Oct 30 01:04:11.675500 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:04:11.675474    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78237fc0_d05e_4377_8727_0aa79b414b05.slice/crio-1a440978bd5820d687df97b795fd13522eeee9203bf4344bd93b02bf007ea859 WatchSource:0}: Error finding container 1a440978bd5820d687df97b795fd13522eeee9203bf4344bd93b02bf007ea859: Status 404 returned error can't find the container with id 1a440978bd5820d687df97b795fd13522eeee9203bf4344bd93b02bf007ea859
Oct 30 01:04:11.675770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:11.675747    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"]
Oct 30 01:04:12.406494 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:12.406460    4425 generic.go:356] "Generic (PLEG): container finished" podID="278cc048-1556-407a-9e61-98799928615d" containerID="d7fd2449972d87c17ccc68a6db36ed683c7ffda2703c98c6983171a720c97c28" exitCode=0
Oct 30 01:04:12.406988 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:12.406542    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h" event={"ID":"278cc048-1556-407a-9e61-98799928615d","Type":"ContainerDied","Data":"d7fd2449972d87c17ccc68a6db36ed683c7ffda2703c98c6983171a720c97c28"}
Oct 30 01:04:12.406988 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:12.406592    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h" event={"ID":"278cc048-1556-407a-9e61-98799928615d","Type":"ContainerStarted","Data":"4c873bce9586df6a995c1b3da3f4735d10f9fff58bbc92a4c587d57daf107970"}
Oct 30 01:04:12.408271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:12.408248    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl" event={"ID":"78237fc0-d05e-4377-8727-0aa79b414b05","Type":"ContainerStarted","Data":"9ed5508f286891875cfcbb8500f759b5aeeac5fd037bb5cf7312fb5fd995ee62"}
Oct 30 01:04:12.408414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:12.408294    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl" event={"ID":"78237fc0-d05e-4377-8727-0aa79b414b05","Type":"ContainerStarted","Data":"1a440978bd5820d687df97b795fd13522eeee9203bf4344bd93b02bf007ea859"}
Oct 30 01:04:12.478082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:12.478034    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl" podStartSLOduration=1.478020419 podStartE2EDuration="1.478020419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:12.476964202 +0000 UTC m=+1361.226619509" watchObservedRunningTime="2026-10-30 01:04:12.478020419 +0000 UTC m=+1361.227675726"
Oct 30 01:04:12.957518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:12.957428    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq" podUID="c963677e-9e3a-4708-be7d-0fa8202a70ef" containerName="payload-processing" containerID="cri-o://0c1043c6e254c10a60c82435ad8793971f33d8dff46b0a004df22874b9bf0c54" gracePeriod=25
Oct 30 01:04:12.983813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:12.983781    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24" podUID="9f3634c9-1768-46e5-9979-9738b4486e10" containerName="payload-pre-processing" containerID="cri-o://ece0cc0ff69a41aa2e74cb2f0cbbbefec743ea204c1bbfc28e3fc867cf75807c" gracePeriod=25
Oct 30 01:04:13.258126 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.258094    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"
Oct 30 01:04:13.335749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.335719    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"
Oct 30 01:04:13.399774 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.399729    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c963677e-9e3a-4708-be7d-0fa8202a70ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c963677e-9e3a-4708-be7d-0fa8202a70ef-plugins-config-volume\") pod \"c963677e-9e3a-4708-be7d-0fa8202a70ef\" (UID: \"c963677e-9e3a-4708-be7d-0fa8202a70ef\") "
Oct 30 01:04:13.400033 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.400011    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c963677e-9e3a-4708-be7d-0fa8202a70ef-kube-api-access-6w6bx\" (UniqueName: \"kubernetes.io/projected/c963677e-9e3a-4708-be7d-0fa8202a70ef-kube-api-access-6w6bx\") pod \"c963677e-9e3a-4708-be7d-0fa8202a70ef\" (UID: \"c963677e-9e3a-4708-be7d-0fa8202a70ef\") "
Oct 30 01:04:13.400250 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.400219    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c963677e-9e3a-4708-be7d-0fa8202a70ef-plugins-config-volume" pod "c963677e-9e3a-4708-be7d-0fa8202a70ef" (UID: "c963677e-9e3a-4708-be7d-0fa8202a70ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:13.400767 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.400743    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c963677e-9e3a-4708-be7d-0fa8202a70ef-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:13.403093 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.403068    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c963677e-9e3a-4708-be7d-0fa8202a70ef-kube-api-access-6w6bx" pod "c963677e-9e3a-4708-be7d-0fa8202a70ef" (UID: "c963677e-9e3a-4708-be7d-0fa8202a70ef"). InnerVolumeSpecName "kube-api-access-6w6bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:13.419483 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.419449    4425 generic.go:356] "Generic (PLEG): container finished" podID="78237fc0-d05e-4377-8727-0aa79b414b05" containerID="9ed5508f286891875cfcbb8500f759b5aeeac5fd037bb5cf7312fb5fd995ee62" exitCode=0
Oct 30 01:04:13.422021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.421991    4425 generic.go:356] "Generic (PLEG): container finished" podID="9f3634c9-1768-46e5-9979-9738b4486e10" containerID="ece0cc0ff69a41aa2e74cb2f0cbbbefec743ea204c1bbfc28e3fc867cf75807c" exitCode=0
Oct 30 01:04:13.422208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.422190    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"
Oct 30 01:04:13.423906 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.423884    4425 generic.go:356] "Generic (PLEG): container finished" podID="c963677e-9e3a-4708-be7d-0fa8202a70ef" containerID="0c1043c6e254c10a60c82435ad8793971f33d8dff46b0a004df22874b9bf0c54" exitCode=0
Oct 30 01:04:13.424019 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.423940    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"
Oct 30 01:04:13.429093 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.429063    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl" event={"ID":"78237fc0-d05e-4377-8727-0aa79b414b05","Type":"ContainerDied","Data":"9ed5508f286891875cfcbb8500f759b5aeeac5fd037bb5cf7312fb5fd995ee62"}
Oct 30 01:04:13.429252 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.429233    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24" event={"ID":"9f3634c9-1768-46e5-9979-9738b4486e10","Type":"ContainerDied","Data":"ece0cc0ff69a41aa2e74cb2f0cbbbefec743ea204c1bbfc28e3fc867cf75807c"}
Oct 30 01:04:13.429331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.429267    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24" event={"ID":"9f3634c9-1768-46e5-9979-9738b4486e10","Type":"ContainerDied","Data":"e2801e5c44489eb590b84110d36b3dbde38ce3f4f86af539c358bac474408dae"}
Oct 30 01:04:13.429331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.429290    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq" event={"ID":"c963677e-9e3a-4708-be7d-0fa8202a70ef","Type":"ContainerDied","Data":"0c1043c6e254c10a60c82435ad8793971f33d8dff46b0a004df22874b9bf0c54"}
Oct 30 01:04:13.429331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.429311    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq" event={"ID":"c963677e-9e3a-4708-be7d-0fa8202a70ef","Type":"ContainerDied","Data":"7a0e4a5870b01d7222c1d795ec4c6e19ffc1256ca5ec9a32b6f3d60f9b5ae5c5"}
Oct 30 01:04:13.429479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.429341    4425 scope.go:122] "RemoveContainer" containerID="ece0cc0ff69a41aa2e74cb2f0cbbbefec743ea204c1bbfc28e3fc867cf75807c"
Oct 30 01:04:13.442696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.442679    4425 scope.go:122] "RemoveContainer" containerID="ece0cc0ff69a41aa2e74cb2f0cbbbefec743ea204c1bbfc28e3fc867cf75807c"
Oct 30 01:04:13.442971 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:04:13.442952    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece0cc0ff69a41aa2e74cb2f0cbbbefec743ea204c1bbfc28e3fc867cf75807c\": container with ID starting with ece0cc0ff69a41aa2e74cb2f0cbbbefec743ea204c1bbfc28e3fc867cf75807c not found: ID does not exist" containerID="ece0cc0ff69a41aa2e74cb2f0cbbbefec743ea204c1bbfc28e3fc867cf75807c"
Oct 30 01:04:13.443043 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.442981    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece0cc0ff69a41aa2e74cb2f0cbbbefec743ea204c1bbfc28e3fc867cf75807c"} err="failed to get container status \"ece0cc0ff69a41aa2e74cb2f0cbbbefec743ea204c1bbfc28e3fc867cf75807c\": rpc error: code = NotFound desc = could not find container \"ece0cc0ff69a41aa2e74cb2f0cbbbefec743ea204c1bbfc28e3fc867cf75807c\": container with ID starting with ece0cc0ff69a41aa2e74cb2f0cbbbefec743ea204c1bbfc28e3fc867cf75807c not found: ID does not exist"
Oct 30 01:04:13.443043 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.443003    4425 scope.go:122] "RemoveContainer" containerID="0c1043c6e254c10a60c82435ad8793971f33d8dff46b0a004df22874b9bf0c54"
Oct 30 01:04:13.460400 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.459827    4425 scope.go:122] "RemoveContainer" containerID="0c1043c6e254c10a60c82435ad8793971f33d8dff46b0a004df22874b9bf0c54"
Oct 30 01:04:13.460650 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:04:13.460617    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c1043c6e254c10a60c82435ad8793971f33d8dff46b0a004df22874b9bf0c54\": container with ID starting with 0c1043c6e254c10a60c82435ad8793971f33d8dff46b0a004df22874b9bf0c54 not found: ID does not exist" containerID="0c1043c6e254c10a60c82435ad8793971f33d8dff46b0a004df22874b9bf0c54"
Oct 30 01:04:13.460739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.460660    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1043c6e254c10a60c82435ad8793971f33d8dff46b0a004df22874b9bf0c54"} err="failed to get container status \"0c1043c6e254c10a60c82435ad8793971f33d8dff46b0a004df22874b9bf0c54\": rpc error: code = NotFound desc = could not find container \"0c1043c6e254c10a60c82435ad8793971f33d8dff46b0a004df22874b9bf0c54\": container with ID starting with 0c1043c6e254c10a60c82435ad8793971f33d8dff46b0a004df22874b9bf0c54 not found: ID does not exist"
Oct 30 01:04:13.502752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.502719    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f3634c9-1768-46e5-9979-9738b4486e10-kube-api-access-8fn47\" (UniqueName: \"kubernetes.io/projected/9f3634c9-1768-46e5-9979-9738b4486e10-kube-api-access-8fn47\") pod \"9f3634c9-1768-46e5-9979-9738b4486e10\" (UID: \"9f3634c9-1768-46e5-9979-9738b4486e10\") "
Oct 30 01:04:13.503835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.503807    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f3634c9-1768-46e5-9979-9738b4486e10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f3634c9-1768-46e5-9979-9738b4486e10-plugins-config-volume\") pod \"9f3634c9-1768-46e5-9979-9738b4486e10\" (UID: \"9f3634c9-1768-46e5-9979-9738b4486e10\") "
Oct 30 01:04:13.504509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.504353    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f3634c9-1768-46e5-9979-9738b4486e10-plugins-config-volume" pod "9f3634c9-1768-46e5-9979-9738b4486e10" (UID: "9f3634c9-1768-46e5-9979-9738b4486e10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:13.504509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.504468    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f3634c9-1768-46e5-9979-9738b4486e10-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:13.504509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.504486    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w6bx\" (UniqueName: \"kubernetes.io/projected/c963677e-9e3a-4708-be7d-0fa8202a70ef-kube-api-access-6w6bx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:13.508460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.508422    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f3634c9-1768-46e5-9979-9738b4486e10-kube-api-access-8fn47" pod "9f3634c9-1768-46e5-9979-9738b4486e10" (UID: "9f3634c9-1768-46e5-9979-9738b4486e10"). InnerVolumeSpecName "kube-api-access-8fn47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:13.515479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.515454    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"]
Oct 30 01:04:13.520115 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.520099    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8363a40b-777cf99df8-zzljq"]
Oct 30 01:04:13.578398 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.578375    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"
Oct 30 01:04:13.606865 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.606822    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fn47\" (UniqueName: \"kubernetes.io/projected/9f3634c9-1768-46e5-9979-9738b4486e10-kube-api-access-8fn47\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:13.707880 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.707851    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/278cc048-1556-407a-9e61-98799928615d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/278cc048-1556-407a-9e61-98799928615d-maas-api-xks-ca\") pod \"278cc048-1556-407a-9e61-98799928615d\" (UID: \"278cc048-1556-407a-9e61-98799928615d\") "
Oct 30 01:04:13.708076 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.707995    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278cc048-1556-407a-9e61-98799928615d-kube-api-access-ck6nw\" (UniqueName: \"kubernetes.io/projected/278cc048-1556-407a-9e61-98799928615d-kube-api-access-ck6nw\") pod \"278cc048-1556-407a-9e61-98799928615d\" (UID: \"278cc048-1556-407a-9e61-98799928615d\") "
Oct 30 01:04:13.708144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.708133    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/278cc048-1556-407a-9e61-98799928615d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/278cc048-1556-407a-9e61-98799928615d-maas-api-openshift-ca\") pod \"278cc048-1556-407a-9e61-98799928615d\" (UID: \"278cc048-1556-407a-9e61-98799928615d\") "
Oct 30 01:04:13.708459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.708432    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/278cc048-1556-407a-9e61-98799928615d-maas-api-openshift-ca" pod "278cc048-1556-407a-9e61-98799928615d" (UID: "278cc048-1556-407a-9e61-98799928615d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:13.708874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.708847    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/278cc048-1556-407a-9e61-98799928615d-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:13.710022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.709997    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/278cc048-1556-407a-9e61-98799928615d-maas-api-xks-ca" pod "278cc048-1556-407a-9e61-98799928615d" (UID: "278cc048-1556-407a-9e61-98799928615d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:13.710186 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.710145    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278cc048-1556-407a-9e61-98799928615d-kube-api-access-ck6nw" pod "278cc048-1556-407a-9e61-98799928615d" (UID: "278cc048-1556-407a-9e61-98799928615d"). InnerVolumeSpecName "kube-api-access-ck6nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:13.768229 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.768188    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"]
Oct 30 01:04:13.777182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.777122    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8363a40b-756859fd87-cgw24"]
Oct 30 01:04:13.810276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.810241    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck6nw\" (UniqueName: \"kubernetes.io/projected/278cc048-1556-407a-9e61-98799928615d-kube-api-access-ck6nw\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:13.810276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:13.810274    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/278cc048-1556-407a-9e61-98799928615d-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:14.433174 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:14.433078    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"
Oct 30 01:04:14.433174 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:14.433114    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h" event={"ID":"278cc048-1556-407a-9e61-98799928615d","Type":"ContainerDied","Data":"4c873bce9586df6a995c1b3da3f4735d10f9fff58bbc92a4c587d57daf107970"}
Oct 30 01:04:14.433634 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:14.433186    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c873bce9586df6a995c1b3da3f4735d10f9fff58bbc92a4c587d57daf107970"
Oct 30 01:04:14.603741 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:14.603717    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"
Oct 30 01:04:14.721780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:14.721700    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78237fc0-d05e-4377-8727-0aa79b414b05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78237fc0-d05e-4377-8727-0aa79b414b05-maas-api-openshift-ca\") pod \"78237fc0-d05e-4377-8727-0aa79b414b05\" (UID: \"78237fc0-d05e-4377-8727-0aa79b414b05\") "
Oct 30 01:04:14.721912 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:14.721844    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78237fc0-d05e-4377-8727-0aa79b414b05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78237fc0-d05e-4377-8727-0aa79b414b05-maas-api-xks-ca\") pod \"78237fc0-d05e-4377-8727-0aa79b414b05\" (UID: \"78237fc0-d05e-4377-8727-0aa79b414b05\") "
Oct 30 01:04:14.722078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:14.722057    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78237fc0-d05e-4377-8727-0aa79b414b05-maas-api-openshift-ca" pod "78237fc0-d05e-4377-8727-0aa79b414b05" (UID: "78237fc0-d05e-4377-8727-0aa79b414b05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:14.722123 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:14.722074    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78237fc0-d05e-4377-8727-0aa79b414b05-kube-api-access-v2zj7\" (UniqueName: \"kubernetes.io/projected/78237fc0-d05e-4377-8727-0aa79b414b05-kube-api-access-v2zj7\") pod \"78237fc0-d05e-4377-8727-0aa79b414b05\" (UID: \"78237fc0-d05e-4377-8727-0aa79b414b05\") "
Oct 30 01:04:14.722546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:14.722528    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78237fc0-d05e-4377-8727-0aa79b414b05-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:14.723836 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:14.723815    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78237fc0-d05e-4377-8727-0aa79b414b05-maas-api-xks-ca" pod "78237fc0-d05e-4377-8727-0aa79b414b05" (UID: "78237fc0-d05e-4377-8727-0aa79b414b05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:14.724134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:14.724114    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78237fc0-d05e-4377-8727-0aa79b414b05-kube-api-access-v2zj7" pod "78237fc0-d05e-4377-8727-0aa79b414b05" (UID: "78237fc0-d05e-4377-8727-0aa79b414b05"). InnerVolumeSpecName "kube-api-access-v2zj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:14.823679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:14.823648    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2zj7\" (UniqueName: \"kubernetes.io/projected/78237fc0-d05e-4377-8727-0aa79b414b05-kube-api-access-v2zj7\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:14.823679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:14.823678    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78237fc0-d05e-4377-8727-0aa79b414b05-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:15.422981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:15.422950    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f3634c9-1768-46e5-9979-9738b4486e10" path="/var/lib/kubelet/pods/9f3634c9-1768-46e5-9979-9738b4486e10/volumes"
Oct 30 01:04:15.423456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:15.423441    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c963677e-9e3a-4708-be7d-0fa8202a70ef" path="/var/lib/kubelet/pods/c963677e-9e3a-4708-be7d-0fa8202a70ef/volumes"
Oct 30 01:04:15.443873 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:15.443844    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"
Oct 30 01:04:15.444420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:15.443976    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl" event={"ID":"78237fc0-d05e-4377-8727-0aa79b414b05","Type":"ContainerDied","Data":"1a440978bd5820d687df97b795fd13522eeee9203bf4344bd93b02bf007ea859"}
Oct 30 01:04:15.444420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:15.444022    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a440978bd5820d687df97b795fd13522eeee9203bf4344bd93b02bf007ea859"
Oct 30 01:04:21.505626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:21.505585    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"]
Oct 30 01:04:21.536805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:21.536773    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"]
Oct 30 01:04:25.155618 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:25.155581    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"]
Oct 30 01:04:25.164775 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:25.164751    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8363a40b-flggq"]
Oct 30 01:04:25.425577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:25.425492    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4733821-6e60-4bd4-bb19-7b16fdb1171d" path="/var/lib/kubelet/pods/b4733821-6e60-4bd4-bb19-7b16fdb1171d/volumes"
Oct 30 01:04:26.506317 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:26.506277    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2" podUID="e23c4d7c-5787-4c5b-ad47-9a6715c1d93d" containerName="payload-processing" containerID="cri-o://8b055146cffca4b70ef0856379a6d32800bdb01f5f83a27dd7f08887e02f1c86" gracePeriod=25
Oct 30 01:04:26.537068 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:26.537030    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47" podUID="c6580527-dd10-4e58-9330-58460660831a" containerName="payload-pre-processing" containerID="cri-o://83872a30949313ddd9ac74031bb7051c0fc80984bb2028ccc126e2f67bdad437" gracePeriod=25
Oct 30 01:04:26.827865 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:26.827834    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"
Oct 30 01:04:26.872871 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:26.872848    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"
Oct 30 01:04:26.899989 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:26.899954    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23c4d7c-5787-4c5b-ad47-9a6715c1d93d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e23c4d7c-5787-4c5b-ad47-9a6715c1d93d-plugins-config-volume\") pod \"e23c4d7c-5787-4c5b-ad47-9a6715c1d93d\" (UID: \"e23c4d7c-5787-4c5b-ad47-9a6715c1d93d\") "
Oct 30 01:04:26.900289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:26.900258    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23c4d7c-5787-4c5b-ad47-9a6715c1d93d-kube-api-access-qlt9b\" (UniqueName: \"kubernetes.io/projected/e23c4d7c-5787-4c5b-ad47-9a6715c1d93d-kube-api-access-qlt9b\") pod \"e23c4d7c-5787-4c5b-ad47-9a6715c1d93d\" (UID: \"e23c4d7c-5787-4c5b-ad47-9a6715c1d93d\") "
Oct 30 01:04:26.900438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:26.900384    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23c4d7c-5787-4c5b-ad47-9a6715c1d93d-plugins-config-volume" pod "e23c4d7c-5787-4c5b-ad47-9a6715c1d93d" (UID: "e23c4d7c-5787-4c5b-ad47-9a6715c1d93d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:26.900956 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:26.900933    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e23c4d7c-5787-4c5b-ad47-9a6715c1d93d-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:26.902341 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:26.902316    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23c4d7c-5787-4c5b-ad47-9a6715c1d93d-kube-api-access-qlt9b" pod "e23c4d7c-5787-4c5b-ad47-9a6715c1d93d" (UID: "e23c4d7c-5787-4c5b-ad47-9a6715c1d93d"). InnerVolumeSpecName "kube-api-access-qlt9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:27.001940 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.001905    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6580527-dd10-4e58-9330-58460660831a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6580527-dd10-4e58-9330-58460660831a-plugins-config-volume\") pod \"c6580527-dd10-4e58-9330-58460660831a\" (UID: \"c6580527-dd10-4e58-9330-58460660831a\") "
Oct 30 01:04:27.002112 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.002004    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6580527-dd10-4e58-9330-58460660831a-kube-api-access-gph6k\" (UniqueName: \"kubernetes.io/projected/c6580527-dd10-4e58-9330-58460660831a-kube-api-access-gph6k\") pod \"c6580527-dd10-4e58-9330-58460660831a\" (UID: \"c6580527-dd10-4e58-9330-58460660831a\") "
Oct 30 01:04:27.002329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.002306    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6580527-dd10-4e58-9330-58460660831a-plugins-config-volume" pod "c6580527-dd10-4e58-9330-58460660831a" (UID: "c6580527-dd10-4e58-9330-58460660831a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:27.002703 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.002676    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlt9b\" (UniqueName: \"kubernetes.io/projected/e23c4d7c-5787-4c5b-ad47-9a6715c1d93d-kube-api-access-qlt9b\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:27.002703 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.002697    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6580527-dd10-4e58-9330-58460660831a-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:27.004067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.004048    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6580527-dd10-4e58-9330-58460660831a-kube-api-access-gph6k" pod "c6580527-dd10-4e58-9330-58460660831a" (UID: "c6580527-dd10-4e58-9330-58460660831a"). InnerVolumeSpecName "kube-api-access-gph6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:27.104298 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.104215    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gph6k\" (UniqueName: \"kubernetes.io/projected/c6580527-dd10-4e58-9330-58460660831a-kube-api-access-gph6k\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:27.459193 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:04:27.459144    4425 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6580527_dd10_4e58_9330_58460660831a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23c4d7c_5787_4c5b_ad47_9a6715c1d93d.slice/crio-2c9461df60e229228f345e567ab6d77d96340fd90cfbe6c69706d9acaf517e11\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23c4d7c_5787_4c5b_ad47_9a6715c1d93d.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 01:04:27.534894 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.534859    4425 generic.go:356] "Generic (PLEG): container finished" podID="e23c4d7c-5787-4c5b-ad47-9a6715c1d93d" containerID="8b055146cffca4b70ef0856379a6d32800bdb01f5f83a27dd7f08887e02f1c86" exitCode=0
Oct 30 01:04:27.535303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.534922    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"
Oct 30 01:04:27.535303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.534939    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2" event={"ID":"e23c4d7c-5787-4c5b-ad47-9a6715c1d93d","Type":"ContainerDied","Data":"8b055146cffca4b70ef0856379a6d32800bdb01f5f83a27dd7f08887e02f1c86"}
Oct 30 01:04:27.535303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.534992    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2" event={"ID":"e23c4d7c-5787-4c5b-ad47-9a6715c1d93d","Type":"ContainerDied","Data":"2c9461df60e229228f345e567ab6d77d96340fd90cfbe6c69706d9acaf517e11"}
Oct 30 01:04:27.535303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.535015    4425 scope.go:122] "RemoveContainer" containerID="8b055146cffca4b70ef0856379a6d32800bdb01f5f83a27dd7f08887e02f1c86"
Oct 30 01:04:27.536471 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.536451    4425 generic.go:356] "Generic (PLEG): container finished" podID="c6580527-dd10-4e58-9330-58460660831a" containerID="83872a30949313ddd9ac74031bb7051c0fc80984bb2028ccc126e2f67bdad437" exitCode=0
Oct 30 01:04:27.536561 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.536506    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47" event={"ID":"c6580527-dd10-4e58-9330-58460660831a","Type":"ContainerDied","Data":"83872a30949313ddd9ac74031bb7051c0fc80984bb2028ccc126e2f67bdad437"}
Oct 30 01:04:27.536561 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.536525    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"
Oct 30 01:04:27.536561 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.536546    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47" event={"ID":"c6580527-dd10-4e58-9330-58460660831a","Type":"ContainerDied","Data":"3104ec02f84c63bf878154ea236c929934a5e5eb9f2aceecd687d29da1560ace"}
Oct 30 01:04:27.546947 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.546922    4425 scope.go:122] "RemoveContainer" containerID="8b055146cffca4b70ef0856379a6d32800bdb01f5f83a27dd7f08887e02f1c86"
Oct 30 01:04:27.547340 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:04:27.547318    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b055146cffca4b70ef0856379a6d32800bdb01f5f83a27dd7f08887e02f1c86\": container with ID starting with 8b055146cffca4b70ef0856379a6d32800bdb01f5f83a27dd7f08887e02f1c86 not found: ID does not exist" containerID="8b055146cffca4b70ef0856379a6d32800bdb01f5f83a27dd7f08887e02f1c86"
Oct 30 01:04:27.547444 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.547346    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b055146cffca4b70ef0856379a6d32800bdb01f5f83a27dd7f08887e02f1c86"} err="failed to get container status \"8b055146cffca4b70ef0856379a6d32800bdb01f5f83a27dd7f08887e02f1c86\": rpc error: code = NotFound desc = could not find container \"8b055146cffca4b70ef0856379a6d32800bdb01f5f83a27dd7f08887e02f1c86\": container with ID starting with 8b055146cffca4b70ef0856379a6d32800bdb01f5f83a27dd7f08887e02f1c86 not found: ID does not exist"
Oct 30 01:04:27.547444 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.547363    4425 scope.go:122] "RemoveContainer" containerID="83872a30949313ddd9ac74031bb7051c0fc80984bb2028ccc126e2f67bdad437"
Oct 30 01:04:27.556569 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.556551    4425 scope.go:122] "RemoveContainer" containerID="83872a30949313ddd9ac74031bb7051c0fc80984bb2028ccc126e2f67bdad437"
Oct 30 01:04:27.556808 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:04:27.556790    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83872a30949313ddd9ac74031bb7051c0fc80984bb2028ccc126e2f67bdad437\": container with ID starting with 83872a30949313ddd9ac74031bb7051c0fc80984bb2028ccc126e2f67bdad437 not found: ID does not exist" containerID="83872a30949313ddd9ac74031bb7051c0fc80984bb2028ccc126e2f67bdad437"
Oct 30 01:04:27.556896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.556816    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83872a30949313ddd9ac74031bb7051c0fc80984bb2028ccc126e2f67bdad437"} err="failed to get container status \"83872a30949313ddd9ac74031bb7051c0fc80984bb2028ccc126e2f67bdad437\": rpc error: code = NotFound desc = could not find container \"83872a30949313ddd9ac74031bb7051c0fc80984bb2028ccc126e2f67bdad437\": container with ID starting with 83872a30949313ddd9ac74031bb7051c0fc80984bb2028ccc126e2f67bdad437 not found: ID does not exist"
Oct 30 01:04:27.583090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.583058    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"]
Oct 30 01:04:27.592040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.592014    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9b2c3f-dc6cf7b84-rjtk2"]
Oct 30 01:04:27.615453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.615421    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"]
Oct 30 01:04:27.625477 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:27.625451    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9b2c3f-d9b896df5-c5d47"]
Oct 30 01:04:29.427392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:29.427349    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6580527-dd10-4e58-9330-58460660831a" path="/var/lib/kubelet/pods/c6580527-dd10-4e58-9330-58460660831a/volumes"
Oct 30 01:04:29.428058 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:29.428035    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23c4d7c-5787-4c5b-ad47-9a6715c1d93d" path="/var/lib/kubelet/pods/e23c4d7c-5787-4c5b-ad47-9a6715c1d93d/volumes"
Oct 30 01:04:30.698565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:30.698528    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"]
Oct 30 01:04:31.045253 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:31.040214    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"]
Oct 30 01:04:31.084708 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:31.084662    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9a8b147-5wj8f"]
Oct 30 01:04:31.468438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:31.466043    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365ddc41-5d93-42dd-9b56-6109b3cd0dc7" path="/var/lib/kubelet/pods/365ddc41-5d93-42dd-9b56-6109b3cd0dc7/volumes"
Oct 30 01:04:32.001642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:32.001610    4425 scope.go:122] "RemoveContainer" containerID="35d5b78f4c45e6257ee1c13c7785b10c47b7de753cb6ac361c6020306f3629d7"
Oct 30 01:04:35.731216 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:35.731184    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"]
Oct 30 01:04:35.740352 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:35.740326    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9a8d85-sklhw"]
Oct 30 01:04:37.424199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:37.424136    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c520da-45a6-4afa-b265-1b227e0a5b36" path="/var/lib/kubelet/pods/02c520da-45a6-4afa-b265-1b227e0a5b36/volumes"
Oct 30 01:04:37.555773 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:04:37.555737    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:04:39.367544 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.367515    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"]
Oct 30 01:04:39.368128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368112    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c963677e-9e3a-4708-be7d-0fa8202a70ef" containerName="payload-processing"
Oct 30 01:04:39.368206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368129    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c963677e-9e3a-4708-be7d-0fa8202a70ef" containerName="payload-processing"
Oct 30 01:04:39.368206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368144    4425 cpu_manager.go:414] "containerMap: removing container" podUID="278cc048-1556-407a-9e61-98799928615d" containerName="revoke-keys"
Oct 30 01:04:39.368206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368149    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="278cc048-1556-407a-9e61-98799928615d" containerName="revoke-keys"
Oct 30 01:04:39.368206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368184    4425 cpu_manager.go:414] "containerMap: removing container" podUID="e23c4d7c-5787-4c5b-ad47-9a6715c1d93d" containerName="payload-processing"
Oct 30 01:04:39.368206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368190    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23c4d7c-5787-4c5b-ad47-9a6715c1d93d" containerName="payload-processing"
Oct 30 01:04:39.368206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368196    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9f3634c9-1768-46e5-9979-9738b4486e10" containerName="payload-pre-processing"
Oct 30 01:04:39.368206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368201    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f3634c9-1768-46e5-9979-9738b4486e10" containerName="payload-pre-processing"
Oct 30 01:04:39.368421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368217    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c6580527-dd10-4e58-9330-58460660831a" containerName="payload-pre-processing"
Oct 30 01:04:39.368421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368222    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6580527-dd10-4e58-9330-58460660831a" containerName="payload-pre-processing"
Oct 30 01:04:39.368421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368238    4425 cpu_manager.go:414] "containerMap: removing container" podUID="78237fc0-d05e-4377-8727-0aa79b414b05" containerName="revoke-keys"
Oct 30 01:04:39.368421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368243    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78237fc0-d05e-4377-8727-0aa79b414b05" containerName="revoke-keys"
Oct 30 01:04:39.368421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368353    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c963677e-9e3a-4708-be7d-0fa8202a70ef" containerName="payload-processing"
Oct 30 01:04:39.368421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368363    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="278cc048-1556-407a-9e61-98799928615d" containerName="revoke-keys"
Oct 30 01:04:39.368421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368371    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23c4d7c-5787-4c5b-ad47-9a6715c1d93d" containerName="payload-processing"
Oct 30 01:04:39.368421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368377    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f3634c9-1768-46e5-9979-9738b4486e10" containerName="payload-pre-processing"
Oct 30 01:04:39.368421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368385    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6580527-dd10-4e58-9330-58460660831a" containerName="payload-pre-processing"
Oct 30 01:04:39.368421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.368392    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="78237fc0-d05e-4377-8727-0aa79b414b05" containerName="revoke-keys"
Oct 30 01:04:39.373819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.373800    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"
Oct 30 01:04:39.390737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.390713    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pczrn\" (UniqueName: \"kubernetes.io/projected/ddddff32-cf5c-4b61-8716-fc388b03216f-kube-api-access-pczrn\") pod \"payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv\" (UID: \"ddddff32-cf5c-4b61-8716-fc388b03216f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"
Oct 30 01:04:39.390872 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.390842    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddddff32-cf5c-4b61-8716-fc388b03216f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv\" (UID: \"ddddff32-cf5c-4b61-8716-fc388b03216f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"
Oct 30 01:04:39.392387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.392365    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"]
Oct 30 01:04:39.406024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.404984    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d4622a-dockercfg-5xjlg\""
Oct 30 01:04:39.430627 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.430593    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d4622a\""
Oct 30 01:04:39.519396 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.519340    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pczrn\" (UniqueName: \"kubernetes.io/projected/ddddff32-cf5c-4b61-8716-fc388b03216f-kube-api-access-pczrn\") pod \"payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv\" (UID: \"ddddff32-cf5c-4b61-8716-fc388b03216f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"
Oct 30 01:04:39.519836 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.519814    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddddff32-cf5c-4b61-8716-fc388b03216f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv\" (UID: \"ddddff32-cf5c-4b61-8716-fc388b03216f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"
Oct 30 01:04:39.520932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.520815    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddddff32-cf5c-4b61-8716-fc388b03216f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv\" (UID: \"ddddff32-cf5c-4b61-8716-fc388b03216f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"
Oct 30 01:04:39.540439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.540375    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pczrn\" (UniqueName: \"kubernetes.io/projected/ddddff32-cf5c-4b61-8716-fc388b03216f-kube-api-access-pczrn\") pod \"payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv\" (UID: \"ddddff32-cf5c-4b61-8716-fc388b03216f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"
Oct 30 01:04:39.689474 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.689369    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"
Oct 30 01:04:39.999403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:39.999363    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"]
Oct 30 01:04:40.000110 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:04:40.000080    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddddff32_cf5c_4b61_8716_fc388b03216f.slice/crio-0f57e7d603016e50889fa6f80458724fcfa3b5b29140ffc2dfa83b996262e75c WatchSource:0}: Error finding container 0f57e7d603016e50889fa6f80458724fcfa3b5b29140ffc2dfa83b996262e75c: Status 404 returned error can't find the container with id 0f57e7d603016e50889fa6f80458724fcfa3b5b29140ffc2dfa83b996262e75c
Oct 30 01:04:40.797511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:40.797463    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv" event={"ID":"ddddff32-cf5c-4b61-8716-fc388b03216f","Type":"ContainerStarted","Data":"d2d5d78f3e9c3e7c82242b02a73455ffbb74e619feae90567a349c07b3f66581"}
Oct 30 01:04:40.797928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:40.797574    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv" event={"ID":"ddddff32-cf5c-4b61-8716-fc388b03216f","Type":"ContainerStarted","Data":"0f57e7d603016e50889fa6f80458724fcfa3b5b29140ffc2dfa83b996262e75c"}
Oct 30 01:04:40.797928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:40.797603    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"
Oct 30 01:04:40.835868 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:40.835812    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv" podStartSLOduration=1.8357949900000001 podStartE2EDuration="1.83579499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:40.833548863 +0000 UTC m=+1389.583204172" watchObservedRunningTime="2026-10-30 01:04:40.83579499 +0000 UTC m=+1389.585450300"
Oct 30 01:04:42.033800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:42.033770    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"]
Oct 30 01:04:42.043964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:42.043939    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-771f502b-b5lnt"]
Oct 30 01:04:42.649849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:42.649818    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"]
Oct 30 01:04:42.650031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:42.650011    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp" podUID="4fa39d8a-a4c2-4e0b-b504-feb83231a183" containerName="istio-proxy" containerID="cri-o://eccfc8ad1888fb4d216db977fb250443ce9385d5aa13e349d39a816877b0d4ce" gracePeriod=30
Oct 30 01:04:43.423325 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:43.423293    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47cca8f2-075c-46d2-9d06-07c5de2e2fc8" path="/var/lib/kubelet/pods/47cca8f2-075c-46d2-9d06-07c5de2e2fc8/volumes"
Oct 30 01:04:45.873230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:45.873109    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h" podUID="0f9fbe79-74be-44e1-977c-963f67b0f054" containerName="main" containerID="cri-o://281cfab270ca913f082dad01e974c36e715f36e28265979a8ac43af1f8979afb" gracePeriod=45
Oct 30 01:04:46.151332 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.151307    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:04:46.312935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.312900    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-model-cache\") pod \"0f9fbe79-74be-44e1-977c-963f67b0f054\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") "
Oct 30 01:04:46.313115 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.312961    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-home\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-home\") pod \"0f9fbe79-74be-44e1-977c-963f67b0f054\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") "
Oct 30 01:04:46.313115 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.313077    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-dshm\") pod \"0f9fbe79-74be-44e1-977c-963f67b0f054\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") "
Oct 30 01:04:46.313303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.313145    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9fbe79-74be-44e1-977c-963f67b0f054-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f9fbe79-74be-44e1-977c-963f67b0f054-tls-certs\") pod \"0f9fbe79-74be-44e1-977c-963f67b0f054\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") "
Oct 30 01:04:46.313303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.313255    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9fbe79-74be-44e1-977c-963f67b0f054-kube-api-access-6hk85\" (UniqueName: \"kubernetes.io/projected/0f9fbe79-74be-44e1-977c-963f67b0f054-kube-api-access-6hk85\") pod \"0f9fbe79-74be-44e1-977c-963f67b0f054\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") "
Oct 30 01:04:46.313303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.313261    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-model-cache" pod "0f9fbe79-74be-44e1-977c-963f67b0f054" (UID: "0f9fbe79-74be-44e1-977c-963f67b0f054"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:46.313303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.313278    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-home" pod "0f9fbe79-74be-44e1-977c-963f67b0f054" (UID: "0f9fbe79-74be-44e1-977c-963f67b0f054"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:46.313503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.313323    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-tmp-dir\") pod \"0f9fbe79-74be-44e1-977c-963f67b0f054\" (UID: \"0f9fbe79-74be-44e1-977c-963f67b0f054\") "
Oct 30 01:04:46.313634 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.313614    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-tmp-dir" pod "0f9fbe79-74be-44e1-977c-963f67b0f054" (UID: "0f9fbe79-74be-44e1-977c-963f67b0f054"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:46.313984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.313965    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:46.313984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.313985    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:46.314137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.313994    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:46.315721 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.315688    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9fbe79-74be-44e1-977c-963f67b0f054-kube-api-access-6hk85" pod "0f9fbe79-74be-44e1-977c-963f67b0f054" (UID: "0f9fbe79-74be-44e1-977c-963f67b0f054"). InnerVolumeSpecName "kube-api-access-6hk85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:46.315835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.315724    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9fbe79-74be-44e1-977c-963f67b0f054-tls-certs" pod "0f9fbe79-74be-44e1-977c-963f67b0f054" (UID: "0f9fbe79-74be-44e1-977c-963f67b0f054"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:46.315835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.315734    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-dshm" pod "0f9fbe79-74be-44e1-977c-963f67b0f054" (UID: "0f9fbe79-74be-44e1-977c-963f67b0f054"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:46.415809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.415724    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f9fbe79-74be-44e1-977c-963f67b0f054-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:46.415809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.415753    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f9fbe79-74be-44e1-977c-963f67b0f054-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:46.415809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.415763    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hk85\" (UniqueName: \"kubernetes.io/projected/0f9fbe79-74be-44e1-977c-963f67b0f054-kube-api-access-6hk85\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:46.849298 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.849257    4425 generic.go:356] "Generic (PLEG): container finished" podID="0f9fbe79-74be-44e1-977c-963f67b0f054" containerID="281cfab270ca913f082dad01e974c36e715f36e28265979a8ac43af1f8979afb" exitCode=0
Oct 30 01:04:46.849515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.849314    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"
Oct 30 01:04:46.849515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.849350    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h" event={"ID":"0f9fbe79-74be-44e1-977c-963f67b0f054","Type":"ContainerDied","Data":"281cfab270ca913f082dad01e974c36e715f36e28265979a8ac43af1f8979afb"}
Oct 30 01:04:46.849515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.849400    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h" event={"ID":"0f9fbe79-74be-44e1-977c-963f67b0f054","Type":"ContainerDied","Data":"5d0039b8b4c5567c1a891ff1c01e9c51cdf1a75c33663cf03fbd77edb64138f4"}
Oct 30 01:04:46.849515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.849421    4425 scope.go:122] "RemoveContainer" containerID="281cfab270ca913f082dad01e974c36e715f36e28265979a8ac43af1f8979afb"
Oct 30 01:04:46.860614 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.860588    4425 scope.go:122] "RemoveContainer" containerID="281cfab270ca913f082dad01e974c36e715f36e28265979a8ac43af1f8979afb"
Oct 30 01:04:46.860903 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:04:46.860885    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281cfab270ca913f082dad01e974c36e715f36e28265979a8ac43af1f8979afb\": container with ID starting with 281cfab270ca913f082dad01e974c36e715f36e28265979a8ac43af1f8979afb not found: ID does not exist" containerID="281cfab270ca913f082dad01e974c36e715f36e28265979a8ac43af1f8979afb"
Oct 30 01:04:46.860953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.860912    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281cfab270ca913f082dad01e974c36e715f36e28265979a8ac43af1f8979afb"} err="failed to get container status \"281cfab270ca913f082dad01e974c36e715f36e28265979a8ac43af1f8979afb\": rpc error: code = NotFound desc = could not find container \"281cfab270ca913f082dad01e974c36e715f36e28265979a8ac43af1f8979afb\": container with ID starting with 281cfab270ca913f082dad01e974c36e715f36e28265979a8ac43af1f8979afb not found: ID does not exist"
Oct 30 01:04:46.895675 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.895635    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"]
Oct 30 01:04:46.905343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:46.905303    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-8363a40b/e2e-lifecycle-model-8363a40b-kserve-58b45c8cf7-jnj7h"]
Oct 30 01:04:47.423973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:47.423935    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9fbe79-74be-44e1-977c-963f67b0f054" path="/var/lib/kubelet/pods/0f9fbe79-74be-44e1-977c-963f67b0f054/volumes"
Oct 30 01:04:47.868942 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:47.868905    4425 generic.go:356] "Generic (PLEG): container finished" podID="4fa39d8a-a4c2-4e0b-b504-feb83231a183" containerID="eccfc8ad1888fb4d216db977fb250443ce9385d5aa13e349d39a816877b0d4ce" exitCode=0
Oct 30 01:04:47.869169 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:47.868978    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp" event={"ID":"4fa39d8a-a4c2-4e0b-b504-feb83231a183","Type":"ContainerDied","Data":"eccfc8ad1888fb4d216db977fb250443ce9385d5aa13e349d39a816877b0d4ce"}
Oct 30 01:04:47.939066 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:47.938992    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:04:48.046446 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.046411    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-data\") pod \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") "
Oct 30 01:04:48.046446 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.046449    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-credential-socket\") pod \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") "
Oct 30 01:04:48.046707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.046492    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-envoy\") pod \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") "
Oct 30 01:04:48.046707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.046559    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-workload-certs\") pod \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") "
Oct 30 01:04:48.046707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.046652    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-token\" (UniqueName: \"kubernetes.io/projected/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-token\") pod \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") "
Oct 30 01:04:48.046856 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.046705    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istiod-ca-cert\") pod \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") "
Oct 30 01:04:48.046856 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.046755    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fa39d8a-a4c2-4e0b-b504-feb83231a183-kube-api-access-shlpj\" (UniqueName: \"kubernetes.io/projected/4fa39d8a-a4c2-4e0b-b504-feb83231a183-kube-api-access-shlpj\") pod \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") "
Oct 30 01:04:48.046856 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.046789    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-workload-socket\") pod \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") "
Oct 30 01:04:48.046856 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.046842    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-podinfo\") pod \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\" (UID: \"4fa39d8a-a4c2-4e0b-b504-feb83231a183\") "
Oct 30 01:04:48.047189 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.046752    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-credential-socket" pod "4fa39d8a-a4c2-4e0b-b504-feb83231a183" (UID: "4fa39d8a-a4c2-4e0b-b504-feb83231a183"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:48.047286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.046769    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-data" pod "4fa39d8a-a4c2-4e0b-b504-feb83231a183" (UID: "4fa39d8a-a4c2-4e0b-b504-feb83231a183"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:48.047385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.047358    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-workload-certs" pod "4fa39d8a-a4c2-4e0b-b504-feb83231a183" (UID: "4fa39d8a-a4c2-4e0b-b504-feb83231a183"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:48.047549 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.047524    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istiod-ca-cert" pod "4fa39d8a-a4c2-4e0b-b504-feb83231a183" (UID: "4fa39d8a-a4c2-4e0b-b504-feb83231a183"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:48.047771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.047750    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-workload-socket" pod "4fa39d8a-a4c2-4e0b-b504-feb83231a183" (UID: "4fa39d8a-a4c2-4e0b-b504-feb83231a183"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:48.048639 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.048610    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.048748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.048641    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.048748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.048658    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.048748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.048673    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.048748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.048687    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.051412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.051382    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-token" pod "4fa39d8a-a4c2-4e0b-b504-feb83231a183" (UID: "4fa39d8a-a4c2-4e0b-b504-feb83231a183"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:48.051724 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.051672    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-podinfo" pod "4fa39d8a-a4c2-4e0b-b504-feb83231a183" (UID: "4fa39d8a-a4c2-4e0b-b504-feb83231a183"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:04:48.051724 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.051677    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa39d8a-a4c2-4e0b-b504-feb83231a183-kube-api-access-shlpj" pod "4fa39d8a-a4c2-4e0b-b504-feb83231a183" (UID: "4fa39d8a-a4c2-4e0b-b504-feb83231a183"). InnerVolumeSpecName "kube-api-access-shlpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:48.051854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.051783    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-envoy" pod "4fa39d8a-a4c2-4e0b-b504-feb83231a183" (UID: "4fa39d8a-a4c2-4e0b-b504-feb83231a183"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:48.150232 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.150196    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.150232 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.150232    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.150500 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.150244    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shlpj\" (UniqueName: \"kubernetes.io/projected/4fa39d8a-a4c2-4e0b-b504-feb83231a183-kube-api-access-shlpj\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.150500 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.150255    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa39d8a-a4c2-4e0b-b504-feb83231a183-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:48.882496 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.882460    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp" event={"ID":"4fa39d8a-a4c2-4e0b-b504-feb83231a183","Type":"ContainerDied","Data":"32d10e780339699754760e5af9f964f600b73bd46a5b4ce1834aec2a16e7a834"}
Oct 30 01:04:48.882496 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.882486    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"
Oct 30 01:04:48.882751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.882561    4425 scope.go:122] "RemoveContainer" containerID="eccfc8ad1888fb4d216db977fb250443ce9385d5aa13e349d39a816877b0d4ce"
Oct 30 01:04:48.938414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.938374    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"]
Oct 30 01:04:48.947749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:48.947717    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9a8d85-openshift-default-84cb77f8df-7rwjp"]
Oct 30 01:04:49.424483 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:49.424448    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fa39d8a-a4c2-4e0b-b504-feb83231a183" path="/var/lib/kubelet/pods/4fa39d8a-a4c2-4e0b-b504-feb83231a183/volumes"
Oct 30 01:04:50.055837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:50.055801    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:04:51.808420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:51.808379    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"
Oct 30 01:04:52.799442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:52.799409    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"]
Oct 30 01:04:52.809710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:52.809602    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9b2c3f-cvdpl"]
Oct 30 01:04:53.426500 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:53.426441    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78237fc0-d05e-4377-8727-0aa79b414b05" path="/var/lib/kubelet/pods/78237fc0-d05e-4377-8727-0aa79b414b05/volumes"
Oct 30 01:04:54.225178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.219958    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"]
Oct 30 01:04:54.225178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.221013    4425 cpu_manager.go:414] "containerMap: removing container" podUID="0f9fbe79-74be-44e1-977c-963f67b0f054" containerName="main"
Oct 30 01:04:54.225178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.221033    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9fbe79-74be-44e1-977c-963f67b0f054" containerName="main"
Oct 30 01:04:54.225178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.221072    4425 cpu_manager.go:414] "containerMap: removing container" podUID="4fa39d8a-a4c2-4e0b-b504-feb83231a183" containerName="istio-proxy"
Oct 30 01:04:54.225178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.221080    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fa39d8a-a4c2-4e0b-b504-feb83231a183" containerName="istio-proxy"
Oct 30 01:04:54.225178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.221254    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9fbe79-74be-44e1-977c-963f67b0f054" containerName="main"
Oct 30 01:04:54.225178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.221278    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fa39d8a-a4c2-4e0b-b504-feb83231a183" containerName="istio-proxy"
Oct 30 01:04:54.228973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.228947    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"
Oct 30 01:04:54.265181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.256132    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"]
Oct 30 01:04:54.266339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.256518    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:04:54.266518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.256591    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:04:54.266604 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.256646    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:04:54.266684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.256749    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:04:54.367759 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.367729    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzfx8\" (UniqueName: \"kubernetes.io/projected/47202689-31b2-4e36-9c83-0c09cedfc8a8-kube-api-access-zzfx8\") pod \"maas-api-revoke-keys-e2e-api-a-645126-92db8\" (UID: \"47202689-31b2-4e36-9c83-0c09cedfc8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"
Oct 30 01:04:54.367957 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.367945    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47202689-31b2-4e36-9c83-0c09cedfc8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-645126-92db8\" (UID: \"47202689-31b2-4e36-9c83-0c09cedfc8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"
Oct 30 01:04:54.368795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.368769    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47202689-31b2-4e36-9c83-0c09cedfc8a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-645126-92db8\" (UID: \"47202689-31b2-4e36-9c83-0c09cedfc8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"
Oct 30 01:04:54.470072 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.470035    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47202689-31b2-4e36-9c83-0c09cedfc8a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-645126-92db8\" (UID: \"47202689-31b2-4e36-9c83-0c09cedfc8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"
Oct 30 01:04:54.470297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.470126    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzfx8\" (UniqueName: \"kubernetes.io/projected/47202689-31b2-4e36-9c83-0c09cedfc8a8-kube-api-access-zzfx8\") pod \"maas-api-revoke-keys-e2e-api-a-645126-92db8\" (UID: \"47202689-31b2-4e36-9c83-0c09cedfc8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"
Oct 30 01:04:54.470297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.470169    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47202689-31b2-4e36-9c83-0c09cedfc8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-645126-92db8\" (UID: \"47202689-31b2-4e36-9c83-0c09cedfc8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"
Oct 30 01:04:54.471025 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.470999    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47202689-31b2-4e36-9c83-0c09cedfc8a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-645126-92db8\" (UID: \"47202689-31b2-4e36-9c83-0c09cedfc8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"
Oct 30 01:04:54.473041 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.473021    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47202689-31b2-4e36-9c83-0c09cedfc8a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-645126-92db8\" (UID: \"47202689-31b2-4e36-9c83-0c09cedfc8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"
Oct 30 01:04:54.491852 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.491799    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzfx8\" (UniqueName: \"kubernetes.io/projected/47202689-31b2-4e36-9c83-0c09cedfc8a8-kube-api-access-zzfx8\") pod \"maas-api-revoke-keys-e2e-api-a-645126-92db8\" (UID: \"47202689-31b2-4e36-9c83-0c09cedfc8a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"
Oct 30 01:04:54.606365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.606324    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"
Oct 30 01:04:54.952720 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:54.952689    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"]
Oct 30 01:04:54.969108 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:04:54.961021    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47202689_31b2_4e36_9c83_0c09cedfc8a8.slice/crio-c6db302b0771820aa9a7c99e605c2dd58ff27859c8f1b92d7b12a5b4637af7d4 WatchSource:0}: Error finding container c6db302b0771820aa9a7c99e605c2dd58ff27859c8f1b92d7b12a5b4637af7d4: Status 404 returned error can't find the container with id c6db302b0771820aa9a7c99e605c2dd58ff27859c8f1b92d7b12a5b4637af7d4
Oct 30 01:04:55.959633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:55.959587    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8" event={"ID":"47202689-31b2-4e36-9c83-0c09cedfc8a8","Type":"ContainerStarted","Data":"0ff7538697963dfce03bc1c5c6a7a763f70f32e2739775c77c58a97f3fc5cb33"}
Oct 30 01:04:55.960180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:55.959662    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8" event={"ID":"47202689-31b2-4e36-9c83-0c09cedfc8a8","Type":"ContainerStarted","Data":"c6db302b0771820aa9a7c99e605c2dd58ff27859c8f1b92d7b12a5b4637af7d4"}
Oct 30 01:04:55.996868 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:55.996813    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8" podStartSLOduration=1.996796315 podStartE2EDuration="1.996796315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:55.994853624 +0000 UTC m=+1404.744508933" watchObservedRunningTime="2026-10-30 01:04:55.996796315 +0000 UTC m=+1404.746451623"
Oct 30 01:04:56.968346 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:56.968315    4425 generic.go:356] "Generic (PLEG): container finished" podID="47202689-31b2-4e36-9c83-0c09cedfc8a8" containerID="0ff7538697963dfce03bc1c5c6a7a763f70f32e2739775c77c58a97f3fc5cb33" exitCode=0
Oct 30 01:04:56.968761 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:56.968410    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8" event={"ID":"47202689-31b2-4e36-9c83-0c09cedfc8a8","Type":"ContainerDied","Data":"0ff7538697963dfce03bc1c5c6a7a763f70f32e2739775c77c58a97f3fc5cb33"}
Oct 30 01:04:58.148012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:58.147986    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"
Oct 30 01:04:58.220697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:58.220661    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47202689-31b2-4e36-9c83-0c09cedfc8a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47202689-31b2-4e36-9c83-0c09cedfc8a8-maas-api-openshift-ca\") pod \"47202689-31b2-4e36-9c83-0c09cedfc8a8\" (UID: \"47202689-31b2-4e36-9c83-0c09cedfc8a8\") "
Oct 30 01:04:58.220887 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:58.220716    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47202689-31b2-4e36-9c83-0c09cedfc8a8-kube-api-access-zzfx8\" (UniqueName: \"kubernetes.io/projected/47202689-31b2-4e36-9c83-0c09cedfc8a8-kube-api-access-zzfx8\") pod \"47202689-31b2-4e36-9c83-0c09cedfc8a8\" (UID: \"47202689-31b2-4e36-9c83-0c09cedfc8a8\") "
Oct 30 01:04:58.220964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:58.220920    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47202689-31b2-4e36-9c83-0c09cedfc8a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47202689-31b2-4e36-9c83-0c09cedfc8a8-maas-api-xks-ca\") pod \"47202689-31b2-4e36-9c83-0c09cedfc8a8\" (UID: \"47202689-31b2-4e36-9c83-0c09cedfc8a8\") "
Oct 30 01:04:58.221094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:58.221066    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47202689-31b2-4e36-9c83-0c09cedfc8a8-maas-api-openshift-ca" pod "47202689-31b2-4e36-9c83-0c09cedfc8a8" (UID: "47202689-31b2-4e36-9c83-0c09cedfc8a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:58.221589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:58.221563    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47202689-31b2-4e36-9c83-0c09cedfc8a8-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:58.223173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:58.223135    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47202689-31b2-4e36-9c83-0c09cedfc8a8-kube-api-access-zzfx8" pod "47202689-31b2-4e36-9c83-0c09cedfc8a8" (UID: "47202689-31b2-4e36-9c83-0c09cedfc8a8"). InnerVolumeSpecName "kube-api-access-zzfx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:58.223444 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:58.223425    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47202689-31b2-4e36-9c83-0c09cedfc8a8-maas-api-xks-ca" pod "47202689-31b2-4e36-9c83-0c09cedfc8a8" (UID: "47202689-31b2-4e36-9c83-0c09cedfc8a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:58.322814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:58.322785    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzfx8\" (UniqueName: \"kubernetes.io/projected/47202689-31b2-4e36-9c83-0c09cedfc8a8-kube-api-access-zzfx8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:58.322814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:58.322812    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47202689-31b2-4e36-9c83-0c09cedfc8a8-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:04:58.987589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:58.987551    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8" event={"ID":"47202689-31b2-4e36-9c83-0c09cedfc8a8","Type":"ContainerDied","Data":"c6db302b0771820aa9a7c99e605c2dd58ff27859c8f1b92d7b12a5b4637af7d4"}
Oct 30 01:04:58.987785 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:58.987610    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6db302b0771820aa9a7c99e605c2dd58ff27859c8f1b92d7b12a5b4637af7d4"
Oct 30 01:04:58.987785 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:04:58.987665    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"
Oct 30 01:05:01.440951 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.440912    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"]
Oct 30 01:05:01.441739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.441717    4425 cpu_manager.go:414] "containerMap: removing container" podUID="47202689-31b2-4e36-9c83-0c09cedfc8a8" containerName="revoke-keys"
Oct 30 01:05:01.441840 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.441741    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47202689-31b2-4e36-9c83-0c09cedfc8a8" containerName="revoke-keys"
Oct 30 01:05:01.441893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.441872    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="47202689-31b2-4e36-9c83-0c09cedfc8a8" containerName="revoke-keys"
Oct 30 01:05:01.445581 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.445560    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"
Oct 30 01:05:01.457932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.457910    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:05:01.467687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.467664    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:05:01.467788 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.467728    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:05:01.471671 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.471649    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"]
Oct 30 01:05:01.557968 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.557940    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:05:01.594129 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.594101    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m5l5\" (UniqueName: \"kubernetes.io/projected/54a88c37-7411-45c7-811d-62ae73ce4fff-kube-api-access-8m5l5\") pod \"maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57\" (UID: \"54a88c37-7411-45c7-811d-62ae73ce4fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"
Oct 30 01:05:01.594261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.594204    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a88c37-7411-45c7-811d-62ae73ce4fff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57\" (UID: \"54a88c37-7411-45c7-811d-62ae73ce4fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"
Oct 30 01:05:01.594389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.594368    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a88c37-7411-45c7-811d-62ae73ce4fff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57\" (UID: \"54a88c37-7411-45c7-811d-62ae73ce4fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"
Oct 30 01:05:01.695342 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.695259    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m5l5\" (UniqueName: \"kubernetes.io/projected/54a88c37-7411-45c7-811d-62ae73ce4fff-kube-api-access-8m5l5\") pod \"maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57\" (UID: \"54a88c37-7411-45c7-811d-62ae73ce4fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"
Oct 30 01:05:01.695510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.695341    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a88c37-7411-45c7-811d-62ae73ce4fff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57\" (UID: \"54a88c37-7411-45c7-811d-62ae73ce4fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"
Oct 30 01:05:01.695572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.695506    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a88c37-7411-45c7-811d-62ae73ce4fff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57\" (UID: \"54a88c37-7411-45c7-811d-62ae73ce4fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"
Oct 30 01:05:01.696119 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.696096    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a88c37-7411-45c7-811d-62ae73ce4fff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57\" (UID: \"54a88c37-7411-45c7-811d-62ae73ce4fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"
Oct 30 01:05:01.697639 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.697622    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a88c37-7411-45c7-811d-62ae73ce4fff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57\" (UID: \"54a88c37-7411-45c7-811d-62ae73ce4fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"
Oct 30 01:05:01.715995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.715966    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m5l5\" (UniqueName: \"kubernetes.io/projected/54a88c37-7411-45c7-811d-62ae73ce4fff-kube-api-access-8m5l5\") pod \"maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57\" (UID: \"54a88c37-7411-45c7-811d-62ae73ce4fff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"
Oct 30 01:05:01.759745 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:01.759720    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"
Oct 30 01:05:02.060338 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:02.060310    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"]
Oct 30 01:05:02.061059 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:05:02.061034    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a88c37_7411_45c7_811d_62ae73ce4fff.slice/crio-d82abe7e8b3f00dcdd3e652f94c4bc6f3c7d6674e11cf25df6e2c34604185f91 WatchSource:0}: Error finding container d82abe7e8b3f00dcdd3e652f94c4bc6f3c7d6674e11cf25df6e2c34604185f91: Status 404 returned error can't find the container with id d82abe7e8b3f00dcdd3e652f94c4bc6f3c7d6674e11cf25df6e2c34604185f91
Oct 30 01:05:03.019192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.019121    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57" event={"ID":"54a88c37-7411-45c7-811d-62ae73ce4fff","Type":"ContainerStarted","Data":"44bfd636bbe46cea34a681da03adbdad82f7e514479d6e9cb4d142839778a683"}
Oct 30 01:05:03.019661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.019230    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57" event={"ID":"54a88c37-7411-45c7-811d-62ae73ce4fff","Type":"ContainerStarted","Data":"d82abe7e8b3f00dcdd3e652f94c4bc6f3c7d6674e11cf25df6e2c34604185f91"}
Oct 30 01:05:03.058616 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.058556    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57" podStartSLOduration=2.058537396 podStartE2EDuration="2.058537396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:03.054419833 +0000 UTC m=+1411.804075145" watchObservedRunningTime="2026-10-30 01:05:03.058537396 +0000 UTC m=+1411.808192706"
Oct 30 01:05:03.718932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.718895    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"]
Oct 30 01:05:03.723263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.723239    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.760236 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.760207    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"]
Oct 30 01:05:03.818342 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.818314    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-76f654-openshift-default-dockercfg-p6m9l\""
Oct 30 01:05:03.824485 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.824456    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-istio-data\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.824649 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.824628    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhn6h\" (UniqueName: \"kubernetes.io/projected/075fdcb4-b628-4603-ab90-e7575e3096df-kube-api-access-bhn6h\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.824694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.824666    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-workload-certs\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.824748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.824693    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/075fdcb4-b628-4603-ab90-e7575e3096df-istio-token\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.824838 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.824817    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/075fdcb4-b628-4603-ab90-e7575e3096df-istio-podinfo\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.824897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.824860    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-workload-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.824937 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.824911    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-credential-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.824978 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.824967    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-istio-envoy\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.825019 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.824995    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/075fdcb4-b628-4603-ab90-e7575e3096df-istiod-ca-cert\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.926355 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.926319    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/075fdcb4-b628-4603-ab90-e7575e3096df-istio-podinfo\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.926355 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.926359    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-workload-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.926605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.926380    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-credential-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.926605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.926435    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-istio-envoy\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.926605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.926460    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/075fdcb4-b628-4603-ab90-e7575e3096df-istiod-ca-cert\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.926605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.926538    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-istio-data\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.926819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.926675    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhn6h\" (UniqueName: \"kubernetes.io/projected/075fdcb4-b628-4603-ab90-e7575e3096df-kube-api-access-bhn6h\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.926819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.926714    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-workload-certs\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.926819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.926750    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/075fdcb4-b628-4603-ab90-e7575e3096df-istio-token\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.926979 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.926814    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-workload-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.926979 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.926865    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-credential-socket\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.927083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.926981    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-istio-data\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.927138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.927097    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-workload-certs\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.927380 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.927358    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/075fdcb4-b628-4603-ab90-e7575e3096df-istiod-ca-cert\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.928942 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.928920    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-istio-envoy\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.929040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.929023    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/075fdcb4-b628-4603-ab90-e7575e3096df-istio-podinfo\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.947699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.947674    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/075fdcb4-b628-4603-ab90-e7575e3096df-istio-token\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:03.947780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:03.947736    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhn6h\" (UniqueName: \"kubernetes.io/projected/075fdcb4-b628-4603-ab90-e7575e3096df-kube-api-access-bhn6h\") pod \"e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") " pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:04.029259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:04.029220    4425 generic.go:356] "Generic (PLEG): container finished" podID="54a88c37-7411-45c7-811d-62ae73ce4fff" containerID="44bfd636bbe46cea34a681da03adbdad82f7e514479d6e9cb4d142839778a683" exitCode=0
Oct 30 01:05:04.029716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:04.029304    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57" event={"ID":"54a88c37-7411-45c7-811d-62ae73ce4fff","Type":"ContainerDied","Data":"44bfd636bbe46cea34a681da03adbdad82f7e514479d6e9cb4d142839778a683"}
Oct 30 01:05:04.037675 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:04.037651    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:04.362655 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:04.362624    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"]
Oct 30 01:05:04.363274 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:05:04.363249    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075fdcb4_b628_4603_ab90_e7575e3096df.slice/crio-19385d999d1efc624fb410a6dafcd9973f7b83ac71d4f0cf679bb203c4dc74e7 WatchSource:0}: Error finding container 19385d999d1efc624fb410a6dafcd9973f7b83ac71d4f0cf679bb203c4dc74e7: Status 404 returned error can't find the container with id 19385d999d1efc624fb410a6dafcd9973f7b83ac71d4f0cf679bb203c4dc74e7
Oct 30 01:05:04.365860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:04.365829    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:05:04.365942 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:04.365892    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:05:04.365942 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:04.365921    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:05:04.581194 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:04.581139    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"]
Oct 30 01:05:04.581592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:04.581417    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt" podUID="8d56cb59-ad4a-47ac-9107-5160fe97dfad" containerName="istio-proxy" containerID="cri-o://68972de5501186f50b967e0886df7b397fd5dfcc88de0f3292adac2fe54269c7" gracePeriod=30
Oct 30 01:05:04.586252 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:04.586122    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"]
Oct 30 01:05:04.586369 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:04.586344    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6" podUID="83443474-b817-4ba6-afb6-dbe9a7bc24f0" containerName="istio-proxy" containerID="cri-o://88e78fda87118261430affe6efdc5467271c3a3775c574f220e610861b8ea40b" gracePeriod=30
Oct 30 01:05:04.660287 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:04.660203    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"]
Oct 30 01:05:04.690961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:04.690918    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"]
Oct 30 01:05:05.039376 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:05.039336    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc" event={"ID":"075fdcb4-b628-4603-ab90-e7575e3096df","Type":"ContainerStarted","Data":"5d611cfd36b883c35667a2991f727f69da7954141ba1863b14d31a2ba84b3fdd"}
Oct 30 01:05:05.039852 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:05.039393    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc" event={"ID":"075fdcb4-b628-4603-ab90-e7575e3096df","Type":"ContainerStarted","Data":"19385d999d1efc624fb410a6dafcd9973f7b83ac71d4f0cf679bb203c4dc74e7"}
Oct 30 01:05:05.083569 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:05.083499    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc" podStartSLOduration=2.08347613 podStartE2EDuration="2.08347613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:05.079616062 +0000 UTC m=+1413.829271371" watchObservedRunningTime="2026-10-30 01:05:05.08347613 +0000 UTC m=+1413.833131442"
Oct 30 01:05:05.216377 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:05.216350    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"
Oct 30 01:05:05.361242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:05.360194    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a88c37-7411-45c7-811d-62ae73ce4fff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a88c37-7411-45c7-811d-62ae73ce4fff-maas-api-openshift-ca\") pod \"54a88c37-7411-45c7-811d-62ae73ce4fff\" (UID: \"54a88c37-7411-45c7-811d-62ae73ce4fff\") "
Oct 30 01:05:05.361242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:05.360309    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a88c37-7411-45c7-811d-62ae73ce4fff-kube-api-access-8m5l5\" (UniqueName: \"kubernetes.io/projected/54a88c37-7411-45c7-811d-62ae73ce4fff-kube-api-access-8m5l5\") pod \"54a88c37-7411-45c7-811d-62ae73ce4fff\" (UID: \"54a88c37-7411-45c7-811d-62ae73ce4fff\") "
Oct 30 01:05:05.361242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:05.360547    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54a88c37-7411-45c7-811d-62ae73ce4fff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a88c37-7411-45c7-811d-62ae73ce4fff-maas-api-xks-ca\") pod \"54a88c37-7411-45c7-811d-62ae73ce4fff\" (UID: \"54a88c37-7411-45c7-811d-62ae73ce4fff\") "
Oct 30 01:05:05.363293 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:05.363261    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a88c37-7411-45c7-811d-62ae73ce4fff-maas-api-openshift-ca" pod "54a88c37-7411-45c7-811d-62ae73ce4fff" (UID: "54a88c37-7411-45c7-811d-62ae73ce4fff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:05.364017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:05.363990    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a88c37-7411-45c7-811d-62ae73ce4fff-maas-api-xks-ca" pod "54a88c37-7411-45c7-811d-62ae73ce4fff" (UID: "54a88c37-7411-45c7-811d-62ae73ce4fff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:05.365018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:05.364557    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a88c37-7411-45c7-811d-62ae73ce4fff-kube-api-access-8m5l5" pod "54a88c37-7411-45c7-811d-62ae73ce4fff" (UID: "54a88c37-7411-45c7-811d-62ae73ce4fff"). InnerVolumeSpecName "kube-api-access-8m5l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:05.471824 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:05.471779    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a88c37-7411-45c7-811d-62ae73ce4fff-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:05.471824 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:05.471824    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a88c37-7411-45c7-811d-62ae73ce4fff-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:05.472099 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:05.471841    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m5l5\" (UniqueName: \"kubernetes.io/projected/54a88c37-7411-45c7-811d-62ae73ce4fff-kube-api-access-8m5l5\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:06.038604 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:06.038572    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:06.040362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:06.040337    4425 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 01:05:06.040696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:06.040385    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc" podUID="075fdcb4-b628-4603-ab90-e7575e3096df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 01:05:06.047596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:06.047569    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"
Oct 30 01:05:06.047736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:06.047596    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57" event={"ID":"54a88c37-7411-45c7-811d-62ae73ce4fff","Type":"ContainerDied","Data":"d82abe7e8b3f00dcdd3e652f94c4bc6f3c7d6674e11cf25df6e2c34604185f91"}
Oct 30 01:05:06.047736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:06.047653    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d82abe7e8b3f00dcdd3e652f94c4bc6f3c7d6674e11cf25df6e2c34604185f91"
Oct 30 01:05:07.038802 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:07.038763    4425 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 01:05:07.039016 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:07.038818    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc" podUID="075fdcb4-b628-4603-ab90-e7575e3096df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 01:05:07.611180 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:05:07.610868    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:05:07.977710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:07.977604    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"]
Oct 30 01:05:07.977909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:07.977874    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls" podUID="421b4772-a3d2-4cf1-aefb-abc655989ad9" containerName="istio-proxy" containerID="cri-o://21de5a9a998b22a17887be397d1bf9ba7c70d1fbcdf0433d0548786413cf2ea3" gracePeriod=30
Oct 30 01:05:07.990870 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:07.990555    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"]
Oct 30 01:05:07.990870 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:07.990807    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx" podUID="251fdcfb-b27b-4f4e-854b-9f3f70304ace" containerName="istio-proxy" containerID="cri-o://cd20f8e9fa01f73b334530f7f24afffbaf97f5d60cb564880285bc4476ab2058" gracePeriod=30
Oct 30 01:05:08.039707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:08.038805    4425 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 01:05:08.039707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:08.038877    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc" podUID="075fdcb4-b628-4603-ab90-e7575e3096df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 01:05:09.038887 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:09.038848    4425 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 01:05:09.039315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:09.038907    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc" podUID="075fdcb4-b628-4603-ab90-e7575e3096df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 01:05:09.675234 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:09.665321    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b" podUID="2fb77042-ce3f-4913-b677-394cd6520fc0" containerName="payload-processing" containerID="cri-o://16fe9b18ec8b6d8ddbc33c023976f83956241a2bc5b1163066ba85333b85778e" gracePeriod=25
Oct 30 01:05:09.691836 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:09.691778    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g" podUID="0eabc38b-35ba-4133-a05d-bba79769f477" containerName="payload-pre-processing" containerID="cri-o://c8f0ff0ec5e0c1225203aef2f50d4ef9f0443b5d6352b4caa62eadec061dcb1c" gracePeriod=25
Oct 30 01:05:10.026583 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.026243    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 01:05:10.039866 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.039798    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 01:05:10.040491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.040461    4425 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 01:05:10.040599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.040512    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc" podUID="075fdcb4-b628-4603-ab90-e7575e3096df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 01:05:10.108631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.106106    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-workload-socket\") pod \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") "
Oct 30 01:05:10.108631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.106426    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-workload-socket" pod "8d56cb59-ad4a-47ac-9107-5160fe97dfad" (UID: "8d56cb59-ad4a-47ac-9107-5160fe97dfad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:10.108631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.108449    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-workload-certs" pod "8d56cb59-ad4a-47ac-9107-5160fe97dfad" (UID: "8d56cb59-ad4a-47ac-9107-5160fe97dfad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:10.108631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.108627    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-workload-certs\") pod \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") "
Oct 30 01:05:10.108977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.108691    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-workload-certs\") pod \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") "
Oct 30 01:05:10.108977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.108935    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-workload-certs" pod "83443474-b817-4ba6-afb6-dbe9a7bc24f0" (UID: "83443474-b817-4ba6-afb6-dbe9a7bc24f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:10.109078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.108992    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-data\") pod \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") "
Oct 30 01:05:10.109298 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.109247    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-data" pod "83443474-b817-4ba6-afb6-dbe9a7bc24f0" (UID: "83443474-b817-4ba6-afb6-dbe9a7bc24f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:10.109652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.109613    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-envoy\") pod \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") "
Oct 30 01:05:10.109760 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.109700    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-envoy\") pod \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") "
Oct 30 01:05:10.110126 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.110067    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-podinfo\") pod \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") "
Oct 30 01:05:10.110442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.110419    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83443474-b817-4ba6-afb6-dbe9a7bc24f0-kube-api-access-w62qd\" (UniqueName: \"kubernetes.io/projected/83443474-b817-4ba6-afb6-dbe9a7bc24f0-kube-api-access-w62qd\") pod \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") "
Oct 30 01:05:10.111304 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.111280    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-credential-socket\") pod \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") "
Oct 30 01:05:10.111439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.111390    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-data\") pod \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") "
Oct 30 01:05:10.111521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.111436    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-token\" (UniqueName: \"kubernetes.io/projected/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-token\") pod \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") "
Oct 30 01:05:10.111521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.111466    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-workload-socket\") pod \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") "
Oct 30 01:05:10.111661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.111517    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-token\" (UniqueName: \"kubernetes.io/projected/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-token\") pod \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") "
Oct 30 01:05:10.112113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.112091    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-workload-socket" pod "83443474-b817-4ba6-afb6-dbe9a7bc24f0" (UID: "83443474-b817-4ba6-afb6-dbe9a7bc24f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:10.112299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.112278    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-credential-socket" pod "8d56cb59-ad4a-47ac-9107-5160fe97dfad" (UID: "8d56cb59-ad4a-47ac-9107-5160fe97dfad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:10.112504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.112485    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-data" pod "8d56cb59-ad4a-47ac-9107-5160fe97dfad" (UID: "8d56cb59-ad4a-47ac-9107-5160fe97dfad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:10.112743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.112723    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-credential-socket\") pod \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") "
Oct 30 01:05:10.112863 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.112795    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istiod-ca-cert\") pod \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") "
Oct 30 01:05:10.112863 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.112828    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-podinfo\") pod \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") "
Oct 30 01:05:10.112976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.112967    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istiod-ca-cert\") pod \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\" (UID: \"83443474-b817-4ba6-afb6-dbe9a7bc24f0\") "
Oct 30 01:05:10.113023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.112996    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d56cb59-ad4a-47ac-9107-5160fe97dfad-kube-api-access-vfshk\" (UniqueName: \"kubernetes.io/projected/8d56cb59-ad4a-47ac-9107-5160fe97dfad-kube-api-access-vfshk\") pod \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\" (UID: \"8d56cb59-ad4a-47ac-9107-5160fe97dfad\") "
Oct 30 01:05:10.113766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.113739    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.113848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.113771    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.113848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.113785    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.113848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.113798    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.113848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.113812    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.113848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.113825    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.113848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.113836    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.123295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.121927    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istiod-ca-cert" pod "8d56cb59-ad4a-47ac-9107-5160fe97dfad" (UID: "8d56cb59-ad4a-47ac-9107-5160fe97dfad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:10.123295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.122186    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-credential-socket" pod "83443474-b817-4ba6-afb6-dbe9a7bc24f0" (UID: "83443474-b817-4ba6-afb6-dbe9a7bc24f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:10.123295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.122670    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-token" pod "8d56cb59-ad4a-47ac-9107-5160fe97dfad" (UID: "8d56cb59-ad4a-47ac-9107-5160fe97dfad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:10.123295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.122717    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istiod-ca-cert" pod "83443474-b817-4ba6-afb6-dbe9a7bc24f0" (UID: "83443474-b817-4ba6-afb6-dbe9a7bc24f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:10.130239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.128369    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d56cb59-ad4a-47ac-9107-5160fe97dfad-kube-api-access-vfshk" pod "8d56cb59-ad4a-47ac-9107-5160fe97dfad" (UID: "8d56cb59-ad4a-47ac-9107-5160fe97dfad"). InnerVolumeSpecName "kube-api-access-vfshk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:10.130239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.129995    4425 generic.go:356] "Generic (PLEG): container finished" podID="83443474-b817-4ba6-afb6-dbe9a7bc24f0" containerID="88e78fda87118261430affe6efdc5467271c3a3775c574f220e610861b8ea40b" exitCode=0
Oct 30 01:05:10.130239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.130074    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6" event={"ID":"83443474-b817-4ba6-afb6-dbe9a7bc24f0","Type":"ContainerDied","Data":"88e78fda87118261430affe6efdc5467271c3a3775c574f220e610861b8ea40b"}
Oct 30 01:05:10.130239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.130134    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6" event={"ID":"83443474-b817-4ba6-afb6-dbe9a7bc24f0","Type":"ContainerDied","Data":"e3e7b0eb9c7df49c5f8ffd67b2cce6ceaedef7b61469ef7d9e7bf61b67d7c655"}
Oct 30 01:05:10.130239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.130197    4425 scope.go:122] "RemoveContainer" containerID="88e78fda87118261430affe6efdc5467271c3a3775c574f220e610861b8ea40b"
Oct 30 01:05:10.130650 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.130314    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"
Oct 30 01:05:10.132905 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.132784    4425 generic.go:356] "Generic (PLEG): container finished" podID="0eabc38b-35ba-4133-a05d-bba79769f477" containerID="c8f0ff0ec5e0c1225203aef2f50d4ef9f0443b5d6352b4caa62eadec061dcb1c" exitCode=0
Oct 30 01:05:10.132996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.132921    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g" event={"ID":"0eabc38b-35ba-4133-a05d-bba79769f477","Type":"ContainerDied","Data":"c8f0ff0ec5e0c1225203aef2f50d4ef9f0443b5d6352b4caa62eadec061dcb1c"}
Oct 30 01:05:10.135133 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.134538    4425 generic.go:356] "Generic (PLEG): container finished" podID="8d56cb59-ad4a-47ac-9107-5160fe97dfad" containerID="68972de5501186f50b967e0886df7b397fd5dfcc88de0f3292adac2fe54269c7" exitCode=0
Oct 30 01:05:10.135133 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.134701    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt" event={"ID":"8d56cb59-ad4a-47ac-9107-5160fe97dfad","Type":"ContainerDied","Data":"68972de5501186f50b967e0886df7b397fd5dfcc88de0f3292adac2fe54269c7"}
Oct 30 01:05:10.135133 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.134739    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt" event={"ID":"8d56cb59-ad4a-47ac-9107-5160fe97dfad","Type":"ContainerDied","Data":"09a96230cbcbebe64c67f2e587a9fad2ff46684624fa0c92f8a20accebf54c75"}
Oct 30 01:05:10.135133 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.134810    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"
Oct 30 01:05:10.138608 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.136333    4425 generic.go:356] "Generic (PLEG): container finished" podID="2fb77042-ce3f-4913-b677-394cd6520fc0" containerID="16fe9b18ec8b6d8ddbc33c023976f83956241a2bc5b1163066ba85333b85778e" exitCode=0
Oct 30 01:05:10.138608 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.136407    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b" event={"ID":"2fb77042-ce3f-4913-b677-394cd6520fc0","Type":"ContainerDied","Data":"16fe9b18ec8b6d8ddbc33c023976f83956241a2bc5b1163066ba85333b85778e"}
Oct 30 01:05:10.148178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.146931    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83443474-b817-4ba6-afb6-dbe9a7bc24f0-kube-api-access-w62qd" pod "83443474-b817-4ba6-afb6-dbe9a7bc24f0" (UID: "83443474-b817-4ba6-afb6-dbe9a7bc24f0"). InnerVolumeSpecName "kube-api-access-w62qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:10.148178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.147045    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"
Oct 30 01:05:10.148178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.146964    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-envoy" pod "83443474-b817-4ba6-afb6-dbe9a7bc24f0" (UID: "83443474-b817-4ba6-afb6-dbe9a7bc24f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:10.148178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.147006    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-envoy" pod "8d56cb59-ad4a-47ac-9107-5160fe97dfad" (UID: "8d56cb59-ad4a-47ac-9107-5160fe97dfad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:10.148178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.147274    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-podinfo" pod "83443474-b817-4ba6-afb6-dbe9a7bc24f0" (UID: "83443474-b817-4ba6-afb6-dbe9a7bc24f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:05:10.148178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.147296    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-token" pod "83443474-b817-4ba6-afb6-dbe9a7bc24f0" (UID: "83443474-b817-4ba6-afb6-dbe9a7bc24f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:10.165205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.165058    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"
Oct 30 01:05:10.186506 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.186327    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-podinfo" pod "8d56cb59-ad4a-47ac-9107-5160fe97dfad" (UID: "8d56cb59-ad4a-47ac-9107-5160fe97dfad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:05:10.186506 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.186420    4425 scope.go:122] "RemoveContainer" containerID="88e78fda87118261430affe6efdc5467271c3a3775c574f220e610861b8ea40b"
Oct 30 01:05:10.186816 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:05:10.186785    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e78fda87118261430affe6efdc5467271c3a3775c574f220e610861b8ea40b\": container with ID starting with 88e78fda87118261430affe6efdc5467271c3a3775c574f220e610861b8ea40b not found: ID does not exist" containerID="88e78fda87118261430affe6efdc5467271c3a3775c574f220e610861b8ea40b"
Oct 30 01:05:10.186884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.186826    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e78fda87118261430affe6efdc5467271c3a3775c574f220e610861b8ea40b"} err="failed to get container status \"88e78fda87118261430affe6efdc5467271c3a3775c574f220e610861b8ea40b\": rpc error: code = NotFound desc = could not find container \"88e78fda87118261430affe6efdc5467271c3a3775c574f220e610861b8ea40b\": container with ID starting with 88e78fda87118261430affe6efdc5467271c3a3775c574f220e610861b8ea40b not found: ID does not exist"
Oct 30 01:05:10.186884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.186852    4425 scope.go:122] "RemoveContainer" containerID="68972de5501186f50b967e0886df7b397fd5dfcc88de0f3292adac2fe54269c7"
Oct 30 01:05:10.204584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.204546    4425 scope.go:122] "RemoveContainer" containerID="68972de5501186f50b967e0886df7b397fd5dfcc88de0f3292adac2fe54269c7"
Oct 30 01:05:10.208738 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:05:10.205138    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68972de5501186f50b967e0886df7b397fd5dfcc88de0f3292adac2fe54269c7\": container with ID starting with 68972de5501186f50b967e0886df7b397fd5dfcc88de0f3292adac2fe54269c7 not found: ID does not exist" containerID="68972de5501186f50b967e0886df7b397fd5dfcc88de0f3292adac2fe54269c7"
Oct 30 01:05:10.208846 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.208755    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68972de5501186f50b967e0886df7b397fd5dfcc88de0f3292adac2fe54269c7"} err="failed to get container status \"68972de5501186f50b967e0886df7b397fd5dfcc88de0f3292adac2fe54269c7\": rpc error: code = NotFound desc = could not find container \"68972de5501186f50b967e0886df7b397fd5dfcc88de0f3292adac2fe54269c7\": container with ID starting with 68972de5501186f50b967e0886df7b397fd5dfcc88de0f3292adac2fe54269c7 not found: ID does not exist"
Oct 30 01:05:10.215293 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.215271    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.215293 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.215293    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.215545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.215305    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.215545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.215314    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.215545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.215322    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.215545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.215330    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfshk\" (UniqueName: \"kubernetes.io/projected/8d56cb59-ad4a-47ac-9107-5160fe97dfad-kube-api-access-vfshk\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.215545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.215365    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.215545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.215377    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d56cb59-ad4a-47ac-9107-5160fe97dfad-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.215545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.215386    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.215545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.215395    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w62qd\" (UniqueName: \"kubernetes.io/projected/83443474-b817-4ba6-afb6-dbe9a7bc24f0-kube-api-access-w62qd\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.215545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.215403    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83443474-b817-4ba6-afb6-dbe9a7bc24f0-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.316114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.316079    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb77042-ce3f-4913-b677-394cd6520fc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb77042-ce3f-4913-b677-394cd6520fc0-plugins-config-volume\") pod \"2fb77042-ce3f-4913-b677-394cd6520fc0\" (UID: \"2fb77042-ce3f-4913-b677-394cd6520fc0\") "
Oct 30 01:05:10.316282 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.316199    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eabc38b-35ba-4133-a05d-bba79769f477-kube-api-access-k89hw\" (UniqueName: \"kubernetes.io/projected/0eabc38b-35ba-4133-a05d-bba79769f477-kube-api-access-k89hw\") pod \"0eabc38b-35ba-4133-a05d-bba79769f477\" (UID: \"0eabc38b-35ba-4133-a05d-bba79769f477\") "
Oct 30 01:05:10.316378 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.316358    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eabc38b-35ba-4133-a05d-bba79769f477-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eabc38b-35ba-4133-a05d-bba79769f477-plugins-config-volume\") pod \"0eabc38b-35ba-4133-a05d-bba79769f477\" (UID: \"0eabc38b-35ba-4133-a05d-bba79769f477\") "
Oct 30 01:05:10.316453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.316438    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb77042-ce3f-4913-b677-394cd6520fc0-kube-api-access-g6hzs\" (UniqueName: \"kubernetes.io/projected/2fb77042-ce3f-4913-b677-394cd6520fc0-kube-api-access-g6hzs\") pod \"2fb77042-ce3f-4913-b677-394cd6520fc0\" (UID: \"2fb77042-ce3f-4913-b677-394cd6520fc0\") "
Oct 30 01:05:10.316712 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.316692    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eabc38b-35ba-4133-a05d-bba79769f477-plugins-config-volume" pod "0eabc38b-35ba-4133-a05d-bba79769f477" (UID: "0eabc38b-35ba-4133-a05d-bba79769f477"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:10.316754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.316693    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb77042-ce3f-4913-b677-394cd6520fc0-plugins-config-volume" pod "2fb77042-ce3f-4913-b677-394cd6520fc0" (UID: "2fb77042-ce3f-4913-b677-394cd6520fc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:10.317030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.317009    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eabc38b-35ba-4133-a05d-bba79769f477-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.317101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.317032    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fb77042-ce3f-4913-b677-394cd6520fc0-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.318641 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.318612    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eabc38b-35ba-4133-a05d-bba79769f477-kube-api-access-k89hw" pod "0eabc38b-35ba-4133-a05d-bba79769f477" (UID: "0eabc38b-35ba-4133-a05d-bba79769f477"). InnerVolumeSpecName "kube-api-access-k89hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:10.318799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.318783    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb77042-ce3f-4913-b677-394cd6520fc0-kube-api-access-g6hzs" pod "2fb77042-ce3f-4913-b677-394cd6520fc0" (UID: "2fb77042-ce3f-4913-b677-394cd6520fc0"). InnerVolumeSpecName "kube-api-access-g6hzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:10.419111 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.419009    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6hzs\" (UniqueName: \"kubernetes.io/projected/2fb77042-ce3f-4913-b677-394cd6520fc0-kube-api-access-g6hzs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.419111 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.419060    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k89hw\" (UniqueName: \"kubernetes.io/projected/0eabc38b-35ba-4133-a05d-bba79769f477-kube-api-access-k89hw\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:10.502988 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.502949    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"]
Oct 30 01:05:10.506515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.506480    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-6z6q6"]
Oct 30 01:05:10.536324 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.536293    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"]
Oct 30 01:05:10.542399 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:10.542371    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-mhflt"]
Oct 30 01:05:11.038976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.038936    4425 patch_prober.go:29] interesting pod/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Oct 30 01:05:11.039185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.039002    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc" podUID="075fdcb4-b628-4603-ab90-e7575e3096df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Oct 30 01:05:11.149877 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.149843    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"
Oct 30 01:05:11.149877 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.149855    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g" event={"ID":"0eabc38b-35ba-4133-a05d-bba79769f477","Type":"ContainerDied","Data":"5b5a4a75acc412741ee789dd0f9a82caea92e7dbc8b3d7c36d1e45a602bcc240"}
Oct 30 01:05:11.150465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.149924    4425 scope.go:122] "RemoveContainer" containerID="c8f0ff0ec5e0c1225203aef2f50d4ef9f0443b5d6352b4caa62eadec061dcb1c"
Oct 30 01:05:11.152894 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.152868    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"
Oct 30 01:05:11.152894 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.152874    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b" event={"ID":"2fb77042-ce3f-4913-b677-394cd6520fc0","Type":"ContainerDied","Data":"c2af9daedbbf72633407c6328ae1ad1ca41e62f76de72e4ceeb1b464b70d507e"}
Oct 30 01:05:11.162822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.162798    4425 scope.go:122] "RemoveContainer" containerID="16fe9b18ec8b6d8ddbc33c023976f83956241a2bc5b1163066ba85333b85778e"
Oct 30 01:05:11.208002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.207969    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"]
Oct 30 01:05:11.211772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.211747    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-645126-7d6898948c-9fm7g"]
Oct 30 01:05:11.243366 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.243334    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"]
Oct 30 01:05:11.250083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.250053    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-645126-6cfdb55698-h7j4b"]
Oct 30 01:05:11.424092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.424008    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eabc38b-35ba-4133-a05d-bba79769f477" path="/var/lib/kubelet/pods/0eabc38b-35ba-4133-a05d-bba79769f477/volumes"
Oct 30 01:05:11.424474 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.424459    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fb77042-ce3f-4913-b677-394cd6520fc0" path="/var/lib/kubelet/pods/2fb77042-ce3f-4913-b677-394cd6520fc0/volumes"
Oct 30 01:05:11.424885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.424873    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83443474-b817-4ba6-afb6-dbe9a7bc24f0" path="/var/lib/kubelet/pods/83443474-b817-4ba6-afb6-dbe9a7bc24f0/volumes"
Oct 30 01:05:11.425620 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.425605    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d56cb59-ad4a-47ac-9107-5160fe97dfad" path="/var/lib/kubelet/pods/8d56cb59-ad4a-47ac-9107-5160fe97dfad/volumes"
Oct 30 01:05:11.859610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.859569    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"]
Oct 30 01:05:11.886825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:11.886788    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"]
Oct 30 01:05:12.042800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:12.042768    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:12.043075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:12.043056    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:12.043746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:12.043725    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:05:13.200693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.200656    4425 generic.go:356] "Generic (PLEG): container finished" podID="421b4772-a3d2-4cf1-aefb-abc655989ad9" containerID="21de5a9a998b22a17887be397d1bf9ba7c70d1fbcdf0433d0548786413cf2ea3" exitCode=0
Oct 30 01:05:13.201320 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.200772    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls" event={"ID":"421b4772-a3d2-4cf1-aefb-abc655989ad9","Type":"ContainerDied","Data":"21de5a9a998b22a17887be397d1bf9ba7c70d1fbcdf0433d0548786413cf2ea3"}
Oct 30 01:05:13.204997 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.202987    4425 generic.go:356] "Generic (PLEG): container finished" podID="251fdcfb-b27b-4f4e-854b-9f3f70304ace" containerID="cd20f8e9fa01f73b334530f7f24afffbaf97f5d60cb564880285bc4476ab2058" exitCode=0
Oct 30 01:05:13.205941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.205913    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx" event={"ID":"251fdcfb-b27b-4f4e-854b-9f3f70304ace","Type":"ContainerDied","Data":"cd20f8e9fa01f73b334530f7f24afffbaf97f5d60cb564880285bc4476ab2058"}
Oct 30 01:05:13.274857 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.274792    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 01:05:13.358891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.358569    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/421b4772-a3d2-4cf1-aefb-abc655989ad9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421b4772-a3d2-4cf1-aefb-abc655989ad9-istiod-ca-cert\") pod \"421b4772-a3d2-4cf1-aefb-abc655989ad9\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") "
Oct 30 01:05:13.358891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.358658    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-credential-socket\") pod \"421b4772-a3d2-4cf1-aefb-abc655989ad9\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") "
Oct 30 01:05:13.358891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.358839    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-podinfo\") pod \"421b4772-a3d2-4cf1-aefb-abc655989ad9\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") "
Oct 30 01:05:13.358891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.358897    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421b4772-a3d2-4cf1-aefb-abc655989ad9-kube-api-access-bdwsz\" (UniqueName: \"kubernetes.io/projected/421b4772-a3d2-4cf1-aefb-abc655989ad9-kube-api-access-bdwsz\") pod \"421b4772-a3d2-4cf1-aefb-abc655989ad9\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") "
Oct 30 01:05:13.359262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.358931    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-workload-certs\") pod \"421b4772-a3d2-4cf1-aefb-abc655989ad9\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") "
Oct 30 01:05:13.359262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.358969    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-token\" (UniqueName: \"kubernetes.io/projected/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-token\") pod \"421b4772-a3d2-4cf1-aefb-abc655989ad9\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") "
Oct 30 01:05:13.359262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.359118    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-workload-socket\") pod \"421b4772-a3d2-4cf1-aefb-abc655989ad9\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") "
Oct 30 01:05:13.359262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.359183    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-data\") pod \"421b4772-a3d2-4cf1-aefb-abc655989ad9\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") "
Oct 30 01:05:13.359262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.359219    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-envoy\") pod \"421b4772-a3d2-4cf1-aefb-abc655989ad9\" (UID: \"421b4772-a3d2-4cf1-aefb-abc655989ad9\") "
Oct 30 01:05:13.373533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.362249    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-data" pod "421b4772-a3d2-4cf1-aefb-abc655989ad9" (UID: "421b4772-a3d2-4cf1-aefb-abc655989ad9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:13.373533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.362354    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-workload-certs" pod "421b4772-a3d2-4cf1-aefb-abc655989ad9" (UID: "421b4772-a3d2-4cf1-aefb-abc655989ad9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:13.373533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.363263    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-workload-socket" pod "421b4772-a3d2-4cf1-aefb-abc655989ad9" (UID: "421b4772-a3d2-4cf1-aefb-abc655989ad9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:13.373533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.363791    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421b4772-a3d2-4cf1-aefb-abc655989ad9-istiod-ca-cert" pod "421b4772-a3d2-4cf1-aefb-abc655989ad9" (UID: "421b4772-a3d2-4cf1-aefb-abc655989ad9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:13.373533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.364054    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-credential-socket" pod "421b4772-a3d2-4cf1-aefb-abc655989ad9" (UID: "421b4772-a3d2-4cf1-aefb-abc655989ad9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:13.380297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.380260    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-envoy" pod "421b4772-a3d2-4cf1-aefb-abc655989ad9" (UID: "421b4772-a3d2-4cf1-aefb-abc655989ad9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:13.382329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.382302    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-podinfo" pod "421b4772-a3d2-4cf1-aefb-abc655989ad9" (UID: "421b4772-a3d2-4cf1-aefb-abc655989ad9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:05:13.382556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.382536    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421b4772-a3d2-4cf1-aefb-abc655989ad9-kube-api-access-bdwsz" pod "421b4772-a3d2-4cf1-aefb-abc655989ad9" (UID: "421b4772-a3d2-4cf1-aefb-abc655989ad9"). InnerVolumeSpecName "kube-api-access-bdwsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:13.383789 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.383762    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-token" pod "421b4772-a3d2-4cf1-aefb-abc655989ad9" (UID: "421b4772-a3d2-4cf1-aefb-abc655989ad9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:13.461230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.461200    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/421b4772-a3d2-4cf1-aefb-abc655989ad9-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:13.461230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.461227    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:13.461230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.461237    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:13.461431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.461247    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdwsz\" (UniqueName: \"kubernetes.io/projected/421b4772-a3d2-4cf1-aefb-abc655989ad9-kube-api-access-bdwsz\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:13.461431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.461255    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:13.461431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.461263    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:13.461431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.461271    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:13.461431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.461278    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:13.461431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.461287    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/421b4772-a3d2-4cf1-aefb-abc655989ad9-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:13.712101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.712071    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 01:05:13.766403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.766361    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istiod-ca-cert\") pod \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") "
Oct 30 01:05:13.766597 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.766436    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-token\" (UniqueName: \"kubernetes.io/projected/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-token\") pod \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") "
Oct 30 01:05:13.766597 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.766496    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-workload-certs\") pod \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") "
Oct 30 01:05:13.766597 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.766579    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251fdcfb-b27b-4f4e-854b-9f3f70304ace-kube-api-access-kbvrx\" (UniqueName: \"kubernetes.io/projected/251fdcfb-b27b-4f4e-854b-9f3f70304ace-kube-api-access-kbvrx\") pod \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") "
Oct 30 01:05:13.766785 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.766636    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-envoy\") pod \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") "
Oct 30 01:05:13.766785 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.766698    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-workload-socket\") pod \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") "
Oct 30 01:05:13.766896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.766788    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-credential-socket\") pod \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") "
Oct 30 01:05:13.766896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.766847    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-workload-certs" pod "251fdcfb-b27b-4f4e-854b-9f3f70304ace" (UID: "251fdcfb-b27b-4f4e-854b-9f3f70304ace"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:13.766896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.766863    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istiod-ca-cert" pod "251fdcfb-b27b-4f4e-854b-9f3f70304ace" (UID: "251fdcfb-b27b-4f4e-854b-9f3f70304ace"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:13.766896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.766882    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-data\") pod \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") "
Oct 30 01:05:13.767094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.766923    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-podinfo\") pod \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\" (UID: \"251fdcfb-b27b-4f4e-854b-9f3f70304ace\") "
Oct 30 01:05:13.767143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.767105    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-workload-socket" pod "251fdcfb-b27b-4f4e-854b-9f3f70304ace" (UID: "251fdcfb-b27b-4f4e-854b-9f3f70304ace"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:13.767772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.767490    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-data" pod "251fdcfb-b27b-4f4e-854b-9f3f70304ace" (UID: "251fdcfb-b27b-4f4e-854b-9f3f70304ace"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:13.767772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.767540    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:13.767772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.767559    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:13.767772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.767571    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:13.767772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.767664    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-credential-socket" pod "251fdcfb-b27b-4f4e-854b-9f3f70304ace" (UID: "251fdcfb-b27b-4f4e-854b-9f3f70304ace"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:13.769814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.769787    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251fdcfb-b27b-4f4e-854b-9f3f70304ace-kube-api-access-kbvrx" pod "251fdcfb-b27b-4f4e-854b-9f3f70304ace" (UID: "251fdcfb-b27b-4f4e-854b-9f3f70304ace"). InnerVolumeSpecName "kube-api-access-kbvrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:13.769924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.769906    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-podinfo" pod "251fdcfb-b27b-4f4e-854b-9f3f70304ace" (UID: "251fdcfb-b27b-4f4e-854b-9f3f70304ace"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:05:13.769994 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.769919    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-envoy" pod "251fdcfb-b27b-4f4e-854b-9f3f70304ace" (UID: "251fdcfb-b27b-4f4e-854b-9f3f70304ace"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:13.770096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.770077    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-token" pod "251fdcfb-b27b-4f4e-854b-9f3f70304ace" (UID: "251fdcfb-b27b-4f4e-854b-9f3f70304ace"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:13.870096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.869993    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:13.870096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.870031    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:13.870096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.870045    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:13.870096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.870058    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbvrx\" (UniqueName: \"kubernetes.io/projected/251fdcfb-b27b-4f4e-854b-9f3f70304ace-kube-api-access-kbvrx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:13.870096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.870074    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:13.870096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:13.870090    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/251fdcfb-b27b-4f4e-854b-9f3f70304ace-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:14.236237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:14.234664    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx" event={"ID":"251fdcfb-b27b-4f4e-854b-9f3f70304ace","Type":"ContainerDied","Data":"a86313568a587a86d9f96ea2aeae02910ecc3cebec6c31e79e90de04db0ccebd"}
Oct 30 01:05:14.236237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:14.234740    4425 scope.go:122] "RemoveContainer" containerID="cd20f8e9fa01f73b334530f7f24afffbaf97f5d60cb564880285bc4476ab2058"
Oct 30 01:05:14.236237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:14.234886    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"
Oct 30 01:05:14.246681 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:14.246659    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"
Oct 30 01:05:14.246920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:14.246649    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls" event={"ID":"421b4772-a3d2-4cf1-aefb-abc655989ad9","Type":"ContainerDied","Data":"df9ac6f381e407faad8c50de5e6831e4981600d38b007821ef0f33733a2641f9"}
Oct 30 01:05:14.266126 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:14.266104    4425 scope.go:122] "RemoveContainer" containerID="21de5a9a998b22a17887be397d1bf9ba7c70d1fbcdf0433d0548786413cf2ea3"
Oct 30 01:05:14.298665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:14.298636    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"]
Oct 30 01:05:14.307578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:14.307549    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-gnqgx"]
Oct 30 01:05:14.333789 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:14.333756    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"]
Oct 30 01:05:14.350218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:14.349775    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-tdzls"]
Oct 30 01:05:15.423913 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:15.423879    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251fdcfb-b27b-4f4e-854b-9f3f70304ace" path="/var/lib/kubelet/pods/251fdcfb-b27b-4f4e-854b-9f3f70304ace/volumes"
Oct 30 01:05:15.424388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:15.424374    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="421b4772-a3d2-4cf1-aefb-abc655989ad9" path="/var/lib/kubelet/pods/421b4772-a3d2-4cf1-aefb-abc655989ad9/volumes"
Oct 30 01:05:16.861263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:16.861207    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj" podUID="7cba143d-77d0-467f-af24-07888c71e960" containerName="payload-processing" containerID="cri-o://44d6dca66d56db84e3fc702b056bd51c1a4fdd7d472b238a27f1b021ec3f5887" gracePeriod=25
Oct 30 01:05:16.888066 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:16.888011    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f" podUID="94ebc6cf-1479-4394-842c-26395091cedd" containerName="payload-pre-processing" containerID="cri-o://2fb91750a75440b03719751fac5dd7debee2be899e70306e7fef1162c94fe2d0" gracePeriod=25
Oct 30 01:05:17.250682 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.250340    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"
Oct 30 01:05:17.278233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.277940    4425 generic.go:356] "Generic (PLEG): container finished" podID="94ebc6cf-1479-4394-842c-26395091cedd" containerID="2fb91750a75440b03719751fac5dd7debee2be899e70306e7fef1162c94fe2d0" exitCode=0
Oct 30 01:05:17.278233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.278182    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f" event={"ID":"94ebc6cf-1479-4394-842c-26395091cedd","Type":"ContainerDied","Data":"2fb91750a75440b03719751fac5dd7debee2be899e70306e7fef1162c94fe2d0"}
Oct 30 01:05:17.278233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.278242    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f" event={"ID":"94ebc6cf-1479-4394-842c-26395091cedd","Type":"ContainerDied","Data":"2d7cdf2a8fcd9cf4eec11f77ed31218c49f8ef999070b1d54fbad326e8b423cb"}
Oct 30 01:05:17.278455 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.278270    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d7cdf2a8fcd9cf4eec11f77ed31218c49f8ef999070b1d54fbad326e8b423cb"
Oct 30 01:05:17.280037 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.280009    4425 generic.go:356] "Generic (PLEG): container finished" podID="7cba143d-77d0-467f-af24-07888c71e960" containerID="44d6dca66d56db84e3fc702b056bd51c1a4fdd7d472b238a27f1b021ec3f5887" exitCode=0
Oct 30 01:05:17.280143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.280058    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"
Oct 30 01:05:17.280143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.280063    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj" event={"ID":"7cba143d-77d0-467f-af24-07888c71e960","Type":"ContainerDied","Data":"44d6dca66d56db84e3fc702b056bd51c1a4fdd7d472b238a27f1b021ec3f5887"}
Oct 30 01:05:17.280298 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.280214    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj" event={"ID":"7cba143d-77d0-467f-af24-07888c71e960","Type":"ContainerDied","Data":"a47de4455372e8b83db01bd7b19046b80466f3460027ba96527551cdf603bbe8"}
Oct 30 01:05:17.280298 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.280231    4425 scope.go:122] "RemoveContainer" containerID="44d6dca66d56db84e3fc702b056bd51c1a4fdd7d472b238a27f1b021ec3f5887"
Oct 30 01:05:17.295380 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.295354    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"
Oct 30 01:05:17.295569 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.295527    4425 scope.go:122] "RemoveContainer" containerID="44d6dca66d56db84e3fc702b056bd51c1a4fdd7d472b238a27f1b021ec3f5887"
Oct 30 01:05:17.295899 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:05:17.295874    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d6dca66d56db84e3fc702b056bd51c1a4fdd7d472b238a27f1b021ec3f5887\": container with ID starting with 44d6dca66d56db84e3fc702b056bd51c1a4fdd7d472b238a27f1b021ec3f5887 not found: ID does not exist" containerID="44d6dca66d56db84e3fc702b056bd51c1a4fdd7d472b238a27f1b021ec3f5887"
Oct 30 01:05:17.296004 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.295910    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d6dca66d56db84e3fc702b056bd51c1a4fdd7d472b238a27f1b021ec3f5887"} err="failed to get container status \"44d6dca66d56db84e3fc702b056bd51c1a4fdd7d472b238a27f1b021ec3f5887\": rpc error: code = NotFound desc = could not find container \"44d6dca66d56db84e3fc702b056bd51c1a4fdd7d472b238a27f1b021ec3f5887\": container with ID starting with 44d6dca66d56db84e3fc702b056bd51c1a4fdd7d472b238a27f1b021ec3f5887 not found: ID does not exist"
Oct 30 01:05:17.326636 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.326595    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cba143d-77d0-467f-af24-07888c71e960-kube-api-access-mtkc6\" (UniqueName: \"kubernetes.io/projected/7cba143d-77d0-467f-af24-07888c71e960-kube-api-access-mtkc6\") pod \"7cba143d-77d0-467f-af24-07888c71e960\" (UID: \"7cba143d-77d0-467f-af24-07888c71e960\") "
Oct 30 01:05:17.326872 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.326852    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cba143d-77d0-467f-af24-07888c71e960-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cba143d-77d0-467f-af24-07888c71e960-plugins-config-volume\") pod \"7cba143d-77d0-467f-af24-07888c71e960\" (UID: \"7cba143d-77d0-467f-af24-07888c71e960\") "
Oct 30 01:05:17.327361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.327333    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cba143d-77d0-467f-af24-07888c71e960-plugins-config-volume" pod "7cba143d-77d0-467f-af24-07888c71e960" (UID: "7cba143d-77d0-467f-af24-07888c71e960"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:17.327510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.327492    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cba143d-77d0-467f-af24-07888c71e960-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:17.329558 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.329528    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cba143d-77d0-467f-af24-07888c71e960-kube-api-access-mtkc6" pod "7cba143d-77d0-467f-af24-07888c71e960" (UID: "7cba143d-77d0-467f-af24-07888c71e960"). InnerVolumeSpecName "kube-api-access-mtkc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:17.428314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.428241    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94ebc6cf-1479-4394-842c-26395091cedd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ebc6cf-1479-4394-842c-26395091cedd-plugins-config-volume\") pod \"94ebc6cf-1479-4394-842c-26395091cedd\" (UID: \"94ebc6cf-1479-4394-842c-26395091cedd\") "
Oct 30 01:05:17.428457 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.428444    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ebc6cf-1479-4394-842c-26395091cedd-kube-api-access-5qmth\" (UniqueName: \"kubernetes.io/projected/94ebc6cf-1479-4394-842c-26395091cedd-kube-api-access-5qmth\") pod \"94ebc6cf-1479-4394-842c-26395091cedd\" (UID: \"94ebc6cf-1479-4394-842c-26395091cedd\") "
Oct 30 01:05:17.428706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.428677    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ebc6cf-1479-4394-842c-26395091cedd-plugins-config-volume" pod "94ebc6cf-1479-4394-842c-26395091cedd" (UID: "94ebc6cf-1479-4394-842c-26395091cedd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:17.428885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.428865    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94ebc6cf-1479-4394-842c-26395091cedd-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:17.428935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.428891    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtkc6\" (UniqueName: \"kubernetes.io/projected/7cba143d-77d0-467f-af24-07888c71e960-kube-api-access-mtkc6\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:17.430633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.430610    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ebc6cf-1479-4394-842c-26395091cedd-kube-api-access-5qmth" pod "94ebc6cf-1479-4394-842c-26395091cedd" (UID: "94ebc6cf-1479-4394-842c-26395091cedd"). InnerVolumeSpecName "kube-api-access-5qmth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:17.530015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.529969    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qmth\" (UniqueName: \"kubernetes.io/projected/94ebc6cf-1479-4394-842c-26395091cedd-kube-api-access-5qmth\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:17.617263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.617232    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"]
Oct 30 01:05:17.626607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:17.626566    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c78b8-54dfc86694-nkkbj"]
Oct 30 01:05:18.289208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:18.289174    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"
Oct 30 01:05:18.336073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:18.336038    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"]
Oct 30 01:05:18.344132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:18.344102    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c78b8-68755cc4c-5mp2f"]
Oct 30 01:05:18.641679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:18.641601    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"]
Oct 30 01:05:19.424384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:19.424350    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cba143d-77d0-467f-af24-07888c71e960" path="/var/lib/kubelet/pods/7cba143d-77d0-467f-af24-07888c71e960/volumes"
Oct 30 01:05:19.424886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:19.424792    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94ebc6cf-1479-4394-842c-26395091cedd" path="/var/lib/kubelet/pods/94ebc6cf-1479-4394-842c-26395091cedd/volumes"
Oct 30 01:05:23.642953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:23.642904    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv" podUID="ddddff32-cf5c-4b61-8716-fc388b03216f" containerName="payload-pre-processing" containerID="cri-o://d2d5d78f3e9c3e7c82242b02a73455ffbb74e619feae90567a349c07b3f66581" gracePeriod=25
Oct 30 01:05:23.910682 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:23.910615    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"
Oct 30 01:05:24.015068 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:24.015036    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddddff32-cf5c-4b61-8716-fc388b03216f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddddff32-cf5c-4b61-8716-fc388b03216f-plugins-config-volume\") pod \"ddddff32-cf5c-4b61-8716-fc388b03216f\" (UID: \"ddddff32-cf5c-4b61-8716-fc388b03216f\") "
Oct 30 01:05:24.015261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:24.015083    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddddff32-cf5c-4b61-8716-fc388b03216f-kube-api-access-pczrn\" (UniqueName: \"kubernetes.io/projected/ddddff32-cf5c-4b61-8716-fc388b03216f-kube-api-access-pczrn\") pod \"ddddff32-cf5c-4b61-8716-fc388b03216f\" (UID: \"ddddff32-cf5c-4b61-8716-fc388b03216f\") "
Oct 30 01:05:24.015449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:24.015424    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddddff32-cf5c-4b61-8716-fc388b03216f-plugins-config-volume" pod "ddddff32-cf5c-4b61-8716-fc388b03216f" (UID: "ddddff32-cf5c-4b61-8716-fc388b03216f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:24.015672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:24.015658    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddddff32-cf5c-4b61-8716-fc388b03216f-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:24.017148 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:24.017119    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddddff32-cf5c-4b61-8716-fc388b03216f-kube-api-access-pczrn" pod "ddddff32-cf5c-4b61-8716-fc388b03216f" (UID: "ddddff32-cf5c-4b61-8716-fc388b03216f"). InnerVolumeSpecName "kube-api-access-pczrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:24.120202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:24.120146    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pczrn\" (UniqueName: \"kubernetes.io/projected/ddddff32-cf5c-4b61-8716-fc388b03216f-kube-api-access-pczrn\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:24.333757 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:24.333722    4425 generic.go:356] "Generic (PLEG): container finished" podID="ddddff32-cf5c-4b61-8716-fc388b03216f" containerID="d2d5d78f3e9c3e7c82242b02a73455ffbb74e619feae90567a349c07b3f66581" exitCode=0
Oct 30 01:05:24.333910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:24.333772    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"
Oct 30 01:05:24.333910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:24.333800    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv" event={"ID":"ddddff32-cf5c-4b61-8716-fc388b03216f","Type":"ContainerDied","Data":"d2d5d78f3e9c3e7c82242b02a73455ffbb74e619feae90567a349c07b3f66581"}
Oct 30 01:05:24.333910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:24.333854    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv" event={"ID":"ddddff32-cf5c-4b61-8716-fc388b03216f","Type":"ContainerDied","Data":"0f57e7d603016e50889fa6f80458724fcfa3b5b29140ffc2dfa83b996262e75c"}
Oct 30 01:05:24.333910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:24.333876    4425 scope.go:122] "RemoveContainer" containerID="d2d5d78f3e9c3e7c82242b02a73455ffbb74e619feae90567a349c07b3f66581"
Oct 30 01:05:24.344374 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:24.344357    4425 scope.go:122] "RemoveContainer" containerID="d2d5d78f3e9c3e7c82242b02a73455ffbb74e619feae90567a349c07b3f66581"
Oct 30 01:05:24.344676 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:05:24.344653    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d5d78f3e9c3e7c82242b02a73455ffbb74e619feae90567a349c07b3f66581\": container with ID starting with d2d5d78f3e9c3e7c82242b02a73455ffbb74e619feae90567a349c07b3f66581 not found: ID does not exist" containerID="d2d5d78f3e9c3e7c82242b02a73455ffbb74e619feae90567a349c07b3f66581"
Oct 30 01:05:24.344718 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:24.344684    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d5d78f3e9c3e7c82242b02a73455ffbb74e619feae90567a349c07b3f66581"} err="failed to get container status \"d2d5d78f3e9c3e7c82242b02a73455ffbb74e619feae90567a349c07b3f66581\": rpc error: code = NotFound desc = could not find container \"d2d5d78f3e9c3e7c82242b02a73455ffbb74e619feae90567a349c07b3f66581\": container with ID starting with d2d5d78f3e9c3e7c82242b02a73455ffbb74e619feae90567a349c07b3f66581 not found: ID does not exist"
Oct 30 01:05:24.379744 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:24.379695    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"]
Oct 30 01:05:24.388856 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:24.388829    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d4622a-7f798db8cf-whmsv"]
Oct 30 01:05:25.423097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:25.423059    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddddff32-cf5c-4b61-8716-fc388b03216f" path="/var/lib/kubelet/pods/ddddff32-cf5c-4b61-8716-fc388b03216f/volumes"
Oct 30 01:05:28.960675 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:28.960633    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"]
Oct 30 01:05:28.961238 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:28.960952    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j" podUID="11e986da-b068-404f-91f7-4a8f109465d7" containerName="istio-proxy" containerID="cri-o://2e4d5be578242ec18254ac72479feeebd360f514144efa950b13c5605f6a0208" gracePeriod=30
Oct 30 01:05:28.969774 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:28.969748    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"]
Oct 30 01:05:28.969974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:28.969950    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq" podUID="7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" containerName="istio-proxy" containerID="cri-o://d49e77c5bc636e47b7173ac6c8d60455a27eb41966adf89adc6060cbbe73d94a" gracePeriod=30
Oct 30 01:05:31.687140 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:31.687104    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"]
Oct 30 01:05:31.697417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:31.697383    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-645126-92db8"]
Oct 30 01:05:32.082382 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:32.082355    4425 scope.go:122] "RemoveContainer" containerID="7c00979705535911102cf16fc745d0e18da9b933c0eb2557f0e28e98eafa2510"
Oct 30 01:05:32.091858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:32.091841    4425 scope.go:122] "RemoveContainer" containerID="b3a18ce18824968f8ac7e40152f5927c449e18e790e50a0ef737f3fddd0a35fb"
Oct 30 01:05:32.100973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:32.100952    4425 scope.go:122] "RemoveContainer" containerID="ceae8b352770ffe17708d8e59b3b8d9891aa3ad2e3bd8af56d0a267a91bcb6d4"
Oct 30 01:05:33.422335 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:33.422304    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47202689-31b2-4e36-9c83-0c09cedfc8a8" path="/var/lib/kubelet/pods/47202689-31b2-4e36-9c83-0c09cedfc8a8/volumes"
Oct 30 01:05:34.263677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.263649    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 01:05:34.289212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.289184    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 01:05:34.364684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.364652    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-credential-socket\") pod \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") "
Oct 30 01:05:34.364944 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.364701    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-podinfo\") pod \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") "
Oct 30 01:05:34.364944 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.364777    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e986da-b068-404f-91f7-4a8f109465d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e986da-b068-404f-91f7-4a8f109465d7-istiod-ca-cert\") pod \"11e986da-b068-404f-91f7-4a8f109465d7\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") "
Oct 30 01:05:34.365056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.364971    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-credential-socket" pod "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" (UID: "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.365106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365095    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e986da-b068-404f-91f7-4a8f109465d7-istio-token\" (UniqueName: \"kubernetes.io/projected/11e986da-b068-404f-91f7-4a8f109465d7-istio-token\") pod \"11e986da-b068-404f-91f7-4a8f109465d7\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") "
Oct 30 01:05:34.365200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365138    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-data\") pod \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") "
Oct 30 01:05:34.365200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365179    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e986da-b068-404f-91f7-4a8f109465d7-istiod-ca-cert" pod "11e986da-b068-404f-91f7-4a8f109465d7" (UID: "11e986da-b068-404f-91f7-4a8f109465d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:34.365330 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365187    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-workload-certs\") pod \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") "
Oct 30 01:05:34.365330 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365247    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istiod-ca-cert\") pod \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") "
Oct 30 01:05:34.365440 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365381    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-workload-certs" pod "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" (UID: "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.365440 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365416    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-credential-socket\") pod \"11e986da-b068-404f-91f7-4a8f109465d7\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") "
Oct 30 01:05:34.365440 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365426    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-data" pod "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" (UID: "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.365599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365463    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-istio-envoy\") pod \"11e986da-b068-404f-91f7-4a8f109465d7\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") "
Oct 30 01:05:34.365599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365498    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e986da-b068-404f-91f7-4a8f109465d7-kube-api-access-8jrpt\" (UniqueName: \"kubernetes.io/projected/11e986da-b068-404f-91f7-4a8f109465d7-kube-api-access-8jrpt\") pod \"11e986da-b068-404f-91f7-4a8f109465d7\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") "
Oct 30 01:05:34.365599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365540    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-workload-socket\") pod \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") "
Oct 30 01:05:34.365599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365566    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-istio-data\") pod \"11e986da-b068-404f-91f7-4a8f109465d7\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") "
Oct 30 01:05:34.365816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365601    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-envoy\") pod \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") "
Oct 30 01:05:34.365816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365645    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-token\" (UniqueName: \"kubernetes.io/projected/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-token\") pod \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") "
Oct 30 01:05:34.365816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365677    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-kube-api-access-9jq77\" (UniqueName: \"kubernetes.io/projected/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-kube-api-access-9jq77\") pod \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\" (UID: \"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5\") "
Oct 30 01:05:34.365816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365688    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istiod-ca-cert" pod "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" (UID: "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:34.365816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365716    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-credential-socket" pod "11e986da-b068-404f-91f7-4a8f109465d7" (UID: "11e986da-b068-404f-91f7-4a8f109465d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.365816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365752    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-workload-certs\") pod \"11e986da-b068-404f-91f7-4a8f109465d7\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") "
Oct 30 01:05:34.365816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365803    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11e986da-b068-404f-91f7-4a8f109465d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e986da-b068-404f-91f7-4a8f109465d7-istio-podinfo\") pod \"11e986da-b068-404f-91f7-4a8f109465d7\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") "
Oct 30 01:05:34.367344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365831    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-workload-socket\") pod \"11e986da-b068-404f-91f7-4a8f109465d7\" (UID: \"11e986da-b068-404f-91f7-4a8f109465d7\") "
Oct 30 01:05:34.367344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365916    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-workload-socket" pod "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" (UID: "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.367344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.365963    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-istio-data" pod "11e986da-b068-404f-91f7-4a8f109465d7" (UID: "11e986da-b068-404f-91f7-4a8f109465d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.367344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.366196    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-workload-socket" pod "11e986da-b068-404f-91f7-4a8f109465d7" (UID: "11e986da-b068-404f-91f7-4a8f109465d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.367344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.366544    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.367344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.366564    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.367344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.366580    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11e986da-b068-404f-91f7-4a8f109465d7-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.367344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.366592    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.367344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.366604    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.367344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.366617    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.367344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.366630    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.367344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.366643    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.367344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.366672    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.368043 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.367585    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-workload-certs" pod "11e986da-b068-404f-91f7-4a8f109465d7" (UID: "11e986da-b068-404f-91f7-4a8f109465d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.368043 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.367732    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e986da-b068-404f-91f7-4a8f109465d7-istio-token" pod "11e986da-b068-404f-91f7-4a8f109465d7" (UID: "11e986da-b068-404f-91f7-4a8f109465d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:34.368236 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.368209    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-podinfo" pod "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" (UID: "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:05:34.369977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.369945    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e986da-b068-404f-91f7-4a8f109465d7-kube-api-access-8jrpt" pod "11e986da-b068-404f-91f7-4a8f109465d7" (UID: "11e986da-b068-404f-91f7-4a8f109465d7"). InnerVolumeSpecName "kube-api-access-8jrpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:34.370260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.370234    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-kube-api-access-9jq77" pod "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" (UID: "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5"). InnerVolumeSpecName "kube-api-access-9jq77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:34.370716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.370587    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-token" pod "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" (UID: "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:34.370809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.370773    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-envoy" pod "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" (UID: "7c4a4f6f-8a1b-4114-be9a-bad89cb053e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.370862 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.370801    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11e986da-b068-404f-91f7-4a8f109465d7-istio-podinfo" pod "11e986da-b068-404f-91f7-4a8f109465d7" (UID: "11e986da-b068-404f-91f7-4a8f109465d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:05:34.371484 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.371458    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-istio-envoy" pod "11e986da-b068-404f-91f7-4a8f109465d7" (UID: "11e986da-b068-404f-91f7-4a8f109465d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:34.409462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.409423    4425 generic.go:356] "Generic (PLEG): container finished" podID="7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" containerID="d49e77c5bc636e47b7173ac6c8d60455a27eb41966adf89adc6060cbbe73d94a" exitCode=0
Oct 30 01:05:34.409612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.409511    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"
Oct 30 01:05:34.409612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.409516    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq" event={"ID":"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5","Type":"ContainerDied","Data":"d49e77c5bc636e47b7173ac6c8d60455a27eb41966adf89adc6060cbbe73d94a"}
Oct 30 01:05:34.409612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.409606    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq" event={"ID":"7c4a4f6f-8a1b-4114-be9a-bad89cb053e5","Type":"ContainerDied","Data":"a8d83ef31e6e886a6a9e97155319c71dd8d8a472518d972a732e7b01619d2321"}
Oct 30 01:05:34.409787 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.409639    4425 scope.go:122] "RemoveContainer" containerID="d49e77c5bc636e47b7173ac6c8d60455a27eb41966adf89adc6060cbbe73d94a"
Oct 30 01:05:34.411189 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.411133    4425 generic.go:356] "Generic (PLEG): container finished" podID="11e986da-b068-404f-91f7-4a8f109465d7" containerID="2e4d5be578242ec18254ac72479feeebd360f514144efa950b13c5605f6a0208" exitCode=0
Oct 30 01:05:34.411327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.411270    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"
Oct 30 01:05:34.411542 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.411514    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j" event={"ID":"11e986da-b068-404f-91f7-4a8f109465d7","Type":"ContainerDied","Data":"2e4d5be578242ec18254ac72479feeebd360f514144efa950b13c5605f6a0208"}
Oct 30 01:05:34.411700 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.411578    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j" event={"ID":"11e986da-b068-404f-91f7-4a8f109465d7","Type":"ContainerDied","Data":"79de5a34941ff05a07a21dc6a3996e2825eecf2aae2387ebf0c6b54ac5a4a52a"}
Oct 30 01:05:34.422219 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.421980    4425 scope.go:122] "RemoveContainer" containerID="d49e77c5bc636e47b7173ac6c8d60455a27eb41966adf89adc6060cbbe73d94a"
Oct 30 01:05:34.422319 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:05:34.422299    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d49e77c5bc636e47b7173ac6c8d60455a27eb41966adf89adc6060cbbe73d94a\": container with ID starting with d49e77c5bc636e47b7173ac6c8d60455a27eb41966adf89adc6060cbbe73d94a not found: ID does not exist" containerID="d49e77c5bc636e47b7173ac6c8d60455a27eb41966adf89adc6060cbbe73d94a"
Oct 30 01:05:34.422387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.422328    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49e77c5bc636e47b7173ac6c8d60455a27eb41966adf89adc6060cbbe73d94a"} err="failed to get container status \"d49e77c5bc636e47b7173ac6c8d60455a27eb41966adf89adc6060cbbe73d94a\": rpc error: code = NotFound desc = could not find container \"d49e77c5bc636e47b7173ac6c8d60455a27eb41966adf89adc6060cbbe73d94a\": container with ID starting with d49e77c5bc636e47b7173ac6c8d60455a27eb41966adf89adc6060cbbe73d94a not found: ID does not exist"
Oct 30 01:05:34.422387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.422349    4425 scope.go:122] "RemoveContainer" containerID="2e4d5be578242ec18254ac72479feeebd360f514144efa950b13c5605f6a0208"
Oct 30 01:05:34.431483 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.431465    4425 scope.go:122] "RemoveContainer" containerID="2e4d5be578242ec18254ac72479feeebd360f514144efa950b13c5605f6a0208"
Oct 30 01:05:34.431848 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:05:34.431808    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e4d5be578242ec18254ac72479feeebd360f514144efa950b13c5605f6a0208\": container with ID starting with 2e4d5be578242ec18254ac72479feeebd360f514144efa950b13c5605f6a0208 not found: ID does not exist" containerID="2e4d5be578242ec18254ac72479feeebd360f514144efa950b13c5605f6a0208"
Oct 30 01:05:34.431937 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.431842    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e4d5be578242ec18254ac72479feeebd360f514144efa950b13c5605f6a0208"} err="failed to get container status \"2e4d5be578242ec18254ac72479feeebd360f514144efa950b13c5605f6a0208\": rpc error: code = NotFound desc = could not find container \"2e4d5be578242ec18254ac72479feeebd360f514144efa950b13c5605f6a0208\": container with ID starting with 2e4d5be578242ec18254ac72479feeebd360f514144efa950b13c5605f6a0208 not found: ID does not exist"
Oct 30 01:05:34.458039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.458014    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"]
Oct 30 01:05:34.462056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.462034    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-ztmqq"]
Oct 30 01:05:34.467512 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.467494    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jq77\" (UniqueName: \"kubernetes.io/projected/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-kube-api-access-9jq77\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.467596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.467514    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.467596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.467523    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11e986da-b068-404f-91f7-4a8f109465d7-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.467596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.467532    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.467596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.467539    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11e986da-b068-404f-91f7-4a8f109465d7-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.467596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.467547    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11e986da-b068-404f-91f7-4a8f109465d7-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.467596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.467554    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jrpt\" (UniqueName: \"kubernetes.io/projected/11e986da-b068-404f-91f7-4a8f109465d7-kube-api-access-8jrpt\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.467596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.467563    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.467596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.467570    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:34.491470 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.491404    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"]
Oct 30 01:05:34.497198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:34.497174    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-65p8j"]
Oct 30 01:05:35.426194 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:35.426115    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11e986da-b068-404f-91f7-4a8f109465d7" path="/var/lib/kubelet/pods/11e986da-b068-404f-91f7-4a8f109465d7/volumes"
Oct 30 01:05:35.426906 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:35.426884    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" path="/var/lib/kubelet/pods/7c4a4f6f-8a1b-4114-be9a-bad89cb053e5/volumes"
Oct 30 01:05:37.554606 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:05:37.554571    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:05:40.925178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:40.924601    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:05:43.123571 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:43.123531    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"]
Oct 30 01:05:43.134122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:43.134092    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c78b8-xwf57"]
Oct 30 01:05:43.426675 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:43.426595    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a88c37-7411-45c7-811d-62ae73ce4fff" path="/var/lib/kubelet/pods/54a88c37-7411-45c7-811d-62ae73ce4fff/volumes"
Oct 30 01:05:49.484242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484209    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"]
Oct 30 01:05:49.484711 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484698    4425 cpu_manager.go:414] "containerMap: removing container" podUID="251fdcfb-b27b-4f4e-854b-9f3f70304ace" containerName="istio-proxy"
Oct 30 01:05:49.484763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484712    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251fdcfb-b27b-4f4e-854b-9f3f70304ace" containerName="istio-proxy"
Oct 30 01:05:49.484763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484726    4425 cpu_manager.go:414] "containerMap: removing container" podUID="11e986da-b068-404f-91f7-4a8f109465d7" containerName="istio-proxy"
Oct 30 01:05:49.484763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484731    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e986da-b068-404f-91f7-4a8f109465d7" containerName="istio-proxy"
Oct 30 01:05:49.484763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484741    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ddddff32-cf5c-4b61-8716-fc388b03216f" containerName="payload-pre-processing"
Oct 30 01:05:49.484763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484746    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddddff32-cf5c-4b61-8716-fc388b03216f" containerName="payload-pre-processing"
Oct 30 01:05:49.484763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484757    4425 cpu_manager.go:414] "containerMap: removing container" podUID="94ebc6cf-1479-4394-842c-26395091cedd" containerName="payload-pre-processing"
Oct 30 01:05:49.484763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484762    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ebc6cf-1479-4394-842c-26395091cedd" containerName="payload-pre-processing"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484770    4425 cpu_manager.go:414] "containerMap: removing container" podUID="421b4772-a3d2-4cf1-aefb-abc655989ad9" containerName="istio-proxy"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484774    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="421b4772-a3d2-4cf1-aefb-abc655989ad9" containerName="istio-proxy"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484784    4425 cpu_manager.go:414] "containerMap: removing container" podUID="2fb77042-ce3f-4913-b677-394cd6520fc0" containerName="payload-processing"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484789    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb77042-ce3f-4913-b677-394cd6520fc0" containerName="payload-processing"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484800    4425 cpu_manager.go:414] "containerMap: removing container" podUID="7cba143d-77d0-467f-af24-07888c71e960" containerName="payload-processing"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484804    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cba143d-77d0-467f-af24-07888c71e960" containerName="payload-processing"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484810    4425 cpu_manager.go:414] "containerMap: removing container" podUID="0eabc38b-35ba-4133-a05d-bba79769f477" containerName="payload-pre-processing"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484815    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eabc38b-35ba-4133-a05d-bba79769f477" containerName="payload-pre-processing"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484821    4425 cpu_manager.go:414] "containerMap: removing container" podUID="83443474-b817-4ba6-afb6-dbe9a7bc24f0" containerName="istio-proxy"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484825    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83443474-b817-4ba6-afb6-dbe9a7bc24f0" containerName="istio-proxy"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484834    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8d56cb59-ad4a-47ac-9107-5160fe97dfad" containerName="istio-proxy"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484838    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d56cb59-ad4a-47ac-9107-5160fe97dfad" containerName="istio-proxy"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484848    4425 cpu_manager.go:414] "containerMap: removing container" podUID="7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" containerName="istio-proxy"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484852    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" containerName="istio-proxy"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484863    4425 cpu_manager.go:414] "containerMap: removing container" podUID="54a88c37-7411-45c7-811d-62ae73ce4fff" containerName="revoke-keys"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484867    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a88c37-7411-45c7-811d-62ae73ce4fff" containerName="revoke-keys"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484937    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a88c37-7411-45c7-811d-62ae73ce4fff" containerName="revoke-keys"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484947    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="251fdcfb-b27b-4f4e-854b-9f3f70304ace" containerName="istio-proxy"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484955    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e986da-b068-404f-91f7-4a8f109465d7" containerName="istio-proxy"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484962    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddddff32-cf5c-4b61-8716-fc388b03216f" containerName="payload-pre-processing"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484969    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ebc6cf-1479-4394-842c-26395091cedd" containerName="payload-pre-processing"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484976    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="421b4772-a3d2-4cf1-aefb-abc655989ad9" containerName="istio-proxy"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484983    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb77042-ce3f-4913-b677-394cd6520fc0" containerName="payload-processing"
Oct 30 01:05:49.484983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484990    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cba143d-77d0-467f-af24-07888c71e960" containerName="payload-processing"
Oct 30 01:05:49.485716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.484996    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eabc38b-35ba-4133-a05d-bba79769f477" containerName="payload-pre-processing"
Oct 30 01:05:49.485716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.485002    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="83443474-b817-4ba6-afb6-dbe9a7bc24f0" containerName="istio-proxy"
Oct 30 01:05:49.485716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.485009    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d56cb59-ad4a-47ac-9107-5160fe97dfad" containerName="istio-proxy"
Oct 30 01:05:49.485716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.485015    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c4a4f6f-8a1b-4114-be9a-bad89cb053e5" containerName="istio-proxy"
Oct 30 01:05:49.489966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.489947    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"
Oct 30 01:05:49.502229 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.502210    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:05:49.511965 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.511942    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:05:49.512115 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.511958    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:05:49.512115 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.511952    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:05:49.516401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.516378    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"]
Oct 30 01:05:49.569861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.569829    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgtnm\" (UniqueName: \"kubernetes.io/projected/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-kube-api-access-dgtnm\") pod \"maas-api-revoke-keys-e2e-api-b-76f654-kwrrf\" (UID: \"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"
Oct 30 01:05:49.570000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.569982    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76f654-kwrrf\" (UID: \"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"
Oct 30 01:05:49.570044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.570035    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76f654-kwrrf\" (UID: \"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"
Oct 30 01:05:49.670840 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.670811    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76f654-kwrrf\" (UID: \"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"
Oct 30 01:05:49.670998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.670856    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76f654-kwrrf\" (UID: \"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"
Oct 30 01:05:49.670998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.670974    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgtnm\" (UniqueName: \"kubernetes.io/projected/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-kube-api-access-dgtnm\") pod \"maas-api-revoke-keys-e2e-api-b-76f654-kwrrf\" (UID: \"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"
Oct 30 01:05:49.671871 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.671845    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76f654-kwrrf\" (UID: \"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"
Oct 30 01:05:49.673338 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.673317    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76f654-kwrrf\" (UID: \"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"
Oct 30 01:05:49.691594 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.691563    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgtnm\" (UniqueName: \"kubernetes.io/projected/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-kube-api-access-dgtnm\") pod \"maas-api-revoke-keys-e2e-api-b-76f654-kwrrf\" (UID: \"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"
Oct 30 01:05:49.803757 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:49.803729    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"
Oct 30 01:05:50.111030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:50.111003    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"]
Oct 30 01:05:50.112481 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:05:50.112455    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859eeb23_8ca1_4f5c_bda7_3c6ee3b72d09.slice/crio-f55be8998a84e233cb2862741b012b4706e0df436a64f302f16cef19bffc51f2 WatchSource:0}: Error finding container f55be8998a84e233cb2862741b012b4706e0df436a64f302f16cef19bffc51f2: Status 404 returned error can't find the container with id f55be8998a84e233cb2862741b012b4706e0df436a64f302f16cef19bffc51f2
Oct 30 01:05:50.530746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:50.530704    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf" event={"ID":"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09","Type":"ContainerStarted","Data":"de6ffc8d49d69f4343c624eda31fc68e3d4e3c8f8810cd98d17a75feeeb4e5de"}
Oct 30 01:05:50.531240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:50.530759    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf" event={"ID":"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09","Type":"ContainerStarted","Data":"f55be8998a84e233cb2862741b012b4706e0df436a64f302f16cef19bffc51f2"}
Oct 30 01:05:50.578197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:50.578127    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf" podStartSLOduration=1.578102071 podStartE2EDuration="1.578102071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:50.567135555 +0000 UTC m=+1459.316790865" watchObservedRunningTime="2026-10-30 01:05:50.578102071 +0000 UTC m=+1459.327757380"
Oct 30 01:05:51.538897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:51.538861    4425 generic.go:356] "Generic (PLEG): container finished" podID="859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09" containerID="de6ffc8d49d69f4343c624eda31fc68e3d4e3c8f8810cd98d17a75feeeb4e5de" exitCode=0
Oct 30 01:05:51.539434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:51.538949    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf" event={"ID":"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09","Type":"ContainerDied","Data":"de6ffc8d49d69f4343c624eda31fc68e3d4e3c8f8810cd98d17a75feeeb4e5de"}
Oct 30 01:05:52.716877 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:52.716851    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"
Oct 30 01:05:52.827816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:52.827726    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-kube-api-access-dgtnm\" (UniqueName: \"kubernetes.io/projected/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-kube-api-access-dgtnm\") pod \"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09\" (UID: \"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09\") "
Oct 30 01:05:52.828202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:52.828178    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-maas-api-openshift-ca\") pod \"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09\" (UID: \"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09\") "
Oct 30 01:05:52.828479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:52.828461    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-maas-api-xks-ca\") pod \"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09\" (UID: \"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09\") "
Oct 30 01:05:52.830859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:52.830834    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-maas-api-openshift-ca" pod "859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09" (UID: "859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:52.833672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:52.833631    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-kube-api-access-dgtnm" pod "859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09" (UID: "859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09"). InnerVolumeSpecName "kube-api-access-dgtnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:52.841010 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:52.840986    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-maas-api-xks-ca" pod "859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09" (UID: "859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:52.930494 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:52.930462    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgtnm\" (UniqueName: \"kubernetes.io/projected/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-kube-api-access-dgtnm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:52.930494 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:52.930493    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:52.930707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:52.930508    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:05:53.062177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.061581    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"]
Oct 30 01:05:53.066175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.062416    4425 cpu_manager.go:414] "containerMap: removing container" podUID="859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09" containerName="revoke-keys"
Oct 30 01:05:53.066175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.062440    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09" containerName="revoke-keys"
Oct 30 01:05:53.066175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.062569    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09" containerName="revoke-keys"
Oct 30 01:05:53.076289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.074866    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"
Oct 30 01:05:53.096346 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.089907    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-11fa15\""
Oct 30 01:05:53.096346 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.089997    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-11fa15-dockercfg-bs5qt\""
Oct 30 01:05:53.096346 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.093525    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"]
Oct 30 01:05:53.137143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.137097    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1190e6ca-0f37-45a6-a01d-662b189a88b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52\" (UID: \"1190e6ca-0f37-45a6-a01d-662b189a88b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"
Oct 30 01:05:53.137462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.137437    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxsrh\" (UniqueName: \"kubernetes.io/projected/1190e6ca-0f37-45a6-a01d-662b189a88b9-kube-api-access-hxsrh\") pod \"payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52\" (UID: \"1190e6ca-0f37-45a6-a01d-662b189a88b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"
Oct 30 01:05:53.239488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.239452    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1190e6ca-0f37-45a6-a01d-662b189a88b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52\" (UID: \"1190e6ca-0f37-45a6-a01d-662b189a88b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"
Oct 30 01:05:53.240648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.240624    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxsrh\" (UniqueName: \"kubernetes.io/projected/1190e6ca-0f37-45a6-a01d-662b189a88b9-kube-api-access-hxsrh\") pod \"payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52\" (UID: \"1190e6ca-0f37-45a6-a01d-662b189a88b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"
Oct 30 01:05:53.240898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.240513    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1190e6ca-0f37-45a6-a01d-662b189a88b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52\" (UID: \"1190e6ca-0f37-45a6-a01d-662b189a88b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"
Oct 30 01:05:53.261786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.261754    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxsrh\" (UniqueName: \"kubernetes.io/projected/1190e6ca-0f37-45a6-a01d-662b189a88b9-kube-api-access-hxsrh\") pod \"payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52\" (UID: \"1190e6ca-0f37-45a6-a01d-662b189a88b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"
Oct 30 01:05:53.399813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.399710    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"
Oct 30 01:05:53.575418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.575380    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf" event={"ID":"859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09","Type":"ContainerDied","Data":"f55be8998a84e233cb2862741b012b4706e0df436a64f302f16cef19bffc51f2"}
Oct 30 01:05:53.575617 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.575431    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f55be8998a84e233cb2862741b012b4706e0df436a64f302f16cef19bffc51f2"
Oct 30 01:05:53.575617 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.575484    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"
Oct 30 01:05:53.715200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:53.715147    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"]
Oct 30 01:05:53.717274 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:05:53.717249    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1190e6ca_0f37_45a6_a01d_662b189a88b9.slice/crio-f223f3495123b470da9165fa0042f05a1be4f2810e28ac2ff7a8b23db07bb682 WatchSource:0}: Error finding container f223f3495123b470da9165fa0042f05a1be4f2810e28ac2ff7a8b23db07bb682: Status 404 returned error can't find the container with id f223f3495123b470da9165fa0042f05a1be4f2810e28ac2ff7a8b23db07bb682
Oct 30 01:05:54.586113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:54.586030    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52" event={"ID":"1190e6ca-0f37-45a6-a01d-662b189a88b9","Type":"ContainerStarted","Data":"f223f3495123b470da9165fa0042f05a1be4f2810e28ac2ff7a8b23db07bb682"}
Oct 30 01:05:55.595664 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:55.595628    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52" event={"ID":"1190e6ca-0f37-45a6-a01d-662b189a88b9","Type":"ContainerStarted","Data":"7adbaeeba36370027531147db7fb97642e26cbb637019e55458747b9532d45f5"}
Oct 30 01:05:55.596071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:55.595813    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"
Oct 30 01:05:55.633847 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:55.633523    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52" podStartSLOduration=1.398452811 podStartE2EDuration="2.633503723s" totalImagesPullingTime="1.235050912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:55.631468998 +0000 UTC m=+1464.381124306" watchObservedRunningTime="2026-10-30 01:05:55.633503723 +0000 UTC m=+1464.383159048"
Oct 30 01:05:56.538988 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:56.538955    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"]
Oct 30 01:05:56.543057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:56.543036    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"
Oct 30 01:05:56.558078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:56.558048    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ad449e\""
Oct 30 01:05:56.558241 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:56.558084    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ad449e-dockercfg-vz9jn\""
Oct 30 01:05:56.571568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:56.571537    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"]
Oct 30 01:05:56.727480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:56.722140    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9hc\" (UniqueName: \"kubernetes.io/projected/99408a11-d68b-4272-94a0-475ef7c72016-kube-api-access-4h9hc\") pod \"payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp\" (UID: \"99408a11-d68b-4272-94a0-475ef7c72016\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"
Oct 30 01:05:56.727480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:56.722273    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99408a11-d68b-4272-94a0-475ef7c72016-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp\" (UID: \"99408a11-d68b-4272-94a0-475ef7c72016\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"
Oct 30 01:05:56.727480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:56.723735    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"]
Oct 30 01:05:56.734800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:56.734767    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"
Oct 30 01:05:56.737950 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:56.737918    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"]
Oct 30 01:05:56.823977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:56.823874    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9hc\" (UniqueName: \"kubernetes.io/projected/99408a11-d68b-4272-94a0-475ef7c72016-kube-api-access-4h9hc\") pod \"payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp\" (UID: \"99408a11-d68b-4272-94a0-475ef7c72016\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"
Oct 30 01:05:56.823977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:56.823942    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99408a11-d68b-4272-94a0-475ef7c72016-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp\" (UID: \"99408a11-d68b-4272-94a0-475ef7c72016\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"
Oct 30 01:05:56.824735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:56.824716    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99408a11-d68b-4272-94a0-475ef7c72016-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp\" (UID: \"99408a11-d68b-4272-94a0-475ef7c72016\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"
Oct 30 01:05:56.852068 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:56.852032    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9hc\" (UniqueName: \"kubernetes.io/projected/99408a11-d68b-4272-94a0-475ef7c72016-kube-api-access-4h9hc\") pod \"payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp\" (UID: \"99408a11-d68b-4272-94a0-475ef7c72016\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"
Oct 30 01:05:56.860717 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:56.860432    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"
Oct 30 01:05:56.925309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:56.925191    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a604e1-2ecc-47c9-8e0c-1babdcaa5113-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9\" (UID: \"26a604e1-2ecc-47c9-8e0c-1babdcaa5113\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"
Oct 30 01:05:56.925520 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:56.925429    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhwk\" (UniqueName: \"kubernetes.io/projected/26a604e1-2ecc-47c9-8e0c-1babdcaa5113-kube-api-access-pjhwk\") pod \"payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9\" (UID: \"26a604e1-2ecc-47c9-8e0c-1babdcaa5113\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"
Oct 30 01:05:57.026909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:57.026873    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhwk\" (UniqueName: \"kubernetes.io/projected/26a604e1-2ecc-47c9-8e0c-1babdcaa5113-kube-api-access-pjhwk\") pod \"payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9\" (UID: \"26a604e1-2ecc-47c9-8e0c-1babdcaa5113\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"
Oct 30 01:05:57.027084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:57.026976    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a604e1-2ecc-47c9-8e0c-1babdcaa5113-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9\" (UID: \"26a604e1-2ecc-47c9-8e0c-1babdcaa5113\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"
Oct 30 01:05:57.027778 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:57.027755    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a604e1-2ecc-47c9-8e0c-1babdcaa5113-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9\" (UID: \"26a604e1-2ecc-47c9-8e0c-1babdcaa5113\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"
Oct 30 01:05:57.057824 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:57.057791    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhwk\" (UniqueName: \"kubernetes.io/projected/26a604e1-2ecc-47c9-8e0c-1babdcaa5113-kube-api-access-pjhwk\") pod \"payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9\" (UID: \"26a604e1-2ecc-47c9-8e0c-1babdcaa5113\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"
Oct 30 01:05:57.183580 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:05:57.183542    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99408a11_d68b_4272_94a0_475ef7c72016.slice/crio-6c88761c79a2d3157ad283896ad50faf2331066b280639b63185ab9f16e688fe WatchSource:0}: Error finding container 6c88761c79a2d3157ad283896ad50faf2331066b280639b63185ab9f16e688fe: Status 404 returned error can't find the container with id 6c88761c79a2d3157ad283896ad50faf2331066b280639b63185ab9f16e688fe
Oct 30 01:05:57.183742 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:57.183715    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"]
Oct 30 01:05:57.353275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:57.353241    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"
Oct 30 01:05:57.611746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:57.611702    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp" event={"ID":"99408a11-d68b-4272-94a0-475ef7c72016","Type":"ContainerStarted","Data":"006e5415b9ba9e00115db44021e2abaf7b1804834393f5ca7ec87b000eb83b1f"}
Oct 30 01:05:57.611916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:57.611763    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp" event={"ID":"99408a11-d68b-4272-94a0-475ef7c72016","Type":"ContainerStarted","Data":"6c88761c79a2d3157ad283896ad50faf2331066b280639b63185ab9f16e688fe"}
Oct 30 01:05:57.611980 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:57.611956    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"
Oct 30 01:05:57.651168 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:57.651092    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp" podStartSLOduration=1.6510746570000001 podStartE2EDuration="1.651074657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:57.648241361 +0000 UTC m=+1466.397896669" watchObservedRunningTime="2026-10-30 01:05:57.651074657 +0000 UTC m=+1466.400729966"
Oct 30 01:05:57.661049 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:05:57.661022    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a604e1_2ecc_47c9_8e0c_1babdcaa5113.slice/crio-0c3bda6e8de1efd2bd34f6424ff74c00742f2143858b328b097357ed6052771a WatchSource:0}: Error finding container 0c3bda6e8de1efd2bd34f6424ff74c00742f2143858b328b097357ed6052771a: Status 404 returned error can't find the container with id 0c3bda6e8de1efd2bd34f6424ff74c00742f2143858b328b097357ed6052771a
Oct 30 01:05:57.662551 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:57.662528    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"]
Oct 30 01:05:58.028783 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.028750    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"]
Oct 30 01:05:58.031757 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.031729    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d27e976-cvbz8"]
Oct 30 01:05:58.276993 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.276961    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"]
Oct 30 01:05:58.280987 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.280931    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"
Oct 30 01:05:58.294567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.294547    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a8ab86\""
Oct 30 01:05:58.296817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.296790    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a8ab86-dockercfg-f7f95\""
Oct 30 01:05:58.304480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.304453    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"]
Oct 30 01:05:58.334272 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.334242    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"]
Oct 30 01:05:58.342126 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.342104    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"
Oct 30 01:05:58.366011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.365986    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"]
Oct 30 01:05:58.445893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.445856    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34e74245-057c-4a90-bcc7-818149e6f991-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f\" (UID: \"34e74245-057c-4a90-bcc7-818149e6f991\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"
Oct 30 01:05:58.446278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.446146    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743ad5a9-7115-4060-90ad-1ae7e8c823d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj\" (UID: \"743ad5a9-7115-4060-90ad-1ae7e8c823d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"
Oct 30 01:05:58.446467 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.446446    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlnhk\" (UniqueName: \"kubernetes.io/projected/34e74245-057c-4a90-bcc7-818149e6f991-kube-api-access-wlnhk\") pod \"payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f\" (UID: \"34e74245-057c-4a90-bcc7-818149e6f991\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"
Oct 30 01:05:58.446647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.446626    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84sq8\" (UniqueName: \"kubernetes.io/projected/743ad5a9-7115-4060-90ad-1ae7e8c823d5-kube-api-access-84sq8\") pod \"payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj\" (UID: \"743ad5a9-7115-4060-90ad-1ae7e8c823d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"
Oct 30 01:05:58.547562 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.547472    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84sq8\" (UniqueName: \"kubernetes.io/projected/743ad5a9-7115-4060-90ad-1ae7e8c823d5-kube-api-access-84sq8\") pod \"payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj\" (UID: \"743ad5a9-7115-4060-90ad-1ae7e8c823d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"
Oct 30 01:05:58.547738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.547669    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34e74245-057c-4a90-bcc7-818149e6f991-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f\" (UID: \"34e74245-057c-4a90-bcc7-818149e6f991\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"
Oct 30 01:05:58.547738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.547720    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743ad5a9-7115-4060-90ad-1ae7e8c823d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj\" (UID: \"743ad5a9-7115-4060-90ad-1ae7e8c823d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"
Oct 30 01:05:58.547853 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.547759    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnhk\" (UniqueName: \"kubernetes.io/projected/34e74245-057c-4a90-bcc7-818149e6f991-kube-api-access-wlnhk\") pod \"payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f\" (UID: \"34e74245-057c-4a90-bcc7-818149e6f991\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"
Oct 30 01:05:58.548512 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.548490    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743ad5a9-7115-4060-90ad-1ae7e8c823d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj\" (UID: \"743ad5a9-7115-4060-90ad-1ae7e8c823d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"
Oct 30 01:05:58.548512 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.548505    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34e74245-057c-4a90-bcc7-818149e6f991-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f\" (UID: \"34e74245-057c-4a90-bcc7-818149e6f991\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"
Oct 30 01:05:58.579735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.571905    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84sq8\" (UniqueName: \"kubernetes.io/projected/743ad5a9-7115-4060-90ad-1ae7e8c823d5-kube-api-access-84sq8\") pod \"payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj\" (UID: \"743ad5a9-7115-4060-90ad-1ae7e8c823d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"
Oct 30 01:05:58.591897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.591614    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlnhk\" (UniqueName: \"kubernetes.io/projected/34e74245-057c-4a90-bcc7-818149e6f991-kube-api-access-wlnhk\") pod \"payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f\" (UID: \"34e74245-057c-4a90-bcc7-818149e6f991\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"
Oct 30 01:05:58.595293 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.595266    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"
Oct 30 01:05:58.624897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.624863    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9" event={"ID":"26a604e1-2ecc-47c9-8e0c-1babdcaa5113","Type":"ContainerStarted","Data":"34d702fe899eade1397c58e48b72819846c58d86eb18fdd79e0a298016411e6b"}
Oct 30 01:05:58.625014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.624927    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"
Oct 30 01:05:58.625014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.624948    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9" event={"ID":"26a604e1-2ecc-47c9-8e0c-1babdcaa5113","Type":"ContainerStarted","Data":"0c3bda6e8de1efd2bd34f6424ff74c00742f2143858b328b097357ed6052771a"}
Oct 30 01:05:58.661961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.660890    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"
Oct 30 01:05:58.667019 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.663783    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9" podStartSLOduration=2.663765137 podStartE2EDuration="2.663765137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:58.660806495 +0000 UTC m=+1467.410461805" watchObservedRunningTime="2026-10-30 01:05:58.663765137 +0000 UTC m=+1467.413420446"
Oct 30 01:05:58.905587 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.905556    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"]
Oct 30 01:05:58.906694 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:05:58.906672    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e74245_057c_4a90_bcc7_818149e6f991.slice/crio-ab53b88ac443603b6dccd3b027eace2a520beb7e522b64025efbb7b2dceb249f WatchSource:0}: Error finding container ab53b88ac443603b6dccd3b027eace2a520beb7e522b64025efbb7b2dceb249f: Status 404 returned error can't find the container with id ab53b88ac443603b6dccd3b027eace2a520beb7e522b64025efbb7b2dceb249f
Oct 30 01:05:58.992125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:58.992096    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"]
Oct 30 01:05:58.996098 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:05:58.996073    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743ad5a9_7115_4060_90ad_1ae7e8c823d5.slice/crio-7968cc06b97f584b2940c61e28c8340e3f6dd5d7548f4b8cbb870a8ac249679d WatchSource:0}: Error finding container 7968cc06b97f584b2940c61e28c8340e3f6dd5d7548f4b8cbb870a8ac249679d: Status 404 returned error can't find the container with id 7968cc06b97f584b2940c61e28c8340e3f6dd5d7548f4b8cbb870a8ac249679d
Oct 30 01:05:59.423748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:59.423718    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff836440-d844-4126-a495-1f8cf256dccc" path="/var/lib/kubelet/pods/ff836440-d844-4126-a495-1f8cf256dccc/volumes"
Oct 30 01:05:59.633669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:59.633637    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f" event={"ID":"34e74245-057c-4a90-bcc7-818149e6f991","Type":"ContainerStarted","Data":"37c69370e44e16212f1d4124327b885f8b9d0156f175478c08d2f553af169620"}
Oct 30 01:05:59.633836 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:59.633684    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f" event={"ID":"34e74245-057c-4a90-bcc7-818149e6f991","Type":"ContainerStarted","Data":"ab53b88ac443603b6dccd3b027eace2a520beb7e522b64025efbb7b2dceb249f"}
Oct 30 01:05:59.633836 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:59.633725    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"
Oct 30 01:05:59.635396 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:59.635366    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj" event={"ID":"743ad5a9-7115-4060-90ad-1ae7e8c823d5","Type":"ContainerStarted","Data":"5b3598289df86b31a9d73c56ff69c45607f80fdc96865e26d66be2dc0632d5f1"}
Oct 30 01:05:59.635507 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:59.635416    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj" event={"ID":"743ad5a9-7115-4060-90ad-1ae7e8c823d5","Type":"ContainerStarted","Data":"7968cc06b97f584b2940c61e28c8340e3f6dd5d7548f4b8cbb870a8ac249679d"}
Oct 30 01:05:59.635507 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:59.635446    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"
Oct 30 01:05:59.671610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:59.671566    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f" podStartSLOduration=1.671553684 podStartE2EDuration="1.671553684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:59.669254545 +0000 UTC m=+1468.418909853" watchObservedRunningTime="2026-10-30 01:05:59.671553684 +0000 UTC m=+1468.421208994"
Oct 30 01:05:59.705890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:05:59.705794    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj" podStartSLOduration=1.705781473 podStartE2EDuration="1.705781473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:59.704181401 +0000 UTC m=+1468.453836700" watchObservedRunningTime="2026-10-30 01:05:59.705781473 +0000 UTC m=+1468.455436782"
Oct 30 01:06:02.641587 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:02.641549    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:06:06.605607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:06.605574    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"
Oct 30 01:06:07.554768 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:06:07.554732    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:06:08.627613 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:08.627588    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"
Oct 30 01:06:09.028058 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:09.028029    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"]
Oct 30 01:06:09.031199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:09.031177    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6617180-d8xzj"]
Oct 30 01:06:09.422454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:09.422364    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680" path="/var/lib/kubelet/pods/2cf3c4f6-5e08-4d85-b3f4-91f2e2d40680/volumes"
Oct 30 01:06:09.639133 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:09.639101    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"
Oct 30 01:06:10.646099 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:10.646073    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"
Oct 30 01:06:10.646575 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:10.646448    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"
Oct 30 01:06:22.035693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:22.035620    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"]
Oct 30 01:06:22.045055 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:22.045029    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af37dd22-gblwq"]
Oct 30 01:06:23.423609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:23.423579    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6974c649-c853-4f80-a251-15cdb72b6de4" path="/var/lib/kubelet/pods/6974c649-c853-4f80-a251-15cdb72b6de4/volumes"
Oct 30 01:06:26.660208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:26.660172    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"]
Oct 30 01:06:26.669419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:26.669393    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76f654-kwrrf"]
Oct 30 01:06:27.422792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:27.422761    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09" path="/var/lib/kubelet/pods/859eeb23-8ca1-4f5c-bda7-3c6ee3b72d09/volumes"
Oct 30 01:06:32.176052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:32.176013    4425 scope.go:122] "RemoveContainer" containerID="87b6d1cb4d299319c2929ce02154c5611b156c411be4d9ccb52cc2673990cbdf"
Oct 30 01:06:34.165804 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:34.165768    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"]
Oct 30 01:06:34.166618 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:34.166565    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc" podUID="075fdcb4-b628-4603-ab90-e7575e3096df" containerName="istio-proxy" containerID="cri-o://5d611cfd36b883c35667a2991f727f69da7954141ba1863b14d31a2ba84b3fdd" gracePeriod=30
Oct 30 01:06:35.686982 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:35.686938    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"]
Oct 30 01:06:35.698459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:35.698425    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"]
Oct 30 01:06:36.555572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:36.555536    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"]
Oct 30 01:06:36.584958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:36.584920    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"]
Oct 30 01:06:37.554582 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:06:37.554536    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:06:37.850978 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:37.850910    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"]
Oct 30 01:06:37.855169 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:37.855138    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"
Oct 30 01:06:37.869483 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:37.869464    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-838fd5-dockercfg-4qjbv\""
Oct 30 01:06:37.869483 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:37.869464    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-838fd5\""
Oct 30 01:06:37.875094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:37.875067    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"]
Oct 30 01:06:37.963136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:37.963101    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftdp9\" (UniqueName: \"kubernetes.io/projected/dd5fae0e-4675-4a1d-b187-279fcafe3984-kube-api-access-ftdp9\") pod \"payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9\" (UID: \"dd5fae0e-4675-4a1d-b187-279fcafe3984\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"
Oct 30 01:06:37.964679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:37.964653    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5fae0e-4675-4a1d-b187-279fcafe3984-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9\" (UID: \"dd5fae0e-4675-4a1d-b187-279fcafe3984\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"
Oct 30 01:06:38.066387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:38.066353    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftdp9\" (UniqueName: \"kubernetes.io/projected/dd5fae0e-4675-4a1d-b187-279fcafe3984-kube-api-access-ftdp9\") pod \"payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9\" (UID: \"dd5fae0e-4675-4a1d-b187-279fcafe3984\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"
Oct 30 01:06:38.066615 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:38.066431    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5fae0e-4675-4a1d-b187-279fcafe3984-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9\" (UID: \"dd5fae0e-4675-4a1d-b187-279fcafe3984\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"
Oct 30 01:06:38.067117 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:38.067092    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5fae0e-4675-4a1d-b187-279fcafe3984-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9\" (UID: \"dd5fae0e-4675-4a1d-b187-279fcafe3984\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"
Oct 30 01:06:38.088169 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:38.088126    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftdp9\" (UniqueName: \"kubernetes.io/projected/dd5fae0e-4675-4a1d-b187-279fcafe3984-kube-api-access-ftdp9\") pod \"payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9\" (UID: \"dd5fae0e-4675-4a1d-b187-279fcafe3984\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"
Oct 30 01:06:38.167430 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:38.167343    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"
Oct 30 01:06:38.484426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:38.484400    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"]
Oct 30 01:06:38.486347 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:06:38.486320    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5fae0e_4675_4a1d_b187_279fcafe3984.slice/crio-072d625a897c0ce3a1493e2b4e8d65e48cb238f4b005ec86db669550498738a1 WatchSource:0}: Error finding container 072d625a897c0ce3a1493e2b4e8d65e48cb238f4b005ec86db669550498738a1: Status 404 returned error can't find the container with id 072d625a897c0ce3a1493e2b4e8d65e48cb238f4b005ec86db669550498738a1
Oct 30 01:06:38.919171 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:38.919128    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9" event={"ID":"dd5fae0e-4675-4a1d-b187-279fcafe3984","Type":"ContainerStarted","Data":"9dd3abe7a6a6d8ee1849a6ae3bf4c13aa48513665ec581074e65f8fb1c3da5fe"}
Oct 30 01:06:38.919585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:38.919194    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9" event={"ID":"dd5fae0e-4675-4a1d-b187-279fcafe3984","Type":"ContainerStarted","Data":"072d625a897c0ce3a1493e2b4e8d65e48cb238f4b005ec86db669550498738a1"}
Oct 30 01:06:38.919585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:38.919221    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"
Oct 30 01:06:38.955996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:38.955948    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9" podStartSLOduration=1.955935967 podStartE2EDuration="1.955935967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:38.955141762 +0000 UTC m=+1507.704797071" watchObservedRunningTime="2026-10-30 01:06:38.955935967 +0000 UTC m=+1507.705591277"
Oct 30 01:06:39.464105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.464081    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:06:39.591004 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.590972    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075fdcb4-b628-4603-ab90-e7575e3096df-istio-token\" (UniqueName: \"kubernetes.io/projected/075fdcb4-b628-4603-ab90-e7575e3096df-istio-token\") pod \"075fdcb4-b628-4603-ab90-e7575e3096df\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") "
Oct 30 01:06:39.591199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.591080    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075fdcb4-b628-4603-ab90-e7575e3096df-kube-api-access-bhn6h\" (UniqueName: \"kubernetes.io/projected/075fdcb4-b628-4603-ab90-e7575e3096df-kube-api-access-bhn6h\") pod \"075fdcb4-b628-4603-ab90-e7575e3096df\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") "
Oct 30 01:06:39.591199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.591110    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-workload-certs\") pod \"075fdcb4-b628-4603-ab90-e7575e3096df\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") "
Oct 30 01:06:39.591347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.591216    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-workload-socket\") pod \"075fdcb4-b628-4603-ab90-e7575e3096df\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") "
Oct 30 01:06:39.591347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.591293    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-istio-data\") pod \"075fdcb4-b628-4603-ab90-e7575e3096df\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") "
Oct 30 01:06:39.591347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.591319    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-istio-envoy\") pod \"075fdcb4-b628-4603-ab90-e7575e3096df\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") "
Oct 30 01:06:39.591509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.591347    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-credential-socket\") pod \"075fdcb4-b628-4603-ab90-e7575e3096df\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") "
Oct 30 01:06:39.591509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.591381    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/075fdcb4-b628-4603-ab90-e7575e3096df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/075fdcb4-b628-4603-ab90-e7575e3096df-istiod-ca-cert\") pod \"075fdcb4-b628-4603-ab90-e7575e3096df\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") "
Oct 30 01:06:39.591615 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.591585    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/075fdcb4-b628-4603-ab90-e7575e3096df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/075fdcb4-b628-4603-ab90-e7575e3096df-istio-podinfo\") pod \"075fdcb4-b628-4603-ab90-e7575e3096df\" (UID: \"075fdcb4-b628-4603-ab90-e7575e3096df\") "
Oct 30 01:06:39.591615 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.591588    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-istio-data" pod "075fdcb4-b628-4603-ab90-e7575e3096df" (UID: "075fdcb4-b628-4603-ab90-e7575e3096df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:39.591947 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.591923    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-workload-certs" pod "075fdcb4-b628-4603-ab90-e7575e3096df" (UID: "075fdcb4-b628-4603-ab90-e7575e3096df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:39.592272 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.592250    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.592355 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.592278    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.592560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.592535    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-credential-socket" pod "075fdcb4-b628-4603-ab90-e7575e3096df" (UID: "075fdcb4-b628-4603-ab90-e7575e3096df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:39.592665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.592566    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-workload-socket" pod "075fdcb4-b628-4603-ab90-e7575e3096df" (UID: "075fdcb4-b628-4603-ab90-e7575e3096df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:39.592802 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.592778    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075fdcb4-b628-4603-ab90-e7575e3096df-istiod-ca-cert" pod "075fdcb4-b628-4603-ab90-e7575e3096df" (UID: "075fdcb4-b628-4603-ab90-e7575e3096df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:39.594541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.594514    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075fdcb4-b628-4603-ab90-e7575e3096df-kube-api-access-bhn6h" pod "075fdcb4-b628-4603-ab90-e7575e3096df" (UID: "075fdcb4-b628-4603-ab90-e7575e3096df"). InnerVolumeSpecName "kube-api-access-bhn6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:39.594779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.594748    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075fdcb4-b628-4603-ab90-e7575e3096df-istio-token" pod "075fdcb4-b628-4603-ab90-e7575e3096df" (UID: "075fdcb4-b628-4603-ab90-e7575e3096df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:39.594932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.594908    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-istio-envoy" pod "075fdcb4-b628-4603-ab90-e7575e3096df" (UID: "075fdcb4-b628-4603-ab90-e7575e3096df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:39.595039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.595021    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/075fdcb4-b628-4603-ab90-e7575e3096df-istio-podinfo" pod "075fdcb4-b628-4603-ab90-e7575e3096df" (UID: "075fdcb4-b628-4603-ab90-e7575e3096df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:06:39.693456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.693382    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhn6h\" (UniqueName: \"kubernetes.io/projected/075fdcb4-b628-4603-ab90-e7575e3096df-kube-api-access-bhn6h\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.693456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.693410    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.693456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.693421    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.693456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.693429    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/075fdcb4-b628-4603-ab90-e7575e3096df-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.693456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.693438    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/075fdcb4-b628-4603-ab90-e7575e3096df-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.693456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.693446    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/075fdcb4-b628-4603-ab90-e7575e3096df-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.693456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.693454    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/075fdcb4-b628-4603-ab90-e7575e3096df-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:06:39.926767 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.926737    4425 generic.go:356] "Generic (PLEG): container finished" podID="075fdcb4-b628-4603-ab90-e7575e3096df" containerID="5d611cfd36b883c35667a2991f727f69da7954141ba1863b14d31a2ba84b3fdd" exitCode=0
Oct 30 01:06:39.927145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.926784    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"
Oct 30 01:06:39.927145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.926824    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc" event={"ID":"075fdcb4-b628-4603-ab90-e7575e3096df","Type":"ContainerDied","Data":"5d611cfd36b883c35667a2991f727f69da7954141ba1863b14d31a2ba84b3fdd"}
Oct 30 01:06:39.927145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.926880    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc" event={"ID":"075fdcb4-b628-4603-ab90-e7575e3096df","Type":"ContainerDied","Data":"19385d999d1efc624fb410a6dafcd9973f7b83ac71d4f0cf679bb203c4dc74e7"}
Oct 30 01:06:39.927145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.926903    4425 scope.go:122] "RemoveContainer" containerID="5d611cfd36b883c35667a2991f727f69da7954141ba1863b14d31a2ba84b3fdd"
Oct 30 01:06:39.938476 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.938456    4425 scope.go:122] "RemoveContainer" containerID="5d611cfd36b883c35667a2991f727f69da7954141ba1863b14d31a2ba84b3fdd"
Oct 30 01:06:39.938757 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:06:39.938740    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d611cfd36b883c35667a2991f727f69da7954141ba1863b14d31a2ba84b3fdd\": container with ID starting with 5d611cfd36b883c35667a2991f727f69da7954141ba1863b14d31a2ba84b3fdd not found: ID does not exist" containerID="5d611cfd36b883c35667a2991f727f69da7954141ba1863b14d31a2ba84b3fdd"
Oct 30 01:06:39.938794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.938765    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d611cfd36b883c35667a2991f727f69da7954141ba1863b14d31a2ba84b3fdd"} err="failed to get container status \"5d611cfd36b883c35667a2991f727f69da7954141ba1863b14d31a2ba84b3fdd\": rpc error: code = NotFound desc = could not find container \"5d611cfd36b883c35667a2991f727f69da7954141ba1863b14d31a2ba84b3fdd\": container with ID starting with 5d611cfd36b883c35667a2991f727f69da7954141ba1863b14d31a2ba84b3fdd not found: ID does not exist"
Oct 30 01:06:39.972636 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.972582    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"]
Oct 30 01:06:39.981870 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:39.981851    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-76f654-openshift-default-cf5f45b55-8glrc"]
Oct 30 01:06:40.687824 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:40.687782    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9" podUID="26a604e1-2ecc-47c9-8e0c-1babdcaa5113" containerName="payload-processing" containerID="cri-o://34d702fe899eade1397c58e48b72819846c58d86eb18fdd79e0a298016411e6b" gracePeriod=25
Oct 30 01:06:40.699222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:40.699190    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp" podUID="99408a11-d68b-4272-94a0-475ef7c72016" containerName="payload-pre-processing" containerID="cri-o://006e5415b9ba9e00115db44021e2abaf7b1804834393f5ca7ec87b000eb83b1f" gracePeriod=25
Oct 30 01:06:40.940560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:40.940392    4425 generic.go:356] "Generic (PLEG): container finished" podID="99408a11-d68b-4272-94a0-475ef7c72016" containerID="006e5415b9ba9e00115db44021e2abaf7b1804834393f5ca7ec87b000eb83b1f" exitCode=0
Oct 30 01:06:40.940911 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:40.940650    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp" event={"ID":"99408a11-d68b-4272-94a0-475ef7c72016","Type":"ContainerDied","Data":"006e5415b9ba9e00115db44021e2abaf7b1804834393f5ca7ec87b000eb83b1f"}
Oct 30 01:06:40.944291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:40.944263    4425 generic.go:356] "Generic (PLEG): container finished" podID="26a604e1-2ecc-47c9-8e0c-1babdcaa5113" containerID="34d702fe899eade1397c58e48b72819846c58d86eb18fdd79e0a298016411e6b" exitCode=0
Oct 30 01:06:40.944433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:40.944354    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9" event={"ID":"26a604e1-2ecc-47c9-8e0c-1babdcaa5113","Type":"ContainerDied","Data":"34d702fe899eade1397c58e48b72819846c58d86eb18fdd79e0a298016411e6b"}
Oct 30 01:06:40.965577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:40.965557    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"
Oct 30 01:06:40.991560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:40.991541    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"
Oct 30 01:06:41.111457 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.111429    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99408a11-d68b-4272-94a0-475ef7c72016-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99408a11-d68b-4272-94a0-475ef7c72016-plugins-config-volume\") pod \"99408a11-d68b-4272-94a0-475ef7c72016\" (UID: \"99408a11-d68b-4272-94a0-475ef7c72016\") "
Oct 30 01:06:41.111625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.111510    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a604e1-2ecc-47c9-8e0c-1babdcaa5113-kube-api-access-pjhwk\" (UniqueName: \"kubernetes.io/projected/26a604e1-2ecc-47c9-8e0c-1babdcaa5113-kube-api-access-pjhwk\") pod \"26a604e1-2ecc-47c9-8e0c-1babdcaa5113\" (UID: \"26a604e1-2ecc-47c9-8e0c-1babdcaa5113\") "
Oct 30 01:06:41.111625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.111601    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a604e1-2ecc-47c9-8e0c-1babdcaa5113-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a604e1-2ecc-47c9-8e0c-1babdcaa5113-plugins-config-volume\") pod \"26a604e1-2ecc-47c9-8e0c-1babdcaa5113\" (UID: \"26a604e1-2ecc-47c9-8e0c-1babdcaa5113\") "
Oct 30 01:06:41.111718 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.111644    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99408a11-d68b-4272-94a0-475ef7c72016-kube-api-access-4h9hc\" (UniqueName: \"kubernetes.io/projected/99408a11-d68b-4272-94a0-475ef7c72016-kube-api-access-4h9hc\") pod \"99408a11-d68b-4272-94a0-475ef7c72016\" (UID: \"99408a11-d68b-4272-94a0-475ef7c72016\") "
Oct 30 01:06:41.111964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.111934    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99408a11-d68b-4272-94a0-475ef7c72016-plugins-config-volume" pod "99408a11-d68b-4272-94a0-475ef7c72016" (UID: "99408a11-d68b-4272-94a0-475ef7c72016"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:41.112038 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.111975    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a604e1-2ecc-47c9-8e0c-1babdcaa5113-plugins-config-volume" pod "26a604e1-2ecc-47c9-8e0c-1babdcaa5113" (UID: "26a604e1-2ecc-47c9-8e0c-1babdcaa5113"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:41.112038 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.112009    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99408a11-d68b-4272-94a0-475ef7c72016-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:06:41.113628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.113604    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99408a11-d68b-4272-94a0-475ef7c72016-kube-api-access-4h9hc" pod "99408a11-d68b-4272-94a0-475ef7c72016" (UID: "99408a11-d68b-4272-94a0-475ef7c72016"). InnerVolumeSpecName "kube-api-access-4h9hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:41.113718 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.113665    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a604e1-2ecc-47c9-8e0c-1babdcaa5113-kube-api-access-pjhwk" pod "26a604e1-2ecc-47c9-8e0c-1babdcaa5113" (UID: "26a604e1-2ecc-47c9-8e0c-1babdcaa5113"). InnerVolumeSpecName "kube-api-access-pjhwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:41.213421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.213354    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a604e1-2ecc-47c9-8e0c-1babdcaa5113-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:06:41.213421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.213379    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h9hc\" (UniqueName: \"kubernetes.io/projected/99408a11-d68b-4272-94a0-475ef7c72016-kube-api-access-4h9hc\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:06:41.213421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.213387    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjhwk\" (UniqueName: \"kubernetes.io/projected/26a604e1-2ecc-47c9-8e0c-1babdcaa5113-kube-api-access-pjhwk\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:06:41.422403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.422374    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="075fdcb4-b628-4603-ab90-e7575e3096df" path="/var/lib/kubelet/pods/075fdcb4-b628-4603-ab90-e7575e3096df/volumes"
Oct 30 01:06:41.556566 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.556532    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f" podUID="34e74245-057c-4a90-bcc7-818149e6f991" containerName="payload-processing" containerID="cri-o://37c69370e44e16212f1d4124327b885f8b9d0156f175478c08d2f553af169620" gracePeriod=25
Oct 30 01:06:41.586225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.586188    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj" podUID="743ad5a9-7115-4060-90ad-1ae7e8c823d5" containerName="payload-pre-processing" containerID="cri-o://5b3598289df86b31a9d73c56ff69c45607f80fdc96865e26d66be2dc0632d5f1" gracePeriod=25
Oct 30 01:06:41.851490 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.851459    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"
Oct 30 01:06:41.908659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.908632    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"
Oct 30 01:06:41.954139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.954015    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"
Oct 30 01:06:41.954139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.954020    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9" event={"ID":"26a604e1-2ecc-47c9-8e0c-1babdcaa5113","Type":"ContainerDied","Data":"0c3bda6e8de1efd2bd34f6424ff74c00742f2143858b328b097357ed6052771a"}
Oct 30 01:06:41.954858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.954207    4425 scope.go:122] "RemoveContainer" containerID="34d702fe899eade1397c58e48b72819846c58d86eb18fdd79e0a298016411e6b"
Oct 30 01:06:41.957875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.957851    4425 generic.go:356] "Generic (PLEG): container finished" podID="34e74245-057c-4a90-bcc7-818149e6f991" containerID="37c69370e44e16212f1d4124327b885f8b9d0156f175478c08d2f553af169620" exitCode=0
Oct 30 01:06:41.957996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.957892    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"
Oct 30 01:06:41.957996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.957896    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f" event={"ID":"34e74245-057c-4a90-bcc7-818149e6f991","Type":"ContainerDied","Data":"37c69370e44e16212f1d4124327b885f8b9d0156f175478c08d2f553af169620"}
Oct 30 01:06:41.957996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.957949    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f" event={"ID":"34e74245-057c-4a90-bcc7-818149e6f991","Type":"ContainerDied","Data":"ab53b88ac443603b6dccd3b027eace2a520beb7e522b64025efbb7b2dceb249f"}
Oct 30 01:06:41.959547 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.959524    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp" event={"ID":"99408a11-d68b-4272-94a0-475ef7c72016","Type":"ContainerDied","Data":"6c88761c79a2d3157ad283896ad50faf2331066b280639b63185ab9f16e688fe"}
Oct 30 01:06:41.959655 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.959536    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"
Oct 30 01:06:41.961067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.961048    4425 generic.go:356] "Generic (PLEG): container finished" podID="743ad5a9-7115-4060-90ad-1ae7e8c823d5" containerID="5b3598289df86b31a9d73c56ff69c45607f80fdc96865e26d66be2dc0632d5f1" exitCode=0
Oct 30 01:06:41.961235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.961093    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"
Oct 30 01:06:41.961346 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.961109    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj" event={"ID":"743ad5a9-7115-4060-90ad-1ae7e8c823d5","Type":"ContainerDied","Data":"5b3598289df86b31a9d73c56ff69c45607f80fdc96865e26d66be2dc0632d5f1"}
Oct 30 01:06:41.961430 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.961372    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj" event={"ID":"743ad5a9-7115-4060-90ad-1ae7e8c823d5","Type":"ContainerDied","Data":"7968cc06b97f584b2940c61e28c8340e3f6dd5d7548f4b8cbb870a8ac249679d"}
Oct 30 01:06:41.967263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.967243    4425 scope.go:122] "RemoveContainer" containerID="37c69370e44e16212f1d4124327b885f8b9d0156f175478c08d2f553af169620"
Oct 30 01:06:41.976991 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.976972    4425 scope.go:122] "RemoveContainer" containerID="37c69370e44e16212f1d4124327b885f8b9d0156f175478c08d2f553af169620"
Oct 30 01:06:41.977331 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:06:41.977303    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37c69370e44e16212f1d4124327b885f8b9d0156f175478c08d2f553af169620\": container with ID starting with 37c69370e44e16212f1d4124327b885f8b9d0156f175478c08d2f553af169620 not found: ID does not exist" containerID="37c69370e44e16212f1d4124327b885f8b9d0156f175478c08d2f553af169620"
Oct 30 01:06:41.977395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.977341    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c69370e44e16212f1d4124327b885f8b9d0156f175478c08d2f553af169620"} err="failed to get container status \"37c69370e44e16212f1d4124327b885f8b9d0156f175478c08d2f553af169620\": rpc error: code = NotFound desc = could not find container \"37c69370e44e16212f1d4124327b885f8b9d0156f175478c08d2f553af169620\": container with ID starting with 37c69370e44e16212f1d4124327b885f8b9d0156f175478c08d2f553af169620 not found: ID does not exist"
Oct 30 01:06:41.977395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.977364    4425 scope.go:122] "RemoveContainer" containerID="006e5415b9ba9e00115db44021e2abaf7b1804834393f5ca7ec87b000eb83b1f"
Oct 30 01:06:41.987268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.987232    4425 scope.go:122] "RemoveContainer" containerID="5b3598289df86b31a9d73c56ff69c45607f80fdc96865e26d66be2dc0632d5f1"
Oct 30 01:06:41.997632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.997620    4425 scope.go:122] "RemoveContainer" containerID="5b3598289df86b31a9d73c56ff69c45607f80fdc96865e26d66be2dc0632d5f1"
Oct 30 01:06:41.997933 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:06:41.997909    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b3598289df86b31a9d73c56ff69c45607f80fdc96865e26d66be2dc0632d5f1\": container with ID starting with 5b3598289df86b31a9d73c56ff69c45607f80fdc96865e26d66be2dc0632d5f1 not found: ID does not exist" containerID="5b3598289df86b31a9d73c56ff69c45607f80fdc96865e26d66be2dc0632d5f1"
Oct 30 01:06:41.998014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:41.997942    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3598289df86b31a9d73c56ff69c45607f80fdc96865e26d66be2dc0632d5f1"} err="failed to get container status \"5b3598289df86b31a9d73c56ff69c45607f80fdc96865e26d66be2dc0632d5f1\": rpc error: code = NotFound desc = could not find container \"5b3598289df86b31a9d73c56ff69c45607f80fdc96865e26d66be2dc0632d5f1\": container with ID starting with 5b3598289df86b31a9d73c56ff69c45607f80fdc96865e26d66be2dc0632d5f1 not found: ID does not exist"
Oct 30 01:06:42.011313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.011291    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"]
Oct 30 01:06:42.017601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.017579    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad449e-7567db9dd4-q5vl9"]
Oct 30 01:06:42.022935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.022914    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34e74245-057c-4a90-bcc7-818149e6f991-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34e74245-057c-4a90-bcc7-818149e6f991-plugins-config-volume\") pod \"34e74245-057c-4a90-bcc7-818149e6f991\" (UID: \"34e74245-057c-4a90-bcc7-818149e6f991\") "
Oct 30 01:06:42.023013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.022952    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/743ad5a9-7115-4060-90ad-1ae7e8c823d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743ad5a9-7115-4060-90ad-1ae7e8c823d5-plugins-config-volume\") pod \"743ad5a9-7115-4060-90ad-1ae7e8c823d5\" (UID: \"743ad5a9-7115-4060-90ad-1ae7e8c823d5\") "
Oct 30 01:06:42.023013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.022992    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34e74245-057c-4a90-bcc7-818149e6f991-kube-api-access-wlnhk\" (UniqueName: \"kubernetes.io/projected/34e74245-057c-4a90-bcc7-818149e6f991-kube-api-access-wlnhk\") pod \"34e74245-057c-4a90-bcc7-818149e6f991\" (UID: \"34e74245-057c-4a90-bcc7-818149e6f991\") "
Oct 30 01:06:42.023204 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.023188    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/743ad5a9-7115-4060-90ad-1ae7e8c823d5-kube-api-access-84sq8\" (UniqueName: \"kubernetes.io/projected/743ad5a9-7115-4060-90ad-1ae7e8c823d5-kube-api-access-84sq8\") pod \"743ad5a9-7115-4060-90ad-1ae7e8c823d5\" (UID: \"743ad5a9-7115-4060-90ad-1ae7e8c823d5\") "
Oct 30 01:06:42.023324 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.023301    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/743ad5a9-7115-4060-90ad-1ae7e8c823d5-plugins-config-volume" pod "743ad5a9-7115-4060-90ad-1ae7e8c823d5" (UID: "743ad5a9-7115-4060-90ad-1ae7e8c823d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:42.023395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.023352    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34e74245-057c-4a90-bcc7-818149e6f991-plugins-config-volume" pod "34e74245-057c-4a90-bcc7-818149e6f991" (UID: "34e74245-057c-4a90-bcc7-818149e6f991"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:42.023568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.023553    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34e74245-057c-4a90-bcc7-818149e6f991-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:06:42.023609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.023571    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743ad5a9-7115-4060-90ad-1ae7e8c823d5-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:06:42.024947 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.024930    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e74245-057c-4a90-bcc7-818149e6f991-kube-api-access-wlnhk" pod "34e74245-057c-4a90-bcc7-818149e6f991" (UID: "34e74245-057c-4a90-bcc7-818149e6f991"). InnerVolumeSpecName "kube-api-access-wlnhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:42.025033 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.025009    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/743ad5a9-7115-4060-90ad-1ae7e8c823d5-kube-api-access-84sq8" pod "743ad5a9-7115-4060-90ad-1ae7e8c823d5" (UID: "743ad5a9-7115-4060-90ad-1ae7e8c823d5"). InnerVolumeSpecName "kube-api-access-84sq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:42.046305 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.046285    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"]
Oct 30 01:06:42.050285 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.050261    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad449e-85444b8bc-ljrzp"]
Oct 30 01:06:42.124943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.124887    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlnhk\" (UniqueName: \"kubernetes.io/projected/34e74245-057c-4a90-bcc7-818149e6f991-kube-api-access-wlnhk\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:06:42.124943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.124910    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84sq8\" (UniqueName: \"kubernetes.io/projected/743ad5a9-7115-4060-90ad-1ae7e8c823d5-kube-api-access-84sq8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:06:42.303120 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.303080    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"]
Oct 30 01:06:42.308348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.308319    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a8ab86-75698cb584-g5z2f"]
Oct 30 01:06:42.339224 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.339195    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"]
Oct 30 01:06:42.345556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:42.345530    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a8ab86-866cb578d7-z2rjj"]
Oct 30 01:06:43.423344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:43.423317    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26a604e1-2ecc-47c9-8e0c-1babdcaa5113" path="/var/lib/kubelet/pods/26a604e1-2ecc-47c9-8e0c-1babdcaa5113/volumes"
Oct 30 01:06:43.423738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:43.423722    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34e74245-057c-4a90-bcc7-818149e6f991" path="/var/lib/kubelet/pods/34e74245-057c-4a90-bcc7-818149e6f991/volumes"
Oct 30 01:06:43.424086 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:43.424072    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="743ad5a9-7115-4060-90ad-1ae7e8c823d5" path="/var/lib/kubelet/pods/743ad5a9-7115-4060-90ad-1ae7e8c823d5/volumes"
Oct 30 01:06:43.424451 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:43.424438    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99408a11-d68b-4272-94a0-475ef7c72016" path="/var/lib/kubelet/pods/99408a11-d68b-4272-94a0-475ef7c72016/volumes"
Oct 30 01:06:47.205404 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:47.205365    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:06:49.930401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:49.930371    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"
Oct 30 01:06:57.382166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:57.382130    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"]
Oct 30 01:06:57.382563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:06:57.382409    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg" podUID="30db65f6-70ad-4a3b-bed9-fb4f74533b3a" containerName="istio-proxy" containerID="cri-o://f746ab01ba64474fcb169e668a363861456a436abd2c5118ea64be738cfa167e" gracePeriod=30
Oct 30 01:07:00.893284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.893248    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"]
Oct 30 01:07:00.893783 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.893755    4425 cpu_manager.go:414] "containerMap: removing container" podUID="99408a11-d68b-4272-94a0-475ef7c72016" containerName="payload-pre-processing"
Oct 30 01:07:00.893783 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.893767    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99408a11-d68b-4272-94a0-475ef7c72016" containerName="payload-pre-processing"
Oct 30 01:07:00.893899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.893796    4425 cpu_manager.go:414] "containerMap: removing container" podUID="075fdcb4-b628-4603-ab90-e7575e3096df" containerName="istio-proxy"
Oct 30 01:07:00.893899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.893802    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="075fdcb4-b628-4603-ab90-e7575e3096df" containerName="istio-proxy"
Oct 30 01:07:00.893899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.893812    4425 cpu_manager.go:414] "containerMap: removing container" podUID="743ad5a9-7115-4060-90ad-1ae7e8c823d5" containerName="payload-pre-processing"
Oct 30 01:07:00.893899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.893817    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="743ad5a9-7115-4060-90ad-1ae7e8c823d5" containerName="payload-pre-processing"
Oct 30 01:07:00.893899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.893824    4425 cpu_manager.go:414] "containerMap: removing container" podUID="34e74245-057c-4a90-bcc7-818149e6f991" containerName="payload-processing"
Oct 30 01:07:00.893899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.893830    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34e74245-057c-4a90-bcc7-818149e6f991" containerName="payload-processing"
Oct 30 01:07:00.893899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.893844    4425 cpu_manager.go:414] "containerMap: removing container" podUID="26a604e1-2ecc-47c9-8e0c-1babdcaa5113" containerName="payload-processing"
Oct 30 01:07:00.893899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.893848    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a604e1-2ecc-47c9-8e0c-1babdcaa5113" containerName="payload-processing"
Oct 30 01:07:00.894328 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.893936    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a604e1-2ecc-47c9-8e0c-1babdcaa5113" containerName="payload-processing"
Oct 30 01:07:00.894328 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.893949    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="99408a11-d68b-4272-94a0-475ef7c72016" containerName="payload-pre-processing"
Oct 30 01:07:00.894328 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.893959    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="075fdcb4-b628-4603-ab90-e7575e3096df" containerName="istio-proxy"
Oct 30 01:07:00.894328 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.893966    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="743ad5a9-7115-4060-90ad-1ae7e8c823d5" containerName="payload-pre-processing"
Oct 30 01:07:00.894328 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.893972    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="34e74245-057c-4a90-bcc7-818149e6f991" containerName="payload-processing"
Oct 30 01:07:00.897377 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.897361    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"
Oct 30 01:07:00.910599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.910577    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f05249\""
Oct 30 01:07:00.924182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.924143    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"]
Oct 30 01:07:00.936840 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.936815    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"]
Oct 30 01:07:00.941382 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.941364    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"
Oct 30 01:07:00.954658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.954639    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f05249-dockercfg-w7hqx\""
Oct 30 01:07:00.962042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.962020    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"]
Oct 30 01:07:00.974206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.974184    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spk8s\" (UniqueName: \"kubernetes.io/projected/402b6055-b012-440a-a5b8-38d82dadd083-kube-api-access-spk8s\") pod \"payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9\" (UID: \"402b6055-b012-440a-a5b8-38d82dadd083\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"
Oct 30 01:07:00.974308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:00.974280    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402b6055-b012-440a-a5b8-38d82dadd083-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9\" (UID: \"402b6055-b012-440a-a5b8-38d82dadd083\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"
Oct 30 01:07:01.080184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:01.079899    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6kt4\" (UniqueName: \"kubernetes.io/projected/a2db3c79-863f-4c15-b6f6-a0eb7300d8e6-kube-api-access-z6kt4\") pod \"payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr\" (UID: \"a2db3c79-863f-4c15-b6f6-a0eb7300d8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"
Oct 30 01:07:01.080184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:01.080122    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spk8s\" (UniqueName: \"kubernetes.io/projected/402b6055-b012-440a-a5b8-38d82dadd083-kube-api-access-spk8s\") pod \"payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9\" (UID: \"402b6055-b012-440a-a5b8-38d82dadd083\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"
Oct 30 01:07:01.080443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:01.080188    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402b6055-b012-440a-a5b8-38d82dadd083-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9\" (UID: \"402b6055-b012-440a-a5b8-38d82dadd083\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"
Oct 30 01:07:01.080443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:01.080292    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2db3c79-863f-4c15-b6f6-a0eb7300d8e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr\" (UID: \"a2db3c79-863f-4c15-b6f6-a0eb7300d8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"
Oct 30 01:07:01.081749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:01.081721    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402b6055-b012-440a-a5b8-38d82dadd083-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9\" (UID: \"402b6055-b012-440a-a5b8-38d82dadd083\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"
Oct 30 01:07:01.130466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:01.130431    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spk8s\" (UniqueName: \"kubernetes.io/projected/402b6055-b012-440a-a5b8-38d82dadd083-kube-api-access-spk8s\") pod \"payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9\" (UID: \"402b6055-b012-440a-a5b8-38d82dadd083\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"
Oct 30 01:07:01.182047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:01.181975    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2db3c79-863f-4c15-b6f6-a0eb7300d8e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr\" (UID: \"a2db3c79-863f-4c15-b6f6-a0eb7300d8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"
Oct 30 01:07:01.182225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:01.182149    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6kt4\" (UniqueName: \"kubernetes.io/projected/a2db3c79-863f-4c15-b6f6-a0eb7300d8e6-kube-api-access-z6kt4\") pod \"payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr\" (UID: \"a2db3c79-863f-4c15-b6f6-a0eb7300d8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"
Oct 30 01:07:01.182756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:01.182737    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2db3c79-863f-4c15-b6f6-a0eb7300d8e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr\" (UID: \"a2db3c79-863f-4c15-b6f6-a0eb7300d8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"
Oct 30 01:07:01.210955 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:01.210928    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6kt4\" (UniqueName: \"kubernetes.io/projected/a2db3c79-863f-4c15-b6f6-a0eb7300d8e6-kube-api-access-z6kt4\") pod \"payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr\" (UID: \"a2db3c79-863f-4c15-b6f6-a0eb7300d8e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"
Oct 30 01:07:01.211184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:01.211135    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"
Oct 30 01:07:01.255043 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:01.255015    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"
Oct 30 01:07:01.517732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:01.517705    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"]
Oct 30 01:07:01.519186 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:07:01.519148    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402b6055_b012_440a_a5b8_38d82dadd083.slice/crio-da9fa53a81764ec95d0610c99a14174c66a7e2ae4471455b60084e847a47e0c8 WatchSource:0}: Error finding container da9fa53a81764ec95d0610c99a14174c66a7e2ae4471455b60084e847a47e0c8: Status 404 returned error can't find the container with id da9fa53a81764ec95d0610c99a14174c66a7e2ae4471455b60084e847a47e0c8
Oct 30 01:07:01.569684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:01.569658    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"]
Oct 30 01:07:01.570492 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:07:01.570471    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2db3c79_863f_4c15_b6f6_a0eb7300d8e6.slice/crio-48124a877e8ee59dcae72397e19f06f3e44492d6cd24b7c4d0a67de0638ef2d6 WatchSource:0}: Error finding container 48124a877e8ee59dcae72397e19f06f3e44492d6cd24b7c4d0a67de0638ef2d6: Status 404 returned error can't find the container with id 48124a877e8ee59dcae72397e19f06f3e44492d6cd24b7c4d0a67de0638ef2d6
Oct 30 01:07:02.140217 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.140185    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9" event={"ID":"402b6055-b012-440a-a5b8-38d82dadd083","Type":"ContainerStarted","Data":"1f2b92376ca8d17a65513539456501d838f64d6176d7617a6231a3615e81167f"}
Oct 30 01:07:02.140656 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.140242    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9" event={"ID":"402b6055-b012-440a-a5b8-38d82dadd083","Type":"ContainerStarted","Data":"da9fa53a81764ec95d0610c99a14174c66a7e2ae4471455b60084e847a47e0c8"}
Oct 30 01:07:02.140656 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.140282    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"
Oct 30 01:07:02.141893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.141868    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr" event={"ID":"a2db3c79-863f-4c15-b6f6-a0eb7300d8e6","Type":"ContainerStarted","Data":"d3f6332ac589eb1968faedecc66e7f5e55842abb558c5442f635144c426511c9"}
Oct 30 01:07:02.141992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.141916    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"
Oct 30 01:07:02.141992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.141937    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr" event={"ID":"a2db3c79-863f-4c15-b6f6-a0eb7300d8e6","Type":"ContainerStarted","Data":"48124a877e8ee59dcae72397e19f06f3e44492d6cd24b7c4d0a67de0638ef2d6"}
Oct 30 01:07:02.178890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.178846    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9" podStartSLOduration=2.178830557 podStartE2EDuration="2.178830557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:02.175320424 +0000 UTC m=+1530.924975732" watchObservedRunningTime="2026-10-30 01:07:02.178830557 +0000 UTC m=+1530.928485866"
Oct 30 01:07:02.212295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.212249    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr" podStartSLOduration=2.212236938 podStartE2EDuration="2.212236938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:02.211090022 +0000 UTC m=+1530.960745331" watchObservedRunningTime="2026-10-30 01:07:02.212236938 +0000 UTC m=+1530.961892246"
Oct 30 01:07:02.643643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.643616    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:07:02.703294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.703212    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-workload-certs\") pod \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") "
Oct 30 01:07:02.703294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.703264    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-envoy\") pod \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") "
Oct 30 01:07:02.703478 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.703361    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-podinfo\") pod \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") "
Oct 30 01:07:02.703521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.703491    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-workload-certs" pod "30db65f6-70ad-4a3b-bed9-fb4f74533b3a" (UID: "30db65f6-70ad-4a3b-bed9-fb4f74533b3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:02.703521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.703504    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istiod-ca-cert\") pod \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") "
Oct 30 01:07:02.703607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.703548    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-data\") pod \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") "
Oct 30 01:07:02.703607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.703592    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-workload-socket\") pod \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") "
Oct 30 01:07:02.703724 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.703706    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-credential-socket\") pod \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") "
Oct 30 01:07:02.703775 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.703746    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-token\" (UniqueName: \"kubernetes.io/projected/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-token\") pod \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") "
Oct 30 01:07:02.703838 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.703792    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-kube-api-access-kv6rn\" (UniqueName: \"kubernetes.io/projected/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-kube-api-access-kv6rn\") pod \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\" (UID: \"30db65f6-70ad-4a3b-bed9-fb4f74533b3a\") "
Oct 30 01:07:02.703916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.703894    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istiod-ca-cert" pod "30db65f6-70ad-4a3b-bed9-fb4f74533b3a" (UID: "30db65f6-70ad-4a3b-bed9-fb4f74533b3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:02.704462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.704249    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-credential-socket" pod "30db65f6-70ad-4a3b-bed9-fb4f74533b3a" (UID: "30db65f6-70ad-4a3b-bed9-fb4f74533b3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:02.704462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.704398    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:02.704462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.704419    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:02.704462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.704435    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:02.705011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.704986    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-workload-socket" pod "30db65f6-70ad-4a3b-bed9-fb4f74533b3a" (UID: "30db65f6-70ad-4a3b-bed9-fb4f74533b3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:02.705118 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.705048    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-data" pod "30db65f6-70ad-4a3b-bed9-fb4f74533b3a" (UID: "30db65f6-70ad-4a3b-bed9-fb4f74533b3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:02.705914 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.705885    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-podinfo" pod "30db65f6-70ad-4a3b-bed9-fb4f74533b3a" (UID: "30db65f6-70ad-4a3b-bed9-fb4f74533b3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:07:02.706011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.705988    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-token" pod "30db65f6-70ad-4a3b-bed9-fb4f74533b3a" (UID: "30db65f6-70ad-4a3b-bed9-fb4f74533b3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:02.706120 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.706098    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-envoy" pod "30db65f6-70ad-4a3b-bed9-fb4f74533b3a" (UID: "30db65f6-70ad-4a3b-bed9-fb4f74533b3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:07:02.706462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.706440    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-kube-api-access-kv6rn" pod "30db65f6-70ad-4a3b-bed9-fb4f74533b3a" (UID: "30db65f6-70ad-4a3b-bed9-fb4f74533b3a"). InnerVolumeSpecName "kube-api-access-kv6rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:02.806814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.806773    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:02.806814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.806816    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:02.807001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.806831    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:02.807001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.806845    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv6rn\" (UniqueName: \"kubernetes.io/projected/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-kube-api-access-kv6rn\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:02.807001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.806859    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:02.807001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:02.806872    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30db65f6-70ad-4a3b-bed9-fb4f74533b3a-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:03.151395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:03.151362    4425 generic.go:356] "Generic (PLEG): container finished" podID="30db65f6-70ad-4a3b-bed9-fb4f74533b3a" containerID="f746ab01ba64474fcb169e668a363861456a436abd2c5118ea64be738cfa167e" exitCode=0
Oct 30 01:07:03.151771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:03.151412    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"
Oct 30 01:07:03.151771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:03.151452    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg" event={"ID":"30db65f6-70ad-4a3b-bed9-fb4f74533b3a","Type":"ContainerDied","Data":"f746ab01ba64474fcb169e668a363861456a436abd2c5118ea64be738cfa167e"}
Oct 30 01:07:03.151771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:03.151512    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg" event={"ID":"30db65f6-70ad-4a3b-bed9-fb4f74533b3a","Type":"ContainerDied","Data":"2a40707767bbaa26283d8c0f6868289a27dc8daad0470316439960fc5befa78d"}
Oct 30 01:07:03.151771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:03.151545    4425 scope.go:122] "RemoveContainer" containerID="f746ab01ba64474fcb169e668a363861456a436abd2c5118ea64be738cfa167e"
Oct 30 01:07:03.162288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:03.162267    4425 scope.go:122] "RemoveContainer" containerID="f746ab01ba64474fcb169e668a363861456a436abd2c5118ea64be738cfa167e"
Oct 30 01:07:03.162575 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:07:03.162556    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f746ab01ba64474fcb169e668a363861456a436abd2c5118ea64be738cfa167e\": container with ID starting with f746ab01ba64474fcb169e668a363861456a436abd2c5118ea64be738cfa167e not found: ID does not exist" containerID="f746ab01ba64474fcb169e668a363861456a436abd2c5118ea64be738cfa167e"
Oct 30 01:07:03.162648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:03.162587    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f746ab01ba64474fcb169e668a363861456a436abd2c5118ea64be738cfa167e"} err="failed to get container status \"f746ab01ba64474fcb169e668a363861456a436abd2c5118ea64be738cfa167e\": rpc error: code = NotFound desc = could not find container \"f746ab01ba64474fcb169e668a363861456a436abd2c5118ea64be738cfa167e\": container with ID starting with f746ab01ba64474fcb169e668a363861456a436abd2c5118ea64be738cfa167e not found: ID does not exist"
Oct 30 01:07:03.196205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:03.196148    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"]
Oct 30 01:07:03.205564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:03.205545    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-6nbmg"]
Oct 30 01:07:03.421932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:03.421858    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30db65f6-70ad-4a3b-bed9-fb4f74533b3a" path="/var/lib/kubelet/pods/30db65f6-70ad-4a3b-bed9-fb4f74533b3a/volumes"
Oct 30 01:07:07.554292 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:07:07.554253    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:07:10.035476 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:10.035433    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"]
Oct 30 01:07:10.044806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:10.044781    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc74a2d0-wkj8z"]
Oct 30 01:07:11.425052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:11.425020    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71736696-7d27-45c2-9768-23978dc9e3aa" path="/var/lib/kubelet/pods/71736696-7d27-45c2-9768-23978dc9e3aa/volumes"
Oct 30 01:07:13.155144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:13.155118    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"
Oct 30 01:07:13.155608 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:13.155208    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"
Oct 30 01:07:17.053528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:17.053491    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"]
Oct 30 01:07:17.063040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:17.063009    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a1a121-pv2fd"]
Oct 30 01:07:17.422003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:17.421925    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2" path="/var/lib/kubelet/pods/9dd68b0f-cf8f-44bd-8f79-2e247fc43ec2/volumes"
Oct 30 01:07:18.746706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.746658    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"]
Oct 30 01:07:18.747398 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.747378    4425 cpu_manager.go:414] "containerMap: removing container" podUID="30db65f6-70ad-4a3b-bed9-fb4f74533b3a" containerName="istio-proxy"
Oct 30 01:07:18.747398 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.747398    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30db65f6-70ad-4a3b-bed9-fb4f74533b3a" containerName="istio-proxy"
Oct 30 01:07:18.747528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.747513    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="30db65f6-70ad-4a3b-bed9-fb4f74533b3a" containerName="istio-proxy"
Oct 30 01:07:18.754190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.754136    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"
Oct 30 01:07:18.776925 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.776898    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:07:18.777079 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.776939    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:07:18.777079 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.777013    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:07:18.777079 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.777079    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:07:18.791191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.781043    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"]
Oct 30 01:07:18.795004 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.794976    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74344f02-85de-4ea4-9da0-7580efea93ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9\" (UID: \"74344f02-85de-4ea4-9da0-7580efea93ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"
Oct 30 01:07:18.795191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.795139    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74344f02-85de-4ea4-9da0-7580efea93ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9\" (UID: \"74344f02-85de-4ea4-9da0-7580efea93ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"
Oct 30 01:07:18.795345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.795320    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw54r\" (UniqueName: \"kubernetes.io/projected/74344f02-85de-4ea4-9da0-7580efea93ca-kube-api-access-jw54r\") pod \"maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9\" (UID: \"74344f02-85de-4ea4-9da0-7580efea93ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"
Oct 30 01:07:18.896918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.896883    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw54r\" (UniqueName: \"kubernetes.io/projected/74344f02-85de-4ea4-9da0-7580efea93ca-kube-api-access-jw54r\") pod \"maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9\" (UID: \"74344f02-85de-4ea4-9da0-7580efea93ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"
Oct 30 01:07:18.897097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.896930    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74344f02-85de-4ea4-9da0-7580efea93ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9\" (UID: \"74344f02-85de-4ea4-9da0-7580efea93ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"
Oct 30 01:07:18.897097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.897056    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74344f02-85de-4ea4-9da0-7580efea93ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9\" (UID: \"74344f02-85de-4ea4-9da0-7580efea93ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"
Oct 30 01:07:18.897755 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.897733    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74344f02-85de-4ea4-9da0-7580efea93ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9\" (UID: \"74344f02-85de-4ea4-9da0-7580efea93ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"
Oct 30 01:07:18.899422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.899391    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74344f02-85de-4ea4-9da0-7580efea93ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9\" (UID: \"74344f02-85de-4ea4-9da0-7580efea93ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"
Oct 30 01:07:18.917791 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:18.917762    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw54r\" (UniqueName: \"kubernetes.io/projected/74344f02-85de-4ea4-9da0-7580efea93ca-kube-api-access-jw54r\") pod \"maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9\" (UID: \"74344f02-85de-4ea4-9da0-7580efea93ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"
Oct 30 01:07:19.068878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:19.068842    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"
Oct 30 01:07:19.373548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:19.373520    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"]
Oct 30 01:07:19.374518 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:07:19.374495    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74344f02_85de_4ea4_9da0_7580efea93ca.slice/crio-72db8fd3d25ad83a16274c73657b86c9231b56ea0be6dbd9cd89cf4fe1b76ba4 WatchSource:0}: Error finding container 72db8fd3d25ad83a16274c73657b86c9231b56ea0be6dbd9cd89cf4fe1b76ba4: Status 404 returned error can't find the container with id 72db8fd3d25ad83a16274c73657b86c9231b56ea0be6dbd9cd89cf4fe1b76ba4
Oct 30 01:07:20.275058 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:20.275027    4425 generic.go:356] "Generic (PLEG): container finished" podID="74344f02-85de-4ea4-9da0-7580efea93ca" containerID="0a9bab0b6a4cc2eca0468b734317b4f1ba338152d6f7478b4635a5b1de4c3148" exitCode=0
Oct 30 01:07:20.275420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:20.275106    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9" event={"ID":"74344f02-85de-4ea4-9da0-7580efea93ca","Type":"ContainerDied","Data":"0a9bab0b6a4cc2eca0468b734317b4f1ba338152d6f7478b4635a5b1de4c3148"}
Oct 30 01:07:20.275420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:20.275182    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9" event={"ID":"74344f02-85de-4ea4-9da0-7580efea93ca","Type":"ContainerStarted","Data":"72db8fd3d25ad83a16274c73657b86c9231b56ea0be6dbd9cd89cf4fe1b76ba4"}
Oct 30 01:07:21.255283 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:21.255248    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"]
Oct 30 01:07:21.502093 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:21.502056    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"
Oct 30 01:07:21.527942 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:21.527912    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74344f02-85de-4ea4-9da0-7580efea93ca-kube-api-access-jw54r\" (UniqueName: \"kubernetes.io/projected/74344f02-85de-4ea4-9da0-7580efea93ca-kube-api-access-jw54r\") pod \"74344f02-85de-4ea4-9da0-7580efea93ca\" (UID: \"74344f02-85de-4ea4-9da0-7580efea93ca\") "
Oct 30 01:07:21.528127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:21.528046    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74344f02-85de-4ea4-9da0-7580efea93ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74344f02-85de-4ea4-9da0-7580efea93ca-maas-api-openshift-ca\") pod \"74344f02-85de-4ea4-9da0-7580efea93ca\" (UID: \"74344f02-85de-4ea4-9da0-7580efea93ca\") "
Oct 30 01:07:21.528215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:21.528128    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74344f02-85de-4ea4-9da0-7580efea93ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74344f02-85de-4ea4-9da0-7580efea93ca-maas-api-xks-ca\") pod \"74344f02-85de-4ea4-9da0-7580efea93ca\" (UID: \"74344f02-85de-4ea4-9da0-7580efea93ca\") "
Oct 30 01:07:21.528494 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:21.528465    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74344f02-85de-4ea4-9da0-7580efea93ca-maas-api-openshift-ca" pod "74344f02-85de-4ea4-9da0-7580efea93ca" (UID: "74344f02-85de-4ea4-9da0-7580efea93ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:21.528752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:21.528693    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74344f02-85de-4ea4-9da0-7580efea93ca-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:21.530574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:21.530546    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74344f02-85de-4ea4-9da0-7580efea93ca-kube-api-access-jw54r" pod "74344f02-85de-4ea4-9da0-7580efea93ca" (UID: "74344f02-85de-4ea4-9da0-7580efea93ca"). InnerVolumeSpecName "kube-api-access-jw54r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:21.531950 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:21.531926    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74344f02-85de-4ea4-9da0-7580efea93ca-maas-api-xks-ca" pod "74344f02-85de-4ea4-9da0-7580efea93ca" (UID: "74344f02-85de-4ea4-9da0-7580efea93ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:07:21.629545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:21.629514    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw54r\" (UniqueName: \"kubernetes.io/projected/74344f02-85de-4ea4-9da0-7580efea93ca-kube-api-access-jw54r\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:21.629545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:21.629544    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74344f02-85de-4ea4-9da0-7580efea93ca-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:22.292656 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:22.292626    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"
Oct 30 01:07:22.292826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:22.292655    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9" event={"ID":"74344f02-85de-4ea4-9da0-7580efea93ca","Type":"ContainerDied","Data":"72db8fd3d25ad83a16274c73657b86c9231b56ea0be6dbd9cd89cf4fe1b76ba4"}
Oct 30 01:07:22.292826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:22.292704    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72db8fd3d25ad83a16274c73657b86c9231b56ea0be6dbd9cd89cf4fe1b76ba4"
Oct 30 01:07:23.967033 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:23.966998    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:07:26.255669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:26.255628    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9" podUID="dd5fae0e-4675-4a1d-b187-279fcafe3984" containerName="payload-processing" containerID="cri-o://9dd3abe7a6a6d8ee1849a6ae3bf4c13aa48513665ec581074e65f8fb1c3da5fe" gracePeriod=25
Oct 30 01:07:26.531292 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:26.531266    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"
Oct 30 01:07:26.587144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:26.587112    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd5fae0e-4675-4a1d-b187-279fcafe3984-kube-api-access-ftdp9\" (UniqueName: \"kubernetes.io/projected/dd5fae0e-4675-4a1d-b187-279fcafe3984-kube-api-access-ftdp9\") pod \"dd5fae0e-4675-4a1d-b187-279fcafe3984\" (UID: \"dd5fae0e-4675-4a1d-b187-279fcafe3984\") "
Oct 30 01:07:26.587295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:26.587229    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd5fae0e-4675-4a1d-b187-279fcafe3984-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5fae0e-4675-4a1d-b187-279fcafe3984-plugins-config-volume\") pod \"dd5fae0e-4675-4a1d-b187-279fcafe3984\" (UID: \"dd5fae0e-4675-4a1d-b187-279fcafe3984\") "
Oct 30 01:07:26.587568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:26.587547    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd5fae0e-4675-4a1d-b187-279fcafe3984-plugins-config-volume" pod "dd5fae0e-4675-4a1d-b187-279fcafe3984" (UID: "dd5fae0e-4675-4a1d-b187-279fcafe3984"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:26.587762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:26.587742    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd5fae0e-4675-4a1d-b187-279fcafe3984-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:26.589096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:26.589074    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5fae0e-4675-4a1d-b187-279fcafe3984-kube-api-access-ftdp9" pod "dd5fae0e-4675-4a1d-b187-279fcafe3984" (UID: "dd5fae0e-4675-4a1d-b187-279fcafe3984"). InnerVolumeSpecName "kube-api-access-ftdp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:26.688955 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:26.688925    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftdp9\" (UniqueName: \"kubernetes.io/projected/dd5fae0e-4675-4a1d-b187-279fcafe3984-kube-api-access-ftdp9\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:27.333921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:27.333884    4425 generic.go:356] "Generic (PLEG): container finished" podID="dd5fae0e-4675-4a1d-b187-279fcafe3984" containerID="9dd3abe7a6a6d8ee1849a6ae3bf4c13aa48513665ec581074e65f8fb1c3da5fe" exitCode=0
Oct 30 01:07:27.334443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:27.333926    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9" event={"ID":"dd5fae0e-4675-4a1d-b187-279fcafe3984","Type":"ContainerDied","Data":"9dd3abe7a6a6d8ee1849a6ae3bf4c13aa48513665ec581074e65f8fb1c3da5fe"}
Oct 30 01:07:27.334443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:27.333972    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9" event={"ID":"dd5fae0e-4675-4a1d-b187-279fcafe3984","Type":"ContainerDied","Data":"072d625a897c0ce3a1493e2b4e8d65e48cb238f4b005ec86db669550498738a1"}
Oct 30 01:07:27.334443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:27.334000    4425 scope.go:122] "RemoveContainer" containerID="9dd3abe7a6a6d8ee1849a6ae3bf4c13aa48513665ec581074e65f8fb1c3da5fe"
Oct 30 01:07:27.334443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:27.333929    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"
Oct 30 01:07:27.345118 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:27.345100    4425 scope.go:122] "RemoveContainer" containerID="9dd3abe7a6a6d8ee1849a6ae3bf4c13aa48513665ec581074e65f8fb1c3da5fe"
Oct 30 01:07:27.345416 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:07:27.345398    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dd3abe7a6a6d8ee1849a6ae3bf4c13aa48513665ec581074e65f8fb1c3da5fe\": container with ID starting with 9dd3abe7a6a6d8ee1849a6ae3bf4c13aa48513665ec581074e65f8fb1c3da5fe not found: ID does not exist" containerID="9dd3abe7a6a6d8ee1849a6ae3bf4c13aa48513665ec581074e65f8fb1c3da5fe"
Oct 30 01:07:27.345486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:27.345431    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd3abe7a6a6d8ee1849a6ae3bf4c13aa48513665ec581074e65f8fb1c3da5fe"} err="failed to get container status \"9dd3abe7a6a6d8ee1849a6ae3bf4c13aa48513665ec581074e65f8fb1c3da5fe\": rpc error: code = NotFound desc = could not find container \"9dd3abe7a6a6d8ee1849a6ae3bf4c13aa48513665ec581074e65f8fb1c3da5fe\": container with ID starting with 9dd3abe7a6a6d8ee1849a6ae3bf4c13aa48513665ec581074e65f8fb1c3da5fe not found: ID does not exist"
Oct 30 01:07:27.381215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:27.381189    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"]
Oct 30 01:07:27.390271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:27.390250    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-838fd5-87b45878-t6nd9"]
Oct 30 01:07:27.423627 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:27.423601    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd5fae0e-4675-4a1d-b187-279fcafe3984" path="/var/lib/kubelet/pods/dd5fae0e-4675-4a1d-b187-279fcafe3984/volumes"
Oct 30 01:07:32.214377 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:32.214346    4425 scope.go:122] "RemoveContainer" containerID="492e0f7a7ae9ae22241308235fd80243ce4f2a055cee10a9cc6a2d3a32f95560"
Oct 30 01:07:32.225504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:32.225484    4425 scope.go:122] "RemoveContainer" containerID="fe010553a6472c59f38a9ac20d969ff5448ddb3c333c758d4b31d539729453d1"
Oct 30 01:07:32.242296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:32.238419    4425 scope.go:122] "RemoveContainer" containerID="29bc7cca1f5c7da995a73d5019c2e52c6bd20e62ac4d2d736dbb81d8fd2fb3ca"
Oct 30 01:07:37.569797 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:07:37.569753    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:07:40.500171 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.500131    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"]
Oct 30 01:07:40.500724 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.500706    4425 cpu_manager.go:414] "containerMap: removing container" podUID="74344f02-85de-4ea4-9da0-7580efea93ca" containerName="revoke-keys"
Oct 30 01:07:40.500724 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.500723    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74344f02-85de-4ea4-9da0-7580efea93ca" containerName="revoke-keys"
Oct 30 01:07:40.500893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.500756    4425 cpu_manager.go:414] "containerMap: removing container" podUID="dd5fae0e-4675-4a1d-b187-279fcafe3984" containerName="payload-processing"
Oct 30 01:07:40.500893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.500762    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd5fae0e-4675-4a1d-b187-279fcafe3984" containerName="payload-processing"
Oct 30 01:07:40.500893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.500844    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="74344f02-85de-4ea4-9da0-7580efea93ca" containerName="revoke-keys"
Oct 30 01:07:40.500893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.500858    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd5fae0e-4675-4a1d-b187-279fcafe3984" containerName="payload-processing"
Oct 30 01:07:40.506449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.506429    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"
Oct 30 01:07:40.525570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.525548    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"]
Oct 30 01:07:40.532444 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.532422    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-429c67\""
Oct 30 01:07:40.532535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.532463    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-429c67-dockercfg-2wz8t\""
Oct 30 01:07:40.554268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.554200    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30ebc35-d671-4159-a22c-4bfe4e628b9a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2\" (UID: \"c30ebc35-d671-4159-a22c-4bfe4e628b9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"
Oct 30 01:07:40.554384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.554340    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg82p\" (UniqueName: \"kubernetes.io/projected/c30ebc35-d671-4159-a22c-4bfe4e628b9a-kube-api-access-bg82p\") pod \"payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2\" (UID: \"c30ebc35-d671-4159-a22c-4bfe4e628b9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"
Oct 30 01:07:40.564918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.564890    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"]
Oct 30 01:07:40.569430 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.569409    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"
Oct 30 01:07:40.596194 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.591890    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"]
Oct 30 01:07:40.656177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.655882    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30ebc35-d671-4159-a22c-4bfe4e628b9a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2\" (UID: \"c30ebc35-d671-4159-a22c-4bfe4e628b9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"
Oct 30 01:07:40.656177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.655962    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrnjm\" (UniqueName: \"kubernetes.io/projected/406b7db9-a1a4-4e81-b692-577620838bb8-kube-api-access-vrnjm\") pod \"payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j\" (UID: \"406b7db9-a1a4-4e81-b692-577620838bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"
Oct 30 01:07:40.656177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.656009    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg82p\" (UniqueName: \"kubernetes.io/projected/c30ebc35-d671-4159-a22c-4bfe4e628b9a-kube-api-access-bg82p\") pod \"payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2\" (UID: \"c30ebc35-d671-4159-a22c-4bfe4e628b9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"
Oct 30 01:07:40.656177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.656101    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406b7db9-a1a4-4e81-b692-577620838bb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j\" (UID: \"406b7db9-a1a4-4e81-b692-577620838bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"
Oct 30 01:07:40.656864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.656839    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30ebc35-d671-4159-a22c-4bfe4e628b9a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2\" (UID: \"c30ebc35-d671-4159-a22c-4bfe4e628b9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"
Oct 30 01:07:40.678036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.678006    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg82p\" (UniqueName: \"kubernetes.io/projected/c30ebc35-d671-4159-a22c-4bfe4e628b9a-kube-api-access-bg82p\") pod \"payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2\" (UID: \"c30ebc35-d671-4159-a22c-4bfe4e628b9a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"
Oct 30 01:07:40.757327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.757233    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrnjm\" (UniqueName: \"kubernetes.io/projected/406b7db9-a1a4-4e81-b692-577620838bb8-kube-api-access-vrnjm\") pod \"payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j\" (UID: \"406b7db9-a1a4-4e81-b692-577620838bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"
Oct 30 01:07:40.757595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.757385    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406b7db9-a1a4-4e81-b692-577620838bb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j\" (UID: \"406b7db9-a1a4-4e81-b692-577620838bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"
Oct 30 01:07:40.758349 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.758320    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406b7db9-a1a4-4e81-b692-577620838bb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j\" (UID: \"406b7db9-a1a4-4e81-b692-577620838bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"
Oct 30 01:07:40.784007 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.783970    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrnjm\" (UniqueName: \"kubernetes.io/projected/406b7db9-a1a4-4e81-b692-577620838bb8-kube-api-access-vrnjm\") pod \"payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j\" (UID: \"406b7db9-a1a4-4e81-b692-577620838bb8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"
Oct 30 01:07:40.820470 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.820427    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"
Oct 30 01:07:40.890399 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:40.889324    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"
Oct 30 01:07:41.236135 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:41.236108    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"]
Oct 30 01:07:41.237712 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:07:41.237682    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30ebc35_d671_4159_a22c_4bfe4e628b9a.slice/crio-ecdc517bd784fa0c2ca51e554bc14bf27e88b675c0279897a7d96bc0f4532658 WatchSource:0}: Error finding container ecdc517bd784fa0c2ca51e554bc14bf27e88b675c0279897a7d96bc0f4532658: Status 404 returned error can't find the container with id ecdc517bd784fa0c2ca51e554bc14bf27e88b675c0279897a7d96bc0f4532658
Oct 30 01:07:41.259871 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:41.259840    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"]
Oct 30 01:07:41.260939 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:07:41.260914    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406b7db9_a1a4_4e81_b692_577620838bb8.slice/crio-b6ca25dd002367e7936e9b6244aa240d2cf8481c4c45a863710ae26ab7b1fbf1 WatchSource:0}: Error finding container b6ca25dd002367e7936e9b6244aa240d2cf8481c4c45a863710ae26ab7b1fbf1: Status 404 returned error can't find the container with id b6ca25dd002367e7936e9b6244aa240d2cf8481c4c45a863710ae26ab7b1fbf1
Oct 30 01:07:41.430392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:41.430354    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j" event={"ID":"406b7db9-a1a4-4e81-b692-577620838bb8","Type":"ContainerStarted","Data":"46351a06b497f92fe80d7ad4bd886008959ed4a00579d8a42390adad71e3ff47"}
Oct 30 01:07:41.430595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:41.430419    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"
Oct 30 01:07:41.430595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:41.430444    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j" event={"ID":"406b7db9-a1a4-4e81-b692-577620838bb8","Type":"ContainerStarted","Data":"b6ca25dd002367e7936e9b6244aa240d2cf8481c4c45a863710ae26ab7b1fbf1"}
Oct 30 01:07:41.432430 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:41.432404    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2" event={"ID":"c30ebc35-d671-4159-a22c-4bfe4e628b9a","Type":"ContainerStarted","Data":"d417c03aa3b666bed7de1fa6785a3f14f1027e7f92d3fb91234e771477bc53dd"}
Oct 30 01:07:41.432545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:41.432453    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2" event={"ID":"c30ebc35-d671-4159-a22c-4bfe4e628b9a","Type":"ContainerStarted","Data":"ecdc517bd784fa0c2ca51e554bc14bf27e88b675c0279897a7d96bc0f4532658"}
Oct 30 01:07:41.432545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:41.432480    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"
Oct 30 01:07:41.467011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:41.466967    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j" podStartSLOduration=1.46695423 podStartE2EDuration="1.46695423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:41.465753285 +0000 UTC m=+1570.215408594" watchObservedRunningTime="2026-10-30 01:07:41.46695423 +0000 UTC m=+1570.216609588"
Oct 30 01:07:41.501908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:41.501806    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2" podStartSLOduration=1.501788305 podStartE2EDuration="1.501788305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:41.500024885 +0000 UTC m=+1570.249680193" watchObservedRunningTime="2026-10-30 01:07:41.501788305 +0000 UTC m=+1570.251443679"
Oct 30 01:07:42.844982 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:42.844948    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"]
Oct 30 01:07:42.850059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:42.850033    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"
Oct 30 01:07:42.875538 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:42.875508    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-838fd5\""
Oct 30 01:07:42.875698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:42.875543    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-838fd5-dockercfg-f4zxz\""
Oct 30 01:07:42.882364 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:42.881340    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"]
Oct 30 01:07:42.913275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:42.913236    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de9afa7c-812e-42be-84d7-19f7d180255d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd\" (UID: \"de9afa7c-812e-42be-84d7-19f7d180255d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"
Oct 30 01:07:42.913573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:42.913545    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp5jf\" (UniqueName: \"kubernetes.io/projected/de9afa7c-812e-42be-84d7-19f7d180255d-kube-api-access-zp5jf\") pod \"payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd\" (UID: \"de9afa7c-812e-42be-84d7-19f7d180255d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"
Oct 30 01:07:43.015241 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:43.015196    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp5jf\" (UniqueName: \"kubernetes.io/projected/de9afa7c-812e-42be-84d7-19f7d180255d-kube-api-access-zp5jf\") pod \"payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd\" (UID: \"de9afa7c-812e-42be-84d7-19f7d180255d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"
Oct 30 01:07:43.015435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:43.015300    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de9afa7c-812e-42be-84d7-19f7d180255d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd\" (UID: \"de9afa7c-812e-42be-84d7-19f7d180255d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"
Oct 30 01:07:43.016240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:43.016211    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de9afa7c-812e-42be-84d7-19f7d180255d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd\" (UID: \"de9afa7c-812e-42be-84d7-19f7d180255d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"
Oct 30 01:07:43.038988 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:43.038958    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp5jf\" (UniqueName: \"kubernetes.io/projected/de9afa7c-812e-42be-84d7-19f7d180255d-kube-api-access-zp5jf\") pod \"payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd\" (UID: \"de9afa7c-812e-42be-84d7-19f7d180255d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"
Oct 30 01:07:43.166382 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:43.166307    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"
Oct 30 01:07:43.474900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:43.474876    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"]
Oct 30 01:07:43.476956 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:07:43.476930    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9afa7c_812e_42be_84d7_19f7d180255d.slice/crio-85e97e8656dad09c0f67eff0b659ca264836e81ba53171780c7d9dbed46d8a7b WatchSource:0}: Error finding container 85e97e8656dad09c0f67eff0b659ca264836e81ba53171780c7d9dbed46d8a7b: Status 404 returned error can't find the container with id 85e97e8656dad09c0f67eff0b659ca264836e81ba53171780c7d9dbed46d8a7b
Oct 30 01:07:44.453167 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:44.453057    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd" event={"ID":"de9afa7c-812e-42be-84d7-19f7d180255d","Type":"ContainerStarted","Data":"2d18797556d743fa4d2458c47bfd35658e04ef9aeb7e110fc05891c8eb8cb6e6"}
Oct 30 01:07:44.453631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:44.453149    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd" event={"ID":"de9afa7c-812e-42be-84d7-19f7d180255d","Type":"ContainerStarted","Data":"85e97e8656dad09c0f67eff0b659ca264836e81ba53171780c7d9dbed46d8a7b"}
Oct 30 01:07:44.453631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:44.453321    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"
Oct 30 01:07:44.497143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:44.494917    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd" podStartSLOduration=2.494888466 podStartE2EDuration="2.494888466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:44.489650672 +0000 UTC m=+1573.239305983" watchObservedRunningTime="2026-10-30 01:07:44.494888466 +0000 UTC m=+1573.244543777"
Oct 30 01:07:46.441365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.441316    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"]
Oct 30 01:07:46.446015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.445984    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"
Oct 30 01:07:46.462329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.462303    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b43bff\""
Oct 30 01:07:46.475521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.475480    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"]
Oct 30 01:07:46.516562 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.516521    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"]
Oct 30 01:07:46.520990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.520964    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"
Oct 30 01:07:46.537484 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.537456    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b43bff-dockercfg-kmj7z\""
Oct 30 01:07:46.538239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.538216    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"]
Oct 30 01:07:46.568666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.568612    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9\" (UID: \"af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"
Oct 30 01:07:46.570546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.568760    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rswv6\" (UniqueName: \"kubernetes.io/projected/a8e9db36-b363-4056-a290-8755b8dd91ec-kube-api-access-rswv6\") pod \"payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c\" (UID: \"a8e9db36-b363-4056-a290-8755b8dd91ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"
Oct 30 01:07:46.570546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.568837    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vng5k\" (UniqueName: \"kubernetes.io/projected/af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3-kube-api-access-vng5k\") pod \"payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9\" (UID: \"af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"
Oct 30 01:07:46.570546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.569016    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e9db36-b363-4056-a290-8755b8dd91ec-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c\" (UID: \"a8e9db36-b363-4056-a290-8755b8dd91ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"
Oct 30 01:07:46.670893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.670851    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9\" (UID: \"af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"
Oct 30 01:07:46.671116 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.670935    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rswv6\" (UniqueName: \"kubernetes.io/projected/a8e9db36-b363-4056-a290-8755b8dd91ec-kube-api-access-rswv6\") pod \"payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c\" (UID: \"a8e9db36-b363-4056-a290-8755b8dd91ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"
Oct 30 01:07:46.671116 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.670973    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vng5k\" (UniqueName: \"kubernetes.io/projected/af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3-kube-api-access-vng5k\") pod \"payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9\" (UID: \"af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"
Oct 30 01:07:46.671258 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.671123    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e9db36-b363-4056-a290-8755b8dd91ec-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c\" (UID: \"a8e9db36-b363-4056-a290-8755b8dd91ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"
Oct 30 01:07:46.671867 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.671840    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9\" (UID: \"af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"
Oct 30 01:07:46.671967 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.671951    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e9db36-b363-4056-a290-8755b8dd91ec-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c\" (UID: \"a8e9db36-b363-4056-a290-8755b8dd91ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"
Oct 30 01:07:46.692861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.692782    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rswv6\" (UniqueName: \"kubernetes.io/projected/a8e9db36-b363-4056-a290-8755b8dd91ec-kube-api-access-rswv6\") pod \"payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c\" (UID: \"a8e9db36-b363-4056-a290-8755b8dd91ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"
Oct 30 01:07:46.693037 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.693009    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vng5k\" (UniqueName: \"kubernetes.io/projected/af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3-kube-api-access-vng5k\") pod \"payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9\" (UID: \"af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"
Oct 30 01:07:46.765046 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.765006    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"
Oct 30 01:07:46.836590 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:46.836493    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"
Oct 30 01:07:47.113768 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:47.113741    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"]
Oct 30 01:07:47.116383 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:07:47.116352    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e9db36_b363_4056_a290_8755b8dd91ec.slice/crio-e708c3da4f1a57d6d06e8aa26d41980b0fa2abf36eb7d5308366ce5b2490c6be WatchSource:0}: Error finding container e708c3da4f1a57d6d06e8aa26d41980b0fa2abf36eb7d5308366ce5b2490c6be: Status 404 returned error can't find the container with id e708c3da4f1a57d6d06e8aa26d41980b0fa2abf36eb7d5308366ce5b2490c6be
Oct 30 01:07:47.168387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:47.168353    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"]
Oct 30 01:07:47.168661 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:07:47.168635    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf25d92a_3a8e_40ae_b4d8_cfc9ee3287b3.slice/crio-2c8cdf1ad5fbe64ea53b0be502e062219795b6147bd0c35c2afb1f5137249fc9 WatchSource:0}: Error finding container 2c8cdf1ad5fbe64ea53b0be502e062219795b6147bd0c35c2afb1f5137249fc9: Status 404 returned error can't find the container with id 2c8cdf1ad5fbe64ea53b0be502e062219795b6147bd0c35c2afb1f5137249fc9
Oct 30 01:07:47.483569 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:47.483454    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9" event={"ID":"af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3","Type":"ContainerStarted","Data":"b09301be4d6ae125613d209f4e607db810a7b8604bf32d8b023792be2e46faec"}
Oct 30 01:07:47.483569 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:47.483540    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"
Oct 30 01:07:47.484133 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:47.483571    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9" event={"ID":"af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3","Type":"ContainerStarted","Data":"2c8cdf1ad5fbe64ea53b0be502e062219795b6147bd0c35c2afb1f5137249fc9"}
Oct 30 01:07:47.485388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:47.485353    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c" event={"ID":"a8e9db36-b363-4056-a290-8755b8dd91ec","Type":"ContainerStarted","Data":"98d0282e1f8b669f7f94f0d49667c8ab0da35b83c551cab13cd7c5cffadf828c"}
Oct 30 01:07:47.485557 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:47.485421    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c" event={"ID":"a8e9db36-b363-4056-a290-8755b8dd91ec","Type":"ContainerStarted","Data":"e708c3da4f1a57d6d06e8aa26d41980b0fa2abf36eb7d5308366ce5b2490c6be"}
Oct 30 01:07:47.485791 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:47.485771    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"
Oct 30 01:07:47.520214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:47.520130    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9" podStartSLOduration=1.5201130649999999 podStartE2EDuration="1.520113065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:47.518535171 +0000 UTC m=+1576.268190479" watchObservedRunningTime="2026-10-30 01:07:47.520113065 +0000 UTC m=+1576.269768374"
Oct 30 01:07:47.554306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:47.554232    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c" podStartSLOduration=1.554216957 podStartE2EDuration="1.554216957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:47.553716983 +0000 UTC m=+1576.303372294" watchObservedRunningTime="2026-10-30 01:07:47.554216957 +0000 UTC m=+1576.303872265"
Oct 30 01:07:49.871708 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:49.871663    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"]
Oct 30 01:07:49.918216 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:49.918143    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"]
Oct 30 01:07:52.442645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:52.442618    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"
Oct 30 01:07:52.443051 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:52.442692    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"
Oct 30 01:07:53.146237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:53.146200    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:07:53.871410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:53.871376    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"]
Oct 30 01:07:53.884068 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:53.884032    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d4622a-8nrn9"]
Oct 30 01:07:54.872056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:54.872010    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9" podUID="402b6055-b012-440a-a5b8-38d82dadd083" containerName="payload-processing" containerID="cri-o://1f2b92376ca8d17a65513539456501d838f64d6176d7617a6231a3615e81167f" gracePeriod=25
Oct 30 01:07:54.918637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:54.918603    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr" podUID="a2db3c79-863f-4c15-b6f6-a0eb7300d8e6" containerName="payload-pre-processing" containerID="cri-o://d3f6332ac589eb1968faedecc66e7f5e55842abb558c5442f635144c426511c9" gracePeriod=25
Oct 30 01:07:55.202525 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.202493    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"
Oct 30 01:07:55.242518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.241924    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"
Oct 30 01:07:55.290902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.290868    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/402b6055-b012-440a-a5b8-38d82dadd083-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402b6055-b012-440a-a5b8-38d82dadd083-plugins-config-volume\") pod \"402b6055-b012-440a-a5b8-38d82dadd083\" (UID: \"402b6055-b012-440a-a5b8-38d82dadd083\") "
Oct 30 01:07:55.291137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.291117    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/402b6055-b012-440a-a5b8-38d82dadd083-kube-api-access-spk8s\" (UniqueName: \"kubernetes.io/projected/402b6055-b012-440a-a5b8-38d82dadd083-kube-api-access-spk8s\") pod \"402b6055-b012-440a-a5b8-38d82dadd083\" (UID: \"402b6055-b012-440a-a5b8-38d82dadd083\") "
Oct 30 01:07:55.291458 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.291321    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402b6055-b012-440a-a5b8-38d82dadd083-plugins-config-volume" pod "402b6055-b012-440a-a5b8-38d82dadd083" (UID: "402b6055-b012-440a-a5b8-38d82dadd083"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:55.291738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.291714    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402b6055-b012-440a-a5b8-38d82dadd083-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:55.293405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.293372    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402b6055-b012-440a-a5b8-38d82dadd083-kube-api-access-spk8s" pod "402b6055-b012-440a-a5b8-38d82dadd083" (UID: "402b6055-b012-440a-a5b8-38d82dadd083"). InnerVolumeSpecName "kube-api-access-spk8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:55.392959 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.392919    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2db3c79-863f-4c15-b6f6-a0eb7300d8e6-kube-api-access-z6kt4\" (UniqueName: \"kubernetes.io/projected/a2db3c79-863f-4c15-b6f6-a0eb7300d8e6-kube-api-access-z6kt4\") pod \"a2db3c79-863f-4c15-b6f6-a0eb7300d8e6\" (UID: \"a2db3c79-863f-4c15-b6f6-a0eb7300d8e6\") "
Oct 30 01:07:55.393177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.393141    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2db3c79-863f-4c15-b6f6-a0eb7300d8e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2db3c79-863f-4c15-b6f6-a0eb7300d8e6-plugins-config-volume\") pod \"a2db3c79-863f-4c15-b6f6-a0eb7300d8e6\" (UID: \"a2db3c79-863f-4c15-b6f6-a0eb7300d8e6\") "
Oct 30 01:07:55.393511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.393488    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2db3c79-863f-4c15-b6f6-a0eb7300d8e6-plugins-config-volume" pod "a2db3c79-863f-4c15-b6f6-a0eb7300d8e6" (UID: "a2db3c79-863f-4c15-b6f6-a0eb7300d8e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:07:55.393729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.393711    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spk8s\" (UniqueName: \"kubernetes.io/projected/402b6055-b012-440a-a5b8-38d82dadd083-kube-api-access-spk8s\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:55.393787 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.393734    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2db3c79-863f-4c15-b6f6-a0eb7300d8e6-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:55.394976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.394955    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2db3c79-863f-4c15-b6f6-a0eb7300d8e6-kube-api-access-z6kt4" pod "a2db3c79-863f-4c15-b6f6-a0eb7300d8e6" (UID: "a2db3c79-863f-4c15-b6f6-a0eb7300d8e6"). InnerVolumeSpecName "kube-api-access-z6kt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:07:55.424698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.424609    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74344f02-85de-4ea4-9da0-7580efea93ca" path="/var/lib/kubelet/pods/74344f02-85de-4ea4-9da0-7580efea93ca/volumes"
Oct 30 01:07:55.462884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.462857    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"
Oct 30 01:07:55.494986 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.494955    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6kt4\" (UniqueName: \"kubernetes.io/projected/a2db3c79-863f-4c15-b6f6-a0eb7300d8e6-kube-api-access-z6kt4\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:07:55.548112 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.548081    4425 generic.go:356] "Generic (PLEG): container finished" podID="402b6055-b012-440a-a5b8-38d82dadd083" containerID="1f2b92376ca8d17a65513539456501d838f64d6176d7617a6231a3615e81167f" exitCode=0
Oct 30 01:07:55.548294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.548135    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"
Oct 30 01:07:55.548294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.548181    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9" event={"ID":"402b6055-b012-440a-a5b8-38d82dadd083","Type":"ContainerDied","Data":"1f2b92376ca8d17a65513539456501d838f64d6176d7617a6231a3615e81167f"}
Oct 30 01:07:55.548294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.548247    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9" event={"ID":"402b6055-b012-440a-a5b8-38d82dadd083","Type":"ContainerDied","Data":"da9fa53a81764ec95d0610c99a14174c66a7e2ae4471455b60084e847a47e0c8"}
Oct 30 01:07:55.548294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.548283    4425 scope.go:122] "RemoveContainer" containerID="1f2b92376ca8d17a65513539456501d838f64d6176d7617a6231a3615e81167f"
Oct 30 01:07:55.549948 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.549917    4425 generic.go:356] "Generic (PLEG): container finished" podID="a2db3c79-863f-4c15-b6f6-a0eb7300d8e6" containerID="d3f6332ac589eb1968faedecc66e7f5e55842abb558c5442f635144c426511c9" exitCode=0
Oct 30 01:07:55.550057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.549974    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr" event={"ID":"a2db3c79-863f-4c15-b6f6-a0eb7300d8e6","Type":"ContainerDied","Data":"d3f6332ac589eb1968faedecc66e7f5e55842abb558c5442f635144c426511c9"}
Oct 30 01:07:55.550057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.549993    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"
Oct 30 01:07:55.550057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.550017    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr" event={"ID":"a2db3c79-863f-4c15-b6f6-a0eb7300d8e6","Type":"ContainerDied","Data":"48124a877e8ee59dcae72397e19f06f3e44492d6cd24b7c4d0a67de0638ef2d6"}
Oct 30 01:07:55.559772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.559742    4425 scope.go:122] "RemoveContainer" containerID="1f2b92376ca8d17a65513539456501d838f64d6176d7617a6231a3615e81167f"
Oct 30 01:07:55.560070 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:07:55.560052    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f2b92376ca8d17a65513539456501d838f64d6176d7617a6231a3615e81167f\": container with ID starting with 1f2b92376ca8d17a65513539456501d838f64d6176d7617a6231a3615e81167f not found: ID does not exist" containerID="1f2b92376ca8d17a65513539456501d838f64d6176d7617a6231a3615e81167f"
Oct 30 01:07:55.560129 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.560078    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f2b92376ca8d17a65513539456501d838f64d6176d7617a6231a3615e81167f"} err="failed to get container status \"1f2b92376ca8d17a65513539456501d838f64d6176d7617a6231a3615e81167f\": rpc error: code = NotFound desc = could not find container \"1f2b92376ca8d17a65513539456501d838f64d6176d7617a6231a3615e81167f\": container with ID starting with 1f2b92376ca8d17a65513539456501d838f64d6176d7617a6231a3615e81167f not found: ID does not exist"
Oct 30 01:07:55.560129 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.560095    4425 scope.go:122] "RemoveContainer" containerID="d3f6332ac589eb1968faedecc66e7f5e55842abb558c5442f635144c426511c9"
Oct 30 01:07:55.572737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.572703    4425 scope.go:122] "RemoveContainer" containerID="d3f6332ac589eb1968faedecc66e7f5e55842abb558c5442f635144c426511c9"
Oct 30 01:07:55.573044 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:07:55.573025    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f6332ac589eb1968faedecc66e7f5e55842abb558c5442f635144c426511c9\": container with ID starting with d3f6332ac589eb1968faedecc66e7f5e55842abb558c5442f635144c426511c9 not found: ID does not exist" containerID="d3f6332ac589eb1968faedecc66e7f5e55842abb558c5442f635144c426511c9"
Oct 30 01:07:55.573119 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.573054    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f6332ac589eb1968faedecc66e7f5e55842abb558c5442f635144c426511c9"} err="failed to get container status \"d3f6332ac589eb1968faedecc66e7f5e55842abb558c5442f635144c426511c9\": rpc error: code = NotFound desc = could not find container \"d3f6332ac589eb1968faedecc66e7f5e55842abb558c5442f635144c426511c9\": container with ID starting with d3f6332ac589eb1968faedecc66e7f5e55842abb558c5442f635144c426511c9 not found: ID does not exist"
Oct 30 01:07:55.600385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.600364    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"]
Oct 30 01:07:55.604834 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.604815    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f05249-664d6f5948-dl5h9"]
Oct 30 01:07:55.635135 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.635094    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"]
Oct 30 01:07:55.639384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:55.639359    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-56895899-h7djr"]
Oct 30 01:07:57.423382 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:57.423347    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="402b6055-b012-440a-a5b8-38d82dadd083" path="/var/lib/kubelet/pods/402b6055-b012-440a-a5b8-38d82dadd083/volumes"
Oct 30 01:07:57.423805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:57.423775    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2db3c79-863f-4c15-b6f6-a0eb7300d8e6" path="/var/lib/kubelet/pods/a2db3c79-863f-4c15-b6f6-a0eb7300d8e6/volumes"
Oct 30 01:07:58.495877 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:58.495847    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"
Oct 30 01:07:58.496382 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:58.495919    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"
Oct 30 01:07:58.870595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:58.870561    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"]
Oct 30 01:07:58.871106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:58.871093    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a2db3c79-863f-4c15-b6f6-a0eb7300d8e6" containerName="payload-pre-processing"
Oct 30 01:07:58.871202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:58.871108    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2db3c79-863f-4c15-b6f6-a0eb7300d8e6" containerName="payload-pre-processing"
Oct 30 01:07:58.871202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:58.871171    4425 cpu_manager.go:414] "containerMap: removing container" podUID="402b6055-b012-440a-a5b8-38d82dadd083" containerName="payload-processing"
Oct 30 01:07:58.871202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:58.871176    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="402b6055-b012-440a-a5b8-38d82dadd083" containerName="payload-processing"
Oct 30 01:07:58.871306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:58.871258    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="402b6055-b012-440a-a5b8-38d82dadd083" containerName="payload-processing"
Oct 30 01:07:58.871306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:58.871269    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2db3c79-863f-4c15-b6f6-a0eb7300d8e6" containerName="payload-pre-processing"
Oct 30 01:07:58.876258 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:58.876230    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"
Oct 30 01:07:58.890791 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:58.890752    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:07:58.890968 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:58.890822    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:07:58.900480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:58.900448    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:07:58.900683 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:58.900495    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:07:58.904299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:58.904267    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"]
Oct 30 01:07:59.043588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:59.043543    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fe6e26-43d4-44c7-a519-85a5c786005f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz\" (UID: \"f7fe6e26-43d4-44c7-a519-85a5c786005f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"
Oct 30 01:07:59.043800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:59.043611    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr997\" (UniqueName: \"kubernetes.io/projected/f7fe6e26-43d4-44c7-a519-85a5c786005f-kube-api-access-pr997\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz\" (UID: \"f7fe6e26-43d4-44c7-a519-85a5c786005f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"
Oct 30 01:07:59.043800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:59.043706    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fe6e26-43d4-44c7-a519-85a5c786005f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz\" (UID: \"f7fe6e26-43d4-44c7-a519-85a5c786005f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"
Oct 30 01:07:59.145568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:59.145460    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fe6e26-43d4-44c7-a519-85a5c786005f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz\" (UID: \"f7fe6e26-43d4-44c7-a519-85a5c786005f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"
Oct 30 01:07:59.145568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:59.145521    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr997\" (UniqueName: \"kubernetes.io/projected/f7fe6e26-43d4-44c7-a519-85a5c786005f-kube-api-access-pr997\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz\" (UID: \"f7fe6e26-43d4-44c7-a519-85a5c786005f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"
Oct 30 01:07:59.145568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:59.145560    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fe6e26-43d4-44c7-a519-85a5c786005f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz\" (UID: \"f7fe6e26-43d4-44c7-a519-85a5c786005f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"
Oct 30 01:07:59.146202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:59.146151    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fe6e26-43d4-44c7-a519-85a5c786005f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz\" (UID: \"f7fe6e26-43d4-44c7-a519-85a5c786005f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"
Oct 30 01:07:59.148144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:59.148118    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fe6e26-43d4-44c7-a519-85a5c786005f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz\" (UID: \"f7fe6e26-43d4-44c7-a519-85a5c786005f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"
Oct 30 01:07:59.166996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:59.166956    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr997\" (UniqueName: \"kubernetes.io/projected/f7fe6e26-43d4-44c7-a519-85a5c786005f-kube-api-access-pr997\") pod \"maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz\" (UID: \"f7fe6e26-43d4-44c7-a519-85a5c786005f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"
Oct 30 01:07:59.190923 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:59.190887    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"
Oct 30 01:07:59.533610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:59.533582    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"]
Oct 30 01:07:59.535373 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:07:59.535329    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fe6e26_43d4_44c7_a519_85a5c786005f.slice/crio-20717df044484931269336b7851bfae7b5dd37724cd1462d5e319d65608ef032 WatchSource:0}: Error finding container 20717df044484931269336b7851bfae7b5dd37724cd1462d5e319d65608ef032: Status 404 returned error can't find the container with id 20717df044484931269336b7851bfae7b5dd37724cd1462d5e319d65608ef032
Oct 30 01:07:59.583860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:07:59.583822    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz" event={"ID":"f7fe6e26-43d4-44c7-a519-85a5c786005f","Type":"ContainerStarted","Data":"20717df044484931269336b7851bfae7b5dd37724cd1462d5e319d65608ef032"}
Oct 30 01:08:00.593265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:00.593226    4425 generic.go:356] "Generic (PLEG): container finished" podID="f7fe6e26-43d4-44c7-a519-85a5c786005f" containerID="d3fe407865ed89aa8928612918129d46a59494500d30e2a7bfc754ef489ff20d" exitCode=0
Oct 30 01:08:00.593694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:00.593314    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz" event={"ID":"f7fe6e26-43d4-44c7-a519-85a5c786005f","Type":"ContainerDied","Data":"d3fe407865ed89aa8928612918129d46a59494500d30e2a7bfc754ef489ff20d"}
Oct 30 01:08:01.766541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:01.766510    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"
Oct 30 01:08:01.782319 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:01.782283    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7fe6e26-43d4-44c7-a519-85a5c786005f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fe6e26-43d4-44c7-a519-85a5c786005f-maas-api-openshift-ca\") pod \"f7fe6e26-43d4-44c7-a519-85a5c786005f\" (UID: \"f7fe6e26-43d4-44c7-a519-85a5c786005f\") "
Oct 30 01:08:01.782479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:01.782343    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7fe6e26-43d4-44c7-a519-85a5c786005f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fe6e26-43d4-44c7-a519-85a5c786005f-maas-api-xks-ca\") pod \"f7fe6e26-43d4-44c7-a519-85a5c786005f\" (UID: \"f7fe6e26-43d4-44c7-a519-85a5c786005f\") "
Oct 30 01:08:01.782479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:01.782402    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7fe6e26-43d4-44c7-a519-85a5c786005f-kube-api-access-pr997\" (UniqueName: \"kubernetes.io/projected/f7fe6e26-43d4-44c7-a519-85a5c786005f-kube-api-access-pr997\") pod \"f7fe6e26-43d4-44c7-a519-85a5c786005f\" (UID: \"f7fe6e26-43d4-44c7-a519-85a5c786005f\") "
Oct 30 01:08:01.782761 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:01.782733    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7fe6e26-43d4-44c7-a519-85a5c786005f-maas-api-openshift-ca" pod "f7fe6e26-43d4-44c7-a519-85a5c786005f" (UID: "f7fe6e26-43d4-44c7-a519-85a5c786005f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:01.782877 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:01.782862    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fe6e26-43d4-44c7-a519-85a5c786005f-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:08:01.784610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:01.784583    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7fe6e26-43d4-44c7-a519-85a5c786005f-maas-api-xks-ca" pod "f7fe6e26-43d4-44c7-a519-85a5c786005f" (UID: "f7fe6e26-43d4-44c7-a519-85a5c786005f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:01.784723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:01.784621    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7fe6e26-43d4-44c7-a519-85a5c786005f-kube-api-access-pr997" pod "f7fe6e26-43d4-44c7-a519-85a5c786005f" (UID: "f7fe6e26-43d4-44c7-a519-85a5c786005f"). InnerVolumeSpecName "kube-api-access-pr997". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:01.884596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:01.884483    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr997\" (UniqueName: \"kubernetes.io/projected/f7fe6e26-43d4-44c7-a519-85a5c786005f-kube-api-access-pr997\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:08:01.884596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:01.884525    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fe6e26-43d4-44c7-a519-85a5c786005f-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:08:02.611624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:02.611592    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"
Oct 30 01:08:02.611813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:02.611587    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz" event={"ID":"f7fe6e26-43d4-44c7-a519-85a5c786005f","Type":"ContainerDied","Data":"20717df044484931269336b7851bfae7b5dd37724cd1462d5e319d65608ef032"}
Oct 30 01:08:02.611813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:02.611732    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20717df044484931269336b7851bfae7b5dd37724cd1462d5e319d65608ef032"
Oct 30 01:08:07.557395 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:08:07.557348    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:08:15.076390 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.076345    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"]
Oct 30 01:08:15.077240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.077215    4425 cpu_manager.go:414] "containerMap: removing container" podUID="f7fe6e26-43d4-44c7-a519-85a5c786005f" containerName="revoke-keys"
Oct 30 01:08:15.077375 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.077242    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7fe6e26-43d4-44c7-a519-85a5c786005f" containerName="revoke-keys"
Oct 30 01:08:15.077491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.077476    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7fe6e26-43d4-44c7-a519-85a5c786005f" containerName="revoke-keys"
Oct 30 01:08:15.082366 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.082337    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"
Oct 30 01:08:15.100392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.100360    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f05249\""
Oct 30 01:08:15.113279 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.113227    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f05249-dockercfg-mh9ck\""
Oct 30 01:08:15.117241 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.117209    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"]
Oct 30 01:08:15.167561 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.167513    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4311a891-a923-436f-8f5c-086dbafcaf8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk\" (UID: \"4311a891-a923-436f-8f5c-086dbafcaf8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"
Oct 30 01:08:15.167794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.167657    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6zcc\" (UniqueName: \"kubernetes.io/projected/4311a891-a923-436f-8f5c-086dbafcaf8c-kube-api-access-x6zcc\") pod \"payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk\" (UID: \"4311a891-a923-436f-8f5c-086dbafcaf8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"
Oct 30 01:08:15.242400 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.242360    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"]
Oct 30 01:08:15.252444 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.252407    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"]
Oct 30 01:08:15.253469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.252518    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"
Oct 30 01:08:15.269586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.269552    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4311a891-a923-436f-8f5c-086dbafcaf8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk\" (UID: \"4311a891-a923-436f-8f5c-086dbafcaf8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"
Oct 30 01:08:15.269930 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.269905    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6zcc\" (UniqueName: \"kubernetes.io/projected/4311a891-a923-436f-8f5c-086dbafcaf8c-kube-api-access-x6zcc\") pod \"payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk\" (UID: \"4311a891-a923-436f-8f5c-086dbafcaf8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"
Oct 30 01:08:15.270495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.270465    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4311a891-a923-436f-8f5c-086dbafcaf8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk\" (UID: \"4311a891-a923-436f-8f5c-086dbafcaf8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"
Oct 30 01:08:15.299361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.299321    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6zcc\" (UniqueName: \"kubernetes.io/projected/4311a891-a923-436f-8f5c-086dbafcaf8c-kube-api-access-x6zcc\") pod \"payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk\" (UID: \"4311a891-a923-436f-8f5c-086dbafcaf8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"
Oct 30 01:08:15.371897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.371794    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4f27\" (UniqueName: \"kubernetes.io/projected/336deb2f-912c-4e44-bbf2-356952a1cebf-kube-api-access-w4f27\") pod \"payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt\" (UID: \"336deb2f-912c-4e44-bbf2-356952a1cebf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"
Oct 30 01:08:15.372092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.371984    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336deb2f-912c-4e44-bbf2-356952a1cebf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt\" (UID: \"336deb2f-912c-4e44-bbf2-356952a1cebf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"
Oct 30 01:08:15.397737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.397688    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"
Oct 30 01:08:15.473471 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.473415    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336deb2f-912c-4e44-bbf2-356952a1cebf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt\" (UID: \"336deb2f-912c-4e44-bbf2-356952a1cebf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"
Oct 30 01:08:15.473830 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.473798    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4f27\" (UniqueName: \"kubernetes.io/projected/336deb2f-912c-4e44-bbf2-356952a1cebf-kube-api-access-w4f27\") pod \"payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt\" (UID: \"336deb2f-912c-4e44-bbf2-356952a1cebf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"
Oct 30 01:08:15.475445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.475390    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336deb2f-912c-4e44-bbf2-356952a1cebf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt\" (UID: \"336deb2f-912c-4e44-bbf2-356952a1cebf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"
Oct 30 01:08:15.495934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.495271    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4f27\" (UniqueName: \"kubernetes.io/projected/336deb2f-912c-4e44-bbf2-356952a1cebf-kube-api-access-w4f27\") pod \"payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt\" (UID: \"336deb2f-912c-4e44-bbf2-356952a1cebf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"
Oct 30 01:08:15.590883 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.590839    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"
Oct 30 01:08:15.714747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.714705    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"]
Oct 30 01:08:15.716452 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:08:15.716420    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4311a891_a923_436f_8f5c_086dbafcaf8c.slice/crio-d95206484846bd351920aa5ec24d865b5003e75256d1a79a2e0802535ffa81d9 WatchSource:0}: Error finding container d95206484846bd351920aa5ec24d865b5003e75256d1a79a2e0802535ffa81d9: Status 404 returned error can't find the container with id d95206484846bd351920aa5ec24d865b5003e75256d1a79a2e0802535ffa81d9
Oct 30 01:08:15.908730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:15.908696    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"]
Oct 30 01:08:15.909867 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:08:15.909827    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336deb2f_912c_4e44_bbf2_356952a1cebf.slice/crio-3719fb4f09ee284fada7823948f359f783885c1488020f0d6d40143c18715422 WatchSource:0}: Error finding container 3719fb4f09ee284fada7823948f359f783885c1488020f0d6d40143c18715422: Status 404 returned error can't find the container with id 3719fb4f09ee284fada7823948f359f783885c1488020f0d6d40143c18715422
Oct 30 01:08:16.718918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:16.718879    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt" event={"ID":"336deb2f-912c-4e44-bbf2-356952a1cebf","Type":"ContainerStarted","Data":"7fd88ab3f029e1ea805446ed6ea4c913cae00f6513a6cdb39af6066a0e476137"}
Oct 30 01:08:16.719432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:16.718953    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"
Oct 30 01:08:16.719432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:16.718978    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt" event={"ID":"336deb2f-912c-4e44-bbf2-356952a1cebf","Type":"ContainerStarted","Data":"3719fb4f09ee284fada7823948f359f783885c1488020f0d6d40143c18715422"}
Oct 30 01:08:16.720892 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:16.720864    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk" event={"ID":"4311a891-a923-436f-8f5c-086dbafcaf8c","Type":"ContainerStarted","Data":"24aebdb0b5390ef11efb755f3c8f376838176a57bb198e06fa494af6d5a51f2c"}
Oct 30 01:08:16.721042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:16.720910    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk" event={"ID":"4311a891-a923-436f-8f5c-086dbafcaf8c","Type":"ContainerStarted","Data":"d95206484846bd351920aa5ec24d865b5003e75256d1a79a2e0802535ffa81d9"}
Oct 30 01:08:16.721042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:16.720961    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"
Oct 30 01:08:16.755396 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:16.755340    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt" podStartSLOduration=1.755324479 podStartE2EDuration="1.755324479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:16.754320672 +0000 UTC m=+1605.503975981" watchObservedRunningTime="2026-10-30 01:08:16.755324479 +0000 UTC m=+1605.504979787"
Oct 30 01:08:16.794070 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:16.794010    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk" podStartSLOduration=1.793994917 podStartE2EDuration="1.793994917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:16.790070683 +0000 UTC m=+1605.539726003" watchObservedRunningTime="2026-10-30 01:08:16.793994917 +0000 UTC m=+1605.543650225"
Oct 30 01:08:23.175754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.175712    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"]
Oct 30 01:08:23.176267 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.176002    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52" podUID="1190e6ca-0f37-45a6-a01d-662b189a88b9" containerName="payload-pre-processing" containerID="cri-o://7adbaeeba36370027531147db7fb97642e26cbb637019e55458747b9532d45f5" gracePeriod=30
Oct 30 01:08:23.512622 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.509265    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"
Oct 30 01:08:23.597603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.596785    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1190e6ca-0f37-45a6-a01d-662b189a88b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1190e6ca-0f37-45a6-a01d-662b189a88b9-plugins-config-volume\") pod \"1190e6ca-0f37-45a6-a01d-662b189a88b9\" (UID: \"1190e6ca-0f37-45a6-a01d-662b189a88b9\") "
Oct 30 01:08:23.597603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.596974    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1190e6ca-0f37-45a6-a01d-662b189a88b9-kube-api-access-hxsrh\" (UniqueName: \"kubernetes.io/projected/1190e6ca-0f37-45a6-a01d-662b189a88b9-kube-api-access-hxsrh\") pod \"1190e6ca-0f37-45a6-a01d-662b189a88b9\" (UID: \"1190e6ca-0f37-45a6-a01d-662b189a88b9\") "
Oct 30 01:08:23.597840 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.597650    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1190e6ca-0f37-45a6-a01d-662b189a88b9-plugins-config-volume" pod "1190e6ca-0f37-45a6-a01d-662b189a88b9" (UID: "1190e6ca-0f37-45a6-a01d-662b189a88b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:23.600106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.600056    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1190e6ca-0f37-45a6-a01d-662b189a88b9-kube-api-access-hxsrh" pod "1190e6ca-0f37-45a6-a01d-662b189a88b9" (UID: "1190e6ca-0f37-45a6-a01d-662b189a88b9"). InnerVolumeSpecName "kube-api-access-hxsrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:23.699077 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.699030    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxsrh\" (UniqueName: \"kubernetes.io/projected/1190e6ca-0f37-45a6-a01d-662b189a88b9-kube-api-access-hxsrh\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:08:23.699077 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.699068    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1190e6ca-0f37-45a6-a01d-662b189a88b9-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:08:23.773907 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.773863    4425 generic.go:356] "Generic (PLEG): container finished" podID="1190e6ca-0f37-45a6-a01d-662b189a88b9" containerID="7adbaeeba36370027531147db7fb97642e26cbb637019e55458747b9532d45f5" exitCode=0
Oct 30 01:08:23.774106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.773917    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"
Oct 30 01:08:23.774106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.773948    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52" event={"ID":"1190e6ca-0f37-45a6-a01d-662b189a88b9","Type":"ContainerDied","Data":"7adbaeeba36370027531147db7fb97642e26cbb637019e55458747b9532d45f5"}
Oct 30 01:08:23.774106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.774008    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52" event={"ID":"1190e6ca-0f37-45a6-a01d-662b189a88b9","Type":"ContainerDied","Data":"f223f3495123b470da9165fa0042f05a1be4f2810e28ac2ff7a8b23db07bb682"}
Oct 30 01:08:23.774106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.774034    4425 scope.go:122] "RemoveContainer" containerID="7adbaeeba36370027531147db7fb97642e26cbb637019e55458747b9532d45f5"
Oct 30 01:08:23.785461 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.785430    4425 scope.go:122] "RemoveContainer" containerID="7adbaeeba36370027531147db7fb97642e26cbb637019e55458747b9532d45f5"
Oct 30 01:08:23.785838 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:08:23.785798    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7adbaeeba36370027531147db7fb97642e26cbb637019e55458747b9532d45f5\": container with ID starting with 7adbaeeba36370027531147db7fb97642e26cbb637019e55458747b9532d45f5 not found: ID does not exist" containerID="7adbaeeba36370027531147db7fb97642e26cbb637019e55458747b9532d45f5"
Oct 30 01:08:23.785917 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.785838    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7adbaeeba36370027531147db7fb97642e26cbb637019e55458747b9532d45f5"} err="failed to get container status \"7adbaeeba36370027531147db7fb97642e26cbb637019e55458747b9532d45f5\": rpc error: code = NotFound desc = could not find container \"7adbaeeba36370027531147db7fb97642e26cbb637019e55458747b9532d45f5\": container with ID starting with 7adbaeeba36370027531147db7fb97642e26cbb637019e55458747b9532d45f5 not found: ID does not exist"
Oct 30 01:08:23.824917 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.824875    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"]
Oct 30 01:08:23.833507 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.833471    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-11fa15-7b6576d547-hgq52"]
Oct 30 01:08:23.872875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.872839    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"]
Oct 30 01:08:23.877256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:23.877208    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-11fa15-v47jz"]
Oct 30 01:08:25.423423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:25.423390    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1190e6ca-0f37-45a6-a01d-662b189a88b9" path="/var/lib/kubelet/pods/1190e6ca-0f37-45a6-a01d-662b189a88b9/volumes"
Oct 30 01:08:25.423910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:25.423895    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7fe6e26-43d4-44c7-a519-85a5c786005f" path="/var/lib/kubelet/pods/f7fe6e26-43d4-44c7-a519-85a5c786005f/volumes"
Oct 30 01:08:27.731075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:27.731033    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"
Oct 30 01:08:27.731615 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:27.731434    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"
Oct 30 01:08:32.310481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:32.310444    4425 scope.go:122] "RemoveContainer" containerID="fd289de2cf8c2cbc92e2a09aab996ceb5c0cc80bbd0bb491658d13c4b50a01e1"
Oct 30 01:08:32.321339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:32.321310    4425 scope.go:122] "RemoveContainer" containerID="3577ce76554f50e2ddf897fe0053971732505b95d0087fe5e81257859f4cdb40"
Oct 30 01:08:32.333733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:32.333709    4425 scope.go:122] "RemoveContainer" containerID="31d7ff2cd851ad05da1b708d85698bcc168dd8086fedacb415cf73bc0108c412"
Oct 30 01:08:32.355426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:32.352292    4425 scope.go:122] "RemoveContainer" containerID="2dbf7d545fda21d84d2363e599a99b89fc8a5d5907d4cb1baafaba83d5a84b95"
Oct 30 01:08:37.565191 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:08:37.565120    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:08:38.462309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:38.462271    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:08:39.918899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:39.918853    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"]
Oct 30 01:08:39.919800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:39.919774    4425 cpu_manager.go:414] "containerMap: removing container" podUID="1190e6ca-0f37-45a6-a01d-662b189a88b9" containerName="payload-pre-processing"
Oct 30 01:08:39.919845 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:39.919803    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1190e6ca-0f37-45a6-a01d-662b189a88b9" containerName="payload-pre-processing"
Oct 30 01:08:39.919970 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:39.919958    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="1190e6ca-0f37-45a6-a01d-662b189a88b9" containerName="payload-pre-processing"
Oct 30 01:08:39.926476 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:39.926445    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"
Oct 30 01:08:39.953573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:39.953528    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b232802-7371-4e20-a226-03ae05e03441-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh\" (UID: \"9b232802-7371-4e20-a226-03ae05e03441\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"
Oct 30 01:08:39.953858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:39.953705    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xps29\" (UniqueName: \"kubernetes.io/projected/9b232802-7371-4e20-a226-03ae05e03441-kube-api-access-xps29\") pod \"payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh\" (UID: \"9b232802-7371-4e20-a226-03ae05e03441\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"
Oct 30 01:08:39.957272 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:39.957232    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"]
Oct 30 01:08:40.025181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:40.024334    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d3c1fa\""
Oct 30 01:08:40.041180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:40.039506    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d3c1fa-dockercfg-4vw8j\""
Oct 30 01:08:40.061187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:40.057933    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b232802-7371-4e20-a226-03ae05e03441-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh\" (UID: \"9b232802-7371-4e20-a226-03ae05e03441\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"
Oct 30 01:08:40.061187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:40.058053    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xps29\" (UniqueName: \"kubernetes.io/projected/9b232802-7371-4e20-a226-03ae05e03441-kube-api-access-xps29\") pod \"payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh\" (UID: \"9b232802-7371-4e20-a226-03ae05e03441\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"
Oct 30 01:08:40.061187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:40.059079    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b232802-7371-4e20-a226-03ae05e03441-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh\" (UID: \"9b232802-7371-4e20-a226-03ae05e03441\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"
Oct 30 01:08:40.081301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:40.081260    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xps29\" (UniqueName: \"kubernetes.io/projected/9b232802-7371-4e20-a226-03ae05e03441-kube-api-access-xps29\") pod \"payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh\" (UID: \"9b232802-7371-4e20-a226-03ae05e03441\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"
Oct 30 01:08:40.244116 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:40.244005    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"
Oct 30 01:08:40.559734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:40.559702    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"]
Oct 30 01:08:40.562700 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:08:40.562670    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b232802_7371_4e20_a226_03ae05e03441.slice/crio-042a04b1e927d4414b3188ba37d53c2f0e8722f8e4063545be6c18376482e71e WatchSource:0}: Error finding container 042a04b1e927d4414b3188ba37d53c2f0e8722f8e4063545be6c18376482e71e: Status 404 returned error can't find the container with id 042a04b1e927d4414b3188ba37d53c2f0e8722f8e4063545be6c18376482e71e
Oct 30 01:08:40.917509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:40.917398    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh" event={"ID":"9b232802-7371-4e20-a226-03ae05e03441","Type":"ContainerStarted","Data":"4a29ffd94216b924b02236f930b6a49dff9bb9fee87abbed748e709d4bd4b4fc"}
Oct 30 01:08:40.917509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:40.917501    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh" event={"ID":"9b232802-7371-4e20-a226-03ae05e03441","Type":"ContainerStarted","Data":"042a04b1e927d4414b3188ba37d53c2f0e8722f8e4063545be6c18376482e71e"}
Oct 30 01:08:40.917751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:40.917696    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"
Oct 30 01:08:40.985477 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:40.985422    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh" podStartSLOduration=1.985402712 podStartE2EDuration="1.985402712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:40.98178571 +0000 UTC m=+1629.731441018" watchObservedRunningTime="2026-10-30 01:08:40.985402712 +0000 UTC m=+1629.735058021"
Oct 30 01:08:41.579812 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:41.579766    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"]
Oct 30 01:08:41.584338 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:41.584303    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"
Oct 30 01:08:41.603782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:41.603741    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:08:41.613507 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:41.613472    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:08:41.613738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:41.613530    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:08:41.613738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:41.613585    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:08:41.617983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:41.617948    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"]
Oct 30 01:08:41.682340 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:41.681851    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683618f7-1ccb-4c55-a792-efff34d5adec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9\" (UID: \"683618f7-1ccb-4c55-a792-efff34d5adec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"
Oct 30 01:08:41.682340 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:41.682021    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683618f7-1ccb-4c55-a792-efff34d5adec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9\" (UID: \"683618f7-1ccb-4c55-a792-efff34d5adec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"
Oct 30 01:08:41.682625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:41.682348    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s55fz\" (UniqueName: \"kubernetes.io/projected/683618f7-1ccb-4c55-a792-efff34d5adec-kube-api-access-s55fz\") pod \"maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9\" (UID: \"683618f7-1ccb-4c55-a792-efff34d5adec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"
Oct 30 01:08:41.784059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:41.784017    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683618f7-1ccb-4c55-a792-efff34d5adec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9\" (UID: \"683618f7-1ccb-4c55-a792-efff34d5adec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"
Oct 30 01:08:41.784289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:41.784096    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683618f7-1ccb-4c55-a792-efff34d5adec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9\" (UID: \"683618f7-1ccb-4c55-a792-efff34d5adec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"
Oct 30 01:08:41.784289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:41.784198    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s55fz\" (UniqueName: \"kubernetes.io/projected/683618f7-1ccb-4c55-a792-efff34d5adec-kube-api-access-s55fz\") pod \"maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9\" (UID: \"683618f7-1ccb-4c55-a792-efff34d5adec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"
Oct 30 01:08:41.784825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:41.784795    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683618f7-1ccb-4c55-a792-efff34d5adec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9\" (UID: \"683618f7-1ccb-4c55-a792-efff34d5adec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"
Oct 30 01:08:41.786690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:41.786665    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683618f7-1ccb-4c55-a792-efff34d5adec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9\" (UID: \"683618f7-1ccb-4c55-a792-efff34d5adec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"
Oct 30 01:08:41.828530 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:41.828494    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s55fz\" (UniqueName: \"kubernetes.io/projected/683618f7-1ccb-4c55-a792-efff34d5adec-kube-api-access-s55fz\") pod \"maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9\" (UID: \"683618f7-1ccb-4c55-a792-efff34d5adec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"
Oct 30 01:08:41.899652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:41.899553    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"
Oct 30 01:08:42.276239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:42.276204    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"]
Oct 30 01:08:42.281209 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:08:42.281164    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683618f7_1ccb_4c55_a792_efff34d5adec.slice/crio-b9eac14a8a5f01cb479a394add5fbd1350c8d2379976ef66b58a0b0c85195ac3 WatchSource:0}: Error finding container b9eac14a8a5f01cb479a394add5fbd1350c8d2379976ef66b58a0b0c85195ac3: Status 404 returned error can't find the container with id b9eac14a8a5f01cb479a394add5fbd1350c8d2379976ef66b58a0b0c85195ac3
Oct 30 01:08:42.940445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:42.940337    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9" event={"ID":"683618f7-1ccb-4c55-a792-efff34d5adec","Type":"ContainerStarted","Data":"598e1939137d46ce745f705fcade7fffe29b83ea1f39d97a141e1159fa37831b"}
Oct 30 01:08:42.940445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:42.940399    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9" event={"ID":"683618f7-1ccb-4c55-a792-efff34d5adec","Type":"ContainerStarted","Data":"b9eac14a8a5f01cb479a394add5fbd1350c8d2379976ef66b58a0b0c85195ac3"}
Oct 30 01:08:42.981509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:42.981447    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9" podStartSLOduration=1.981428732 podStartE2EDuration="1.981428732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:42.978284098 +0000 UTC m=+1631.727939407" watchObservedRunningTime="2026-10-30 01:08:42.981428732 +0000 UTC m=+1631.731084040"
Oct 30 01:08:43.948931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:43.948894    4425 generic.go:356] "Generic (PLEG): container finished" podID="683618f7-1ccb-4c55-a792-efff34d5adec" containerID="598e1939137d46ce745f705fcade7fffe29b83ea1f39d97a141e1159fa37831b" exitCode=0
Oct 30 01:08:43.949377 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:43.948982    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9" event={"ID":"683618f7-1ccb-4c55-a792-efff34d5adec","Type":"ContainerDied","Data":"598e1939137d46ce745f705fcade7fffe29b83ea1f39d97a141e1159fa37831b"}
Oct 30 01:08:44.765643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:44.765596    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"]
Oct 30 01:08:44.799047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:44.799011    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"]
Oct 30 01:08:45.163675 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:45.163633    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"
Oct 30 01:08:45.233647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:45.233605    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/683618f7-1ccb-4c55-a792-efff34d5adec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683618f7-1ccb-4c55-a792-efff34d5adec-maas-api-openshift-ca\") pod \"683618f7-1ccb-4c55-a792-efff34d5adec\" (UID: \"683618f7-1ccb-4c55-a792-efff34d5adec\") "
Oct 30 01:08:45.233863 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:45.233712    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/683618f7-1ccb-4c55-a792-efff34d5adec-kube-api-access-s55fz\" (UniqueName: \"kubernetes.io/projected/683618f7-1ccb-4c55-a792-efff34d5adec-kube-api-access-s55fz\") pod \"683618f7-1ccb-4c55-a792-efff34d5adec\" (UID: \"683618f7-1ccb-4c55-a792-efff34d5adec\") "
Oct 30 01:08:45.233863 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:45.233837    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/683618f7-1ccb-4c55-a792-efff34d5adec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683618f7-1ccb-4c55-a792-efff34d5adec-maas-api-xks-ca\") pod \"683618f7-1ccb-4c55-a792-efff34d5adec\" (UID: \"683618f7-1ccb-4c55-a792-efff34d5adec\") "
Oct 30 01:08:45.234097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:45.234066    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/683618f7-1ccb-4c55-a792-efff34d5adec-maas-api-openshift-ca" pod "683618f7-1ccb-4c55-a792-efff34d5adec" (UID: "683618f7-1ccb-4c55-a792-efff34d5adec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:45.234553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:45.234522    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683618f7-1ccb-4c55-a792-efff34d5adec-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:08:45.236961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:45.236917    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/683618f7-1ccb-4c55-a792-efff34d5adec-maas-api-xks-ca" pod "683618f7-1ccb-4c55-a792-efff34d5adec" (UID: "683618f7-1ccb-4c55-a792-efff34d5adec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:45.238341 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:45.238310    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683618f7-1ccb-4c55-a792-efff34d5adec-kube-api-access-s55fz" pod "683618f7-1ccb-4c55-a792-efff34d5adec" (UID: "683618f7-1ccb-4c55-a792-efff34d5adec"). InnerVolumeSpecName "kube-api-access-s55fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:45.335765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:45.335661    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s55fz\" (UniqueName: \"kubernetes.io/projected/683618f7-1ccb-4c55-a792-efff34d5adec-kube-api-access-s55fz\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:08:45.335765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:45.335701    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683618f7-1ccb-4c55-a792-efff34d5adec-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:08:45.968592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:45.968558    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"
Oct 30 01:08:45.968790 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:45.968560    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9" event={"ID":"683618f7-1ccb-4c55-a792-efff34d5adec","Type":"ContainerDied","Data":"b9eac14a8a5f01cb479a394add5fbd1350c8d2379976ef66b58a0b0c85195ac3"}
Oct 30 01:08:45.968790 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:45.968681    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9eac14a8a5f01cb479a394add5fbd1350c8d2379976ef66b58a0b0c85195ac3"
Oct 30 01:08:46.685075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.685033    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"]
Oct 30 01:08:46.685724 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.685708    4425 cpu_manager.go:414] "containerMap: removing container" podUID="683618f7-1ccb-4c55-a792-efff34d5adec" containerName="revoke-keys"
Oct 30 01:08:46.685771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.685726    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="683618f7-1ccb-4c55-a792-efff34d5adec" containerName="revoke-keys"
Oct 30 01:08:46.685866 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.685856    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="683618f7-1ccb-4c55-a792-efff34d5adec" containerName="revoke-keys"
Oct 30 01:08:46.689719 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.689695    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"
Oct 30 01:08:46.707018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.706983    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:08:46.707232 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.707096    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:08:46.717061 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.717025    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:08:46.717373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.717073    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:08:46.724465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.724429    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"]
Oct 30 01:08:46.759641 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.759597    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdz52\" (UniqueName: \"kubernetes.io/projected/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-kube-api-access-qdz52\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6\" (UID: \"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"
Oct 30 01:08:46.759909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.759859    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6\" (UID: \"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"
Oct 30 01:08:46.759909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.759904    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6\" (UID: \"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"
Oct 30 01:08:46.861770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.861733    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6\" (UID: \"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"
Oct 30 01:08:46.861770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.861781    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6\" (UID: \"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"
Oct 30 01:08:46.862072 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.861867    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdz52\" (UniqueName: \"kubernetes.io/projected/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-kube-api-access-qdz52\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6\" (UID: \"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"
Oct 30 01:08:46.862642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.862614    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6\" (UID: \"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"
Oct 30 01:08:46.865857 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.865831    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6\" (UID: \"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"
Oct 30 01:08:46.888292 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:46.888252    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdz52\" (UniqueName: \"kubernetes.io/projected/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-kube-api-access-qdz52\") pod \"maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6\" (UID: \"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"
Oct 30 01:08:47.007294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:47.007189    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"
Oct 30 01:08:47.350280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:47.350248    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"]
Oct 30 01:08:47.351756 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:08:47.351724    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4eeac73_9b9c_4dac_af5d_62d0d3dd00aa.slice/crio-0b406db2ff2b9a9df71193c5f9c30b7052d72c1682f13ec88c8dcf06b29cdd62 WatchSource:0}: Error finding container 0b406db2ff2b9a9df71193c5f9c30b7052d72c1682f13ec88c8dcf06b29cdd62: Status 404 returned error can't find the container with id 0b406db2ff2b9a9df71193c5f9c30b7052d72c1682f13ec88c8dcf06b29cdd62
Oct 30 01:08:47.988345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:47.988217    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6" event={"ID":"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa","Type":"ContainerStarted","Data":"68da545cc1431df306b1d65c3d0e754d07f225cf81776cf8d9f072ee0584fc06"}
Oct 30 01:08:47.988345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:47.988303    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6" event={"ID":"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa","Type":"ContainerStarted","Data":"0b406db2ff2b9a9df71193c5f9c30b7052d72c1682f13ec88c8dcf06b29cdd62"}
Oct 30 01:08:48.025274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:48.025212    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6" podStartSLOduration=2.025198107 podStartE2EDuration="2.025198107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:48.023961874 +0000 UTC m=+1636.773617183" watchObservedRunningTime="2026-10-30 01:08:48.025198107 +0000 UTC m=+1636.774853416"
Oct 30 01:08:48.997463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:48.997422    4425 generic.go:356] "Generic (PLEG): container finished" podID="d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa" containerID="68da545cc1431df306b1d65c3d0e754d07f225cf81776cf8d9f072ee0584fc06" exitCode=0
Oct 30 01:08:48.997939 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:48.997509    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6" event={"ID":"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa","Type":"ContainerDied","Data":"68da545cc1431df306b1d65c3d0e754d07f225cf81776cf8d9f072ee0584fc06"}
Oct 30 01:08:49.766790 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:49.766736    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2" podUID="c30ebc35-d671-4159-a22c-4bfe4e628b9a" containerName="payload-processing" containerID="cri-o://d417c03aa3b666bed7de1fa6785a3f14f1027e7f92d3fb91234e771477bc53dd" gracePeriod=25
Oct 30 01:08:49.799646 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:49.799571    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j" podUID="406b7db9-a1a4-4e81-b692-577620838bb8" containerName="payload-pre-processing" containerID="cri-o://46351a06b497f92fe80d7ad4bd886008959ed4a00579d8a42390adad71e3ff47" gracePeriod=25
Oct 30 01:08:50.041370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.041267    4425 generic.go:356] "Generic (PLEG): container finished" podID="406b7db9-a1a4-4e81-b692-577620838bb8" containerID="46351a06b497f92fe80d7ad4bd886008959ed4a00579d8a42390adad71e3ff47" exitCode=0
Oct 30 01:08:50.041860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.041460    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j" event={"ID":"406b7db9-a1a4-4e81-b692-577620838bb8","Type":"ContainerDied","Data":"46351a06b497f92fe80d7ad4bd886008959ed4a00579d8a42390adad71e3ff47"}
Oct 30 01:08:50.043946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.043894    4425 generic.go:356] "Generic (PLEG): container finished" podID="c30ebc35-d671-4159-a22c-4bfe4e628b9a" containerID="d417c03aa3b666bed7de1fa6785a3f14f1027e7f92d3fb91234e771477bc53dd" exitCode=0
Oct 30 01:08:50.044130 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.043970    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2" event={"ID":"c30ebc35-d671-4159-a22c-4bfe4e628b9a","Type":"ContainerDied","Data":"d417c03aa3b666bed7de1fa6785a3f14f1027e7f92d3fb91234e771477bc53dd"}
Oct 30 01:08:50.124119 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.124085    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"
Oct 30 01:08:50.211506 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.211475    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"
Oct 30 01:08:50.224738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.224704    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c30ebc35-d671-4159-a22c-4bfe4e628b9a-kube-api-access-bg82p\" (UniqueName: \"kubernetes.io/projected/c30ebc35-d671-4159-a22c-4bfe4e628b9a-kube-api-access-bg82p\") pod \"c30ebc35-d671-4159-a22c-4bfe4e628b9a\" (UID: \"c30ebc35-d671-4159-a22c-4bfe4e628b9a\") "
Oct 30 01:08:50.224950 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.224784    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c30ebc35-d671-4159-a22c-4bfe4e628b9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30ebc35-d671-4159-a22c-4bfe4e628b9a-plugins-config-volume\") pod \"c30ebc35-d671-4159-a22c-4bfe4e628b9a\" (UID: \"c30ebc35-d671-4159-a22c-4bfe4e628b9a\") "
Oct 30 01:08:50.225380 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.225349    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c30ebc35-d671-4159-a22c-4bfe4e628b9a-plugins-config-volume" pod "c30ebc35-d671-4159-a22c-4bfe4e628b9a" (UID: "c30ebc35-d671-4159-a22c-4bfe4e628b9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:50.227460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.227430    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30ebc35-d671-4159-a22c-4bfe4e628b9a-kube-api-access-bg82p" pod "c30ebc35-d671-4159-a22c-4bfe4e628b9a" (UID: "c30ebc35-d671-4159-a22c-4bfe4e628b9a"). InnerVolumeSpecName "kube-api-access-bg82p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:50.257294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.257258    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"
Oct 30 01:08:50.326053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.325957    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-maas-api-xks-ca\") pod \"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa\" (UID: \"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa\") "
Oct 30 01:08:50.326226 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.326108    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-kube-api-access-qdz52\" (UniqueName: \"kubernetes.io/projected/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-kube-api-access-qdz52\") pod \"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa\" (UID: \"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa\") "
Oct 30 01:08:50.326226 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.326175    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-maas-api-openshift-ca\") pod \"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa\" (UID: \"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa\") "
Oct 30 01:08:50.326360 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.326294    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/406b7db9-a1a4-4e81-b692-577620838bb8-kube-api-access-vrnjm\" (UniqueName: \"kubernetes.io/projected/406b7db9-a1a4-4e81-b692-577620838bb8-kube-api-access-vrnjm\") pod \"406b7db9-a1a4-4e81-b692-577620838bb8\" (UID: \"406b7db9-a1a4-4e81-b692-577620838bb8\") "
Oct 30 01:08:50.326360 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.326338    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/406b7db9-a1a4-4e81-b692-577620838bb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406b7db9-a1a4-4e81-b692-577620838bb8-plugins-config-volume\") pod \"406b7db9-a1a4-4e81-b692-577620838bb8\" (UID: \"406b7db9-a1a4-4e81-b692-577620838bb8\") "
Oct 30 01:08:50.326561 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.326526    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-maas-api-openshift-ca" pod "d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa" (UID: "d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:50.326762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.326736    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/406b7db9-a1a4-4e81-b692-577620838bb8-plugins-config-volume" pod "406b7db9-a1a4-4e81-b692-577620838bb8" (UID: "406b7db9-a1a4-4e81-b692-577620838bb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:50.327874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.327052    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:08:50.327874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.327081    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg82p\" (UniqueName: \"kubernetes.io/projected/c30ebc35-d671-4159-a22c-4bfe4e628b9a-kube-api-access-bg82p\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:08:50.327874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.327096    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/406b7db9-a1a4-4e81-b692-577620838bb8-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:08:50.327874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.327113    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30ebc35-d671-4159-a22c-4bfe4e628b9a-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:08:50.328803 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.328775    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406b7db9-a1a4-4e81-b692-577620838bb8-kube-api-access-vrnjm" pod "406b7db9-a1a4-4e81-b692-577620838bb8" (UID: "406b7db9-a1a4-4e81-b692-577620838bb8"). InnerVolumeSpecName "kube-api-access-vrnjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:50.328907 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.328819    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-kube-api-access-qdz52" pod "d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa" (UID: "d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa"). InnerVolumeSpecName "kube-api-access-qdz52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:50.329225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.329203    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-maas-api-xks-ca" pod "d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa" (UID: "d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:50.428427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.428382    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdz52\" (UniqueName: \"kubernetes.io/projected/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-kube-api-access-qdz52\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:08:50.428427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.428426    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrnjm\" (UniqueName: \"kubernetes.io/projected/406b7db9-a1a4-4e81-b692-577620838bb8-kube-api-access-vrnjm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:08:50.428658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:50.428442    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:08:51.059194 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:51.059076    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"
Oct 30 01:08:51.059194 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:51.059098    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j" event={"ID":"406b7db9-a1a4-4e81-b692-577620838bb8","Type":"ContainerDied","Data":"b6ca25dd002367e7936e9b6244aa240d2cf8481c4c45a863710ae26ab7b1fbf1"}
Oct 30 01:08:51.059768 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:51.059206    4425 scope.go:122] "RemoveContainer" containerID="46351a06b497f92fe80d7ad4bd886008959ed4a00579d8a42390adad71e3ff47"
Oct 30 01:08:51.062087 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:51.062044    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2" event={"ID":"c30ebc35-d671-4159-a22c-4bfe4e628b9a","Type":"ContainerDied","Data":"ecdc517bd784fa0c2ca51e554bc14bf27e88b675c0279897a7d96bc0f4532658"}
Oct 30 01:08:51.066535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:51.062667    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"
Oct 30 01:08:51.076384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:51.076356    4425 scope.go:122] "RemoveContainer" containerID="d417c03aa3b666bed7de1fa6785a3f14f1027e7f92d3fb91234e771477bc53dd"
Oct 30 01:08:51.079104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:51.079070    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6" event={"ID":"d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa","Type":"ContainerDied","Data":"0b406db2ff2b9a9df71193c5f9c30b7052d72c1682f13ec88c8dcf06b29cdd62"}
Oct 30 01:08:51.079248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:51.079208    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b406db2ff2b9a9df71193c5f9c30b7052d72c1682f13ec88c8dcf06b29cdd62"
Oct 30 01:08:51.079317 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:51.079273    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"
Oct 30 01:08:51.124963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:51.124919    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"]
Oct 30 01:08:51.133975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:51.133941    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-429c67-7bb95b4cd7-v8v4j"]
Oct 30 01:08:51.166033 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:51.165992    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"]
Oct 30 01:08:51.177040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:51.177003    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-429c67-c6fc6bd48-jwxh2"]
Oct 30 01:08:51.423510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:51.423419    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="406b7db9-a1a4-4e81-b692-577620838bb8" path="/var/lib/kubelet/pods/406b7db9-a1a4-4e81-b692-577620838bb8/volumes"
Oct 30 01:08:51.424029 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:51.424005    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c30ebc35-d671-4159-a22c-4bfe4e628b9a" path="/var/lib/kubelet/pods/c30ebc35-d671-4159-a22c-4bfe4e628b9a/volumes"
Oct 30 01:08:51.934489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:08:51.934452    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"
Oct 30 01:09:02.249676 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:02.245638    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"]
Oct 30 01:09:02.250658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:02.250597    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd" podUID="de9afa7c-812e-42be-84d7-19f7d180255d" containerName="payload-pre-processing" containerID="cri-o://2d18797556d743fa4d2458c47bfd35658e04ef9aeb7e110fc05891c8eb8cb6e6" gracePeriod=30
Oct 30 01:09:02.545113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:02.545079    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"
Oct 30 01:09:02.593682 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:02.593623    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de9afa7c-812e-42be-84d7-19f7d180255d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de9afa7c-812e-42be-84d7-19f7d180255d-plugins-config-volume\") pod \"de9afa7c-812e-42be-84d7-19f7d180255d\" (UID: \"de9afa7c-812e-42be-84d7-19f7d180255d\") "
Oct 30 01:09:02.594095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:02.594043    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de9afa7c-812e-42be-84d7-19f7d180255d-plugins-config-volume" pod "de9afa7c-812e-42be-84d7-19f7d180255d" (UID: "de9afa7c-812e-42be-84d7-19f7d180255d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:02.594574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:02.594130    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de9afa7c-812e-42be-84d7-19f7d180255d-kube-api-access-zp5jf\" (UniqueName: \"kubernetes.io/projected/de9afa7c-812e-42be-84d7-19f7d180255d-kube-api-access-zp5jf\") pod \"de9afa7c-812e-42be-84d7-19f7d180255d\" (UID: \"de9afa7c-812e-42be-84d7-19f7d180255d\") "
Oct 30 01:09:02.595663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:02.595634    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de9afa7c-812e-42be-84d7-19f7d180255d-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:09:02.596940 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:02.596908    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9afa7c-812e-42be-84d7-19f7d180255d-kube-api-access-zp5jf" pod "de9afa7c-812e-42be-84d7-19f7d180255d" (UID: "de9afa7c-812e-42be-84d7-19f7d180255d"). InnerVolumeSpecName "kube-api-access-zp5jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:02.720130 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:02.720091    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp5jf\" (UniqueName: \"kubernetes.io/projected/de9afa7c-812e-42be-84d7-19f7d180255d-kube-api-access-zp5jf\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:09:03.047231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:03.047189    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"]
Oct 30 01:09:03.050549 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:03.050497    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-838fd5-bm4t9"]
Oct 30 01:09:03.175634 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:03.175593    4425 generic.go:356] "Generic (PLEG): container finished" podID="de9afa7c-812e-42be-84d7-19f7d180255d" containerID="2d18797556d743fa4d2458c47bfd35658e04ef9aeb7e110fc05891c8eb8cb6e6" exitCode=0
Oct 30 01:09:03.175634 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:03.175637    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"
Oct 30 01:09:03.175904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:03.175668    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd" event={"ID":"de9afa7c-812e-42be-84d7-19f7d180255d","Type":"ContainerDied","Data":"2d18797556d743fa4d2458c47bfd35658e04ef9aeb7e110fc05891c8eb8cb6e6"}
Oct 30 01:09:03.175904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:03.175729    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd" event={"ID":"de9afa7c-812e-42be-84d7-19f7d180255d","Type":"ContainerDied","Data":"85e97e8656dad09c0f67eff0b659ca264836e81ba53171780c7d9dbed46d8a7b"}
Oct 30 01:09:03.175904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:03.175755    4425 scope.go:122] "RemoveContainer" containerID="2d18797556d743fa4d2458c47bfd35658e04ef9aeb7e110fc05891c8eb8cb6e6"
Oct 30 01:09:03.187740 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:03.187716    4425 scope.go:122] "RemoveContainer" containerID="2d18797556d743fa4d2458c47bfd35658e04ef9aeb7e110fc05891c8eb8cb6e6"
Oct 30 01:09:03.188172 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:09:03.188133    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d18797556d743fa4d2458c47bfd35658e04ef9aeb7e110fc05891c8eb8cb6e6\": container with ID starting with 2d18797556d743fa4d2458c47bfd35658e04ef9aeb7e110fc05891c8eb8cb6e6 not found: ID does not exist" containerID="2d18797556d743fa4d2458c47bfd35658e04ef9aeb7e110fc05891c8eb8cb6e6"
Oct 30 01:09:03.188280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:03.188183    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d18797556d743fa4d2458c47bfd35658e04ef9aeb7e110fc05891c8eb8cb6e6"} err="failed to get container status \"2d18797556d743fa4d2458c47bfd35658e04ef9aeb7e110fc05891c8eb8cb6e6\": rpc error: code = NotFound desc = could not find container \"2d18797556d743fa4d2458c47bfd35658e04ef9aeb7e110fc05891c8eb8cb6e6\": container with ID starting with 2d18797556d743fa4d2458c47bfd35658e04ef9aeb7e110fc05891c8eb8cb6e6 not found: ID does not exist"
Oct 30 01:09:03.226138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:03.226096    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"]
Oct 30 01:09:03.233429 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:03.233394    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-838fd5-7b8b9b695-rxkqd"]
Oct 30 01:09:03.433365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:03.433283    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="683618f7-1ccb-4c55-a792-efff34d5adec" path="/var/lib/kubelet/pods/683618f7-1ccb-4c55-a792-efff34d5adec/volumes"
Oct 30 01:09:03.433761 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:03.433745    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de9afa7c-812e-42be-84d7-19f7d180255d" path="/var/lib/kubelet/pods/de9afa7c-812e-42be-84d7-19f7d180255d/volumes"
Oct 30 01:09:07.557740 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:09:07.557703    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:09:14.060945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:14.060898    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"]
Oct 30 01:09:14.071002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:14.070961    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a6bc3f-nhr4h"]
Oct 30 01:09:15.408181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:15.407889    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:09:15.438401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:15.438244    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="278cc048-1556-407a-9e61-98799928615d" path="/var/lib/kubelet/pods/278cc048-1556-407a-9e61-98799928615d/volumes"
Oct 30 01:09:17.605533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.605482    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"]
Oct 30 01:09:17.606065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.606044    4425 cpu_manager.go:414] "containerMap: removing container" podUID="406b7db9-a1a4-4e81-b692-577620838bb8" containerName="payload-pre-processing"
Oct 30 01:09:17.606065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.606056    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="406b7db9-a1a4-4e81-b692-577620838bb8" containerName="payload-pre-processing"
Oct 30 01:09:17.606065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.606068    4425 cpu_manager.go:414] "containerMap: removing container" podUID="de9afa7c-812e-42be-84d7-19f7d180255d" containerName="payload-pre-processing"
Oct 30 01:09:17.606257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.606074    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de9afa7c-812e-42be-84d7-19f7d180255d" containerName="payload-pre-processing"
Oct 30 01:09:17.606257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.606098    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c30ebc35-d671-4159-a22c-4bfe4e628b9a" containerName="payload-processing"
Oct 30 01:09:17.606257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.606103    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c30ebc35-d671-4159-a22c-4bfe4e628b9a" containerName="payload-processing"
Oct 30 01:09:17.606257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.606122    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa" containerName="revoke-keys"
Oct 30 01:09:17.606257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.606128    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa" containerName="revoke-keys"
Oct 30 01:09:17.606257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.606232    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa" containerName="revoke-keys"
Oct 30 01:09:17.606257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.606246    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="406b7db9-a1a4-4e81-b692-577620838bb8" containerName="payload-pre-processing"
Oct 30 01:09:17.606257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.606252    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="de9afa7c-812e-42be-84d7-19f7d180255d" containerName="payload-pre-processing"
Oct 30 01:09:17.606257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.606261    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c30ebc35-d671-4159-a22c-4bfe4e628b9a" containerName="payload-processing"
Oct 30 01:09:17.611660 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.611620    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"
Oct 30 01:09:17.625873 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.625837    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:09:17.638746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.638713    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:09:17.638958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.638830    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:09:17.638958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.638902    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:09:17.641815 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.641767    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"]
Oct 30 01:09:17.671762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.671213    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9\" (UID: \"5c335ebd-e66e-42e4-8555-c0d29f8e83a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"
Oct 30 01:09:17.671762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.671330    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whz8c\" (UniqueName: \"kubernetes.io/projected/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-kube-api-access-whz8c\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9\" (UID: \"5c335ebd-e66e-42e4-8555-c0d29f8e83a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"
Oct 30 01:09:17.671762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.671517    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9\" (UID: \"5c335ebd-e66e-42e4-8555-c0d29f8e83a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"
Oct 30 01:09:17.772489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.772443    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9\" (UID: \"5c335ebd-e66e-42e4-8555-c0d29f8e83a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"
Oct 30 01:09:17.772489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.772502    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whz8c\" (UniqueName: \"kubernetes.io/projected/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-kube-api-access-whz8c\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9\" (UID: \"5c335ebd-e66e-42e4-8555-c0d29f8e83a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"
Oct 30 01:09:17.772795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.772636    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9\" (UID: \"5c335ebd-e66e-42e4-8555-c0d29f8e83a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"
Oct 30 01:09:17.773275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.773247    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9\" (UID: \"5c335ebd-e66e-42e4-8555-c0d29f8e83a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"
Oct 30 01:09:17.775419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.775389    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9\" (UID: \"5c335ebd-e66e-42e4-8555-c0d29f8e83a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"
Oct 30 01:09:17.794722 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.794684    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whz8c\" (UniqueName: \"kubernetes.io/projected/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-kube-api-access-whz8c\") pod \"maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9\" (UID: \"5c335ebd-e66e-42e4-8555-c0d29f8e83a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"
Oct 30 01:09:17.931847 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:17.931755    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"
Oct 30 01:09:18.240925 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:18.240888    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"]
Oct 30 01:09:18.243130 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:09:18.243102    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c335ebd_e66e_42e4_8555_c0d29f8e83a1.slice/crio-4ef8dca1dc9bf487cf4f8ed633693bab3a18cbc33f6c8de9497ea1d42f45b5c3 WatchSource:0}: Error finding container 4ef8dca1dc9bf487cf4f8ed633693bab3a18cbc33f6c8de9497ea1d42f45b5c3: Status 404 returned error can't find the container with id 4ef8dca1dc9bf487cf4f8ed633693bab3a18cbc33f6c8de9497ea1d42f45b5c3
Oct 30 01:09:18.322918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:18.322879    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9" event={"ID":"5c335ebd-e66e-42e4-8555-c0d29f8e83a1","Type":"ContainerStarted","Data":"4ef8dca1dc9bf487cf4f8ed633693bab3a18cbc33f6c8de9497ea1d42f45b5c3"}
Oct 30 01:09:19.331278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:19.331236    4425 generic.go:356] "Generic (PLEG): container finished" podID="5c335ebd-e66e-42e4-8555-c0d29f8e83a1" containerID="935be13dec14ae8de7adc8d08a48a9400464f2bc5f780437faa3dbf6f632f09b" exitCode=0
Oct 30 01:09:19.331760 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:19.331325    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9" event={"ID":"5c335ebd-e66e-42e4-8555-c0d29f8e83a1","Type":"ContainerDied","Data":"935be13dec14ae8de7adc8d08a48a9400464f2bc5f780437faa3dbf6f632f09b"}
Oct 30 01:09:20.282550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.282504    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"]
Oct 30 01:09:20.286876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.286846    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"
Oct 30 01:09:20.307393 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.307356    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"]
Oct 30 01:09:20.313423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.313385    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-4ec143\""
Oct 30 01:09:20.313423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.313396    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-4ec143-dockercfg-dnzh8\""
Oct 30 01:09:20.345938 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.345902    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"]
Oct 30 01:09:20.351072 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.351040    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"
Oct 30 01:09:20.371726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.371677    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"]
Oct 30 01:09:20.408969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.408782    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99vkl\" (UniqueName: \"kubernetes.io/projected/414e0450-fc6f-4380-9b46-381fc11cc95e-kube-api-access-99vkl\") pod \"payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f\" (UID: \"414e0450-fc6f-4380-9b46-381fc11cc95e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"
Oct 30 01:09:20.408969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.408877    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50d15c6-f321-40a9-96fa-c70c6d1509fc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr\" (UID: \"a50d15c6-f321-40a9-96fa-c70c6d1509fc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"
Oct 30 01:09:20.409275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.409078    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26txx\" (UniqueName: \"kubernetes.io/projected/a50d15c6-f321-40a9-96fa-c70c6d1509fc-kube-api-access-26txx\") pod \"payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr\" (UID: \"a50d15c6-f321-40a9-96fa-c70c6d1509fc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"
Oct 30 01:09:20.409275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.409114    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/414e0450-fc6f-4380-9b46-381fc11cc95e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f\" (UID: \"414e0450-fc6f-4380-9b46-381fc11cc95e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"
Oct 30 01:09:20.521898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.516280    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99vkl\" (UniqueName: \"kubernetes.io/projected/414e0450-fc6f-4380-9b46-381fc11cc95e-kube-api-access-99vkl\") pod \"payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f\" (UID: \"414e0450-fc6f-4380-9b46-381fc11cc95e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"
Oct 30 01:09:20.521898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.518948    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50d15c6-f321-40a9-96fa-c70c6d1509fc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr\" (UID: \"a50d15c6-f321-40a9-96fa-c70c6d1509fc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"
Oct 30 01:09:20.521898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.519268    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26txx\" (UniqueName: \"kubernetes.io/projected/a50d15c6-f321-40a9-96fa-c70c6d1509fc-kube-api-access-26txx\") pod \"payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr\" (UID: \"a50d15c6-f321-40a9-96fa-c70c6d1509fc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"
Oct 30 01:09:20.521898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.519313    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/414e0450-fc6f-4380-9b46-381fc11cc95e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f\" (UID: \"414e0450-fc6f-4380-9b46-381fc11cc95e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"
Oct 30 01:09:20.521898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.520003    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50d15c6-f321-40a9-96fa-c70c6d1509fc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr\" (UID: \"a50d15c6-f321-40a9-96fa-c70c6d1509fc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"
Oct 30 01:09:20.521898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.520306    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/414e0450-fc6f-4380-9b46-381fc11cc95e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f\" (UID: \"414e0450-fc6f-4380-9b46-381fc11cc95e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"
Oct 30 01:09:20.538498 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.538415    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99vkl\" (UniqueName: \"kubernetes.io/projected/414e0450-fc6f-4380-9b46-381fc11cc95e-kube-api-access-99vkl\") pod \"payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f\" (UID: \"414e0450-fc6f-4380-9b46-381fc11cc95e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"
Oct 30 01:09:20.541460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.541134    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26txx\" (UniqueName: \"kubernetes.io/projected/a50d15c6-f321-40a9-96fa-c70c6d1509fc-kube-api-access-26txx\") pod \"payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr\" (UID: \"a50d15c6-f321-40a9-96fa-c70c6d1509fc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"
Oct 30 01:09:20.601684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.601648    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"
Oct 30 01:09:20.608424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.608402    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"
Oct 30 01:09:20.668490 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.668460    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"
Oct 30 01:09:20.722633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.722592    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-maas-api-xks-ca\") pod \"5c335ebd-e66e-42e4-8555-c0d29f8e83a1\" (UID: \"5c335ebd-e66e-42e4-8555-c0d29f8e83a1\") "
Oct 30 01:09:20.722828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.722784    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-kube-api-access-whz8c\" (UniqueName: \"kubernetes.io/projected/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-kube-api-access-whz8c\") pod \"5c335ebd-e66e-42e4-8555-c0d29f8e83a1\" (UID: \"5c335ebd-e66e-42e4-8555-c0d29f8e83a1\") "
Oct 30 01:09:20.722828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.722818    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-maas-api-openshift-ca\") pod \"5c335ebd-e66e-42e4-8555-c0d29f8e83a1\" (UID: \"5c335ebd-e66e-42e4-8555-c0d29f8e83a1\") "
Oct 30 01:09:20.725138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.725095    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-maas-api-openshift-ca" pod "5c335ebd-e66e-42e4-8555-c0d29f8e83a1" (UID: "5c335ebd-e66e-42e4-8555-c0d29f8e83a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:20.726882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.726848    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-maas-api-xks-ca" pod "5c335ebd-e66e-42e4-8555-c0d29f8e83a1" (UID: "5c335ebd-e66e-42e4-8555-c0d29f8e83a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:20.730147 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.730111    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-kube-api-access-whz8c" pod "5c335ebd-e66e-42e4-8555-c0d29f8e83a1" (UID: "5c335ebd-e66e-42e4-8555-c0d29f8e83a1"). InnerVolumeSpecName "kube-api-access-whz8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:20.824833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.824735    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:09:20.824833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.824772    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whz8c\" (UniqueName: \"kubernetes.io/projected/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-kube-api-access-whz8c\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:09:20.824833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.824783    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c335ebd-e66e-42e4-8555-c0d29f8e83a1-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:09:20.925693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.925661    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"]
Oct 30 01:09:20.927659 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:09:20.927632    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50d15c6_f321_40a9_96fa_c70c6d1509fc.slice/crio-aad1accdbaa958beba02f6855f408afda0559ad1695da72f6392a2d52547ce4e WatchSource:0}: Error finding container aad1accdbaa958beba02f6855f408afda0559ad1695da72f6392a2d52547ce4e: Status 404 returned error can't find the container with id aad1accdbaa958beba02f6855f408afda0559ad1695da72f6392a2d52547ce4e
Oct 30 01:09:20.987051 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:20.987019    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"]
Oct 30 01:09:20.989027 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:09:20.989004    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414e0450_fc6f_4380_9b46_381fc11cc95e.slice/crio-45233d9e20d8a4c95a9da41ef04f4f22cff4ab488f74157806cd9d4c065d1535 WatchSource:0}: Error finding container 45233d9e20d8a4c95a9da41ef04f4f22cff4ab488f74157806cd9d4c065d1535: Status 404 returned error can't find the container with id 45233d9e20d8a4c95a9da41ef04f4f22cff4ab488f74157806cd9d4c065d1535
Oct 30 01:09:21.354086 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:21.354038    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f" event={"ID":"414e0450-fc6f-4380-9b46-381fc11cc95e","Type":"ContainerStarted","Data":"f83181df34fefbac740cda3c6d5d6c53374510cddbe9a1ba094f408bcdb482b1"}
Oct 30 01:09:21.354697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:21.354173    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"
Oct 30 01:09:21.354697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:21.354203    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f" event={"ID":"414e0450-fc6f-4380-9b46-381fc11cc95e","Type":"ContainerStarted","Data":"45233d9e20d8a4c95a9da41ef04f4f22cff4ab488f74157806cd9d4c065d1535"}
Oct 30 01:09:21.356407 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:21.356378    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"
Oct 30 01:09:21.356578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:21.356401    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9" event={"ID":"5c335ebd-e66e-42e4-8555-c0d29f8e83a1","Type":"ContainerDied","Data":"4ef8dca1dc9bf487cf4f8ed633693bab3a18cbc33f6c8de9497ea1d42f45b5c3"}
Oct 30 01:09:21.356578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:21.356458    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ef8dca1dc9bf487cf4f8ed633693bab3a18cbc33f6c8de9497ea1d42f45b5c3"
Oct 30 01:09:21.358546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:21.358514    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr" event={"ID":"a50d15c6-f321-40a9-96fa-c70c6d1509fc","Type":"ContainerStarted","Data":"b33fce9af725db8bd92e76265a8f39504b6210a1b0821acf370786e4a9bfbb7f"}
Oct 30 01:09:21.358707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:21.358570    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr" event={"ID":"a50d15c6-f321-40a9-96fa-c70c6d1509fc","Type":"ContainerStarted","Data":"aad1accdbaa958beba02f6855f408afda0559ad1695da72f6392a2d52547ce4e"}
Oct 30 01:09:21.358707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:21.358618    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"
Oct 30 01:09:21.395956 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:21.395880    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f" podStartSLOduration=1.395858215 podStartE2EDuration="1.395858215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:21.390803639 +0000 UTC m=+1670.140458959" watchObservedRunningTime="2026-10-30 01:09:21.395858215 +0000 UTC m=+1670.145513526"
Oct 30 01:09:21.445722 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:21.445650    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr" podStartSLOduration=1.445629758 podStartE2EDuration="1.445629758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:21.444668918 +0000 UTC m=+1670.194324228" watchObservedRunningTime="2026-10-30 01:09:21.445629758 +0000 UTC m=+1670.195285069"
Oct 30 01:09:22.091288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:22.091239    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"]
Oct 30 01:09:22.094602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:22.094568    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ad449e-2kjr6"]
Oct 30 01:09:23.424419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:23.424384    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa" path="/var/lib/kubelet/pods/d4eeac73-9b9c-4dac-af5d-62d0d3dd00aa/volumes"
Oct 30 01:09:32.370040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:32.370007    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"
Oct 30 01:09:32.370516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:32.370080    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"
Oct 30 01:09:32.409004 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:32.408973    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"]
Oct 30 01:09:32.424562 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:32.424532    4425 scope.go:122] "RemoveContainer" containerID="2fb91750a75440b03719751fac5dd7debee2be899e70306e7fef1162c94fe2d0"
Oct 30 01:09:32.439428 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:32.439395    4425 scope.go:122] "RemoveContainer" containerID="5109c1e4e5264d48048d711267f17436d1606f1844eb0bc6de27e3fc590b4167"
Oct 30 01:09:32.456284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:32.456260    4425 scope.go:122] "RemoveContainer" containerID="ac0722eae50aa47ab9ed124f1d57875d0d1386c3979a27faaafc0f69d19f5c41"
Oct 30 01:09:37.410132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:37.410076    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh" podUID="9b232802-7371-4e20-a226-03ae05e03441" containerName="payload-processing" containerID="cri-o://4a29ffd94216b924b02236f930b6a49dff9bb9fee87abbed748e709d4bd4b4fc" gracePeriod=25
Oct 30 01:09:37.535730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:37.535689    4425 generic.go:356] "Generic (PLEG): container finished" podID="9b232802-7371-4e20-a226-03ae05e03441" containerID="4a29ffd94216b924b02236f930b6a49dff9bb9fee87abbed748e709d4bd4b4fc" exitCode=0
Oct 30 01:09:37.535925 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:37.535727    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh" event={"ID":"9b232802-7371-4e20-a226-03ae05e03441","Type":"ContainerDied","Data":"4a29ffd94216b924b02236f930b6a49dff9bb9fee87abbed748e709d4bd4b4fc"}
Oct 30 01:09:37.561330 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:09:37.561285    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:09:37.702258 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:37.702228    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"
Oct 30 01:09:37.781555 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:37.781513    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b232802-7371-4e20-a226-03ae05e03441-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b232802-7371-4e20-a226-03ae05e03441-plugins-config-volume\") pod \"9b232802-7371-4e20-a226-03ae05e03441\" (UID: \"9b232802-7371-4e20-a226-03ae05e03441\") "
Oct 30 01:09:37.781779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:37.781602    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b232802-7371-4e20-a226-03ae05e03441-kube-api-access-xps29\" (UniqueName: \"kubernetes.io/projected/9b232802-7371-4e20-a226-03ae05e03441-kube-api-access-xps29\") pod \"9b232802-7371-4e20-a226-03ae05e03441\" (UID: \"9b232802-7371-4e20-a226-03ae05e03441\") "
Oct 30 01:09:37.782059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:37.782026    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b232802-7371-4e20-a226-03ae05e03441-plugins-config-volume" pod "9b232802-7371-4e20-a226-03ae05e03441" (UID: "9b232802-7371-4e20-a226-03ae05e03441"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:37.782271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:37.782253    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b232802-7371-4e20-a226-03ae05e03441-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:09:37.784847 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:37.784807    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b232802-7371-4e20-a226-03ae05e03441-kube-api-access-xps29" pod "9b232802-7371-4e20-a226-03ae05e03441" (UID: "9b232802-7371-4e20-a226-03ae05e03441"). InnerVolumeSpecName "kube-api-access-xps29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:37.883365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:37.883321    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xps29\" (UniqueName: \"kubernetes.io/projected/9b232802-7371-4e20-a226-03ae05e03441-kube-api-access-xps29\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:09:38.543971 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:38.543940    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"
Oct 30 01:09:38.544495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:38.543932    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh" event={"ID":"9b232802-7371-4e20-a226-03ae05e03441","Type":"ContainerDied","Data":"042a04b1e927d4414b3188ba37d53c2f0e8722f8e4063545be6c18376482e71e"}
Oct 30 01:09:38.544495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:38.544081    4425 scope.go:122] "RemoveContainer" containerID="4a29ffd94216b924b02236f930b6a49dff9bb9fee87abbed748e709d4bd4b4fc"
Oct 30 01:09:38.590529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:38.590489    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"]
Oct 30 01:09:38.600567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:38.600530    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d3c1fa-77dc6d8d78-ggvgh"]
Oct 30 01:09:39.423552 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:39.423514    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b232802-7371-4e20-a226-03ae05e03441" path="/var/lib/kubelet/pods/9b232802-7371-4e20-a226-03ae05e03441/volumes"
Oct 30 01:09:42.758182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:42.757895    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:09:44.925455 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:44.925413    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"]
Oct 30 01:09:44.926572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:44.926534    4425 cpu_manager.go:414] "containerMap: removing container" podUID="5c335ebd-e66e-42e4-8555-c0d29f8e83a1" containerName="revoke-keys"
Oct 30 01:09:44.926572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:44.926563    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c335ebd-e66e-42e4-8555-c0d29f8e83a1" containerName="revoke-keys"
Oct 30 01:09:44.926801 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:44.926605    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9b232802-7371-4e20-a226-03ae05e03441" containerName="payload-processing"
Oct 30 01:09:44.926801 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:44.926614    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b232802-7371-4e20-a226-03ae05e03441" containerName="payload-processing"
Oct 30 01:09:44.926801 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:44.926765    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c335ebd-e66e-42e4-8555-c0d29f8e83a1" containerName="revoke-keys"
Oct 30 01:09:44.926801 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:44.926784    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b232802-7371-4e20-a226-03ae05e03441" containerName="payload-processing"
Oct 30 01:09:44.932414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:44.932383    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"
Oct 30 01:09:44.947708 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:44.947673    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"]
Oct 30 01:09:44.954050 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:44.953980    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:09:44.954561 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:44.954532    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:09:44.954719 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:44.954599    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:09:44.954719 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:44.954653    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:09:45.082219 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:45.082171    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sn2f\" (UniqueName: \"kubernetes.io/projected/9638d640-9480-46a2-a031-03a7bcb81868-kube-api-access-6sn2f\") pod \"maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg\" (UID: \"9638d640-9480-46a2-a031-03a7bcb81868\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"
Oct 30 01:09:45.082416 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:45.082324    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9638d640-9480-46a2-a031-03a7bcb81868-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg\" (UID: \"9638d640-9480-46a2-a031-03a7bcb81868\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"
Oct 30 01:09:45.082528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:45.082508    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9638d640-9480-46a2-a031-03a7bcb81868-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg\" (UID: \"9638d640-9480-46a2-a031-03a7bcb81868\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"
Oct 30 01:09:45.184632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:45.184516    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9638d640-9480-46a2-a031-03a7bcb81868-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg\" (UID: \"9638d640-9480-46a2-a031-03a7bcb81868\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"
Oct 30 01:09:45.184830 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:45.184670    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sn2f\" (UniqueName: \"kubernetes.io/projected/9638d640-9480-46a2-a031-03a7bcb81868-kube-api-access-6sn2f\") pod \"maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg\" (UID: \"9638d640-9480-46a2-a031-03a7bcb81868\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"
Oct 30 01:09:45.184830 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:45.184755    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9638d640-9480-46a2-a031-03a7bcb81868-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg\" (UID: \"9638d640-9480-46a2-a031-03a7bcb81868\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"
Oct 30 01:09:45.186977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:45.185473    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9638d640-9480-46a2-a031-03a7bcb81868-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg\" (UID: \"9638d640-9480-46a2-a031-03a7bcb81868\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"
Oct 30 01:09:45.188324 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:45.188291    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9638d640-9480-46a2-a031-03a7bcb81868-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg\" (UID: \"9638d640-9480-46a2-a031-03a7bcb81868\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"
Oct 30 01:09:45.207231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:45.207189    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sn2f\" (UniqueName: \"kubernetes.io/projected/9638d640-9480-46a2-a031-03a7bcb81868-kube-api-access-6sn2f\") pod \"maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg\" (UID: \"9638d640-9480-46a2-a031-03a7bcb81868\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"
Oct 30 01:09:45.250902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:45.250860    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"
Oct 30 01:09:45.568114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:45.568078    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"]
Oct 30 01:09:45.569719 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:09:45.569689    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9638d640_9480_46a2_a031_03a7bcb81868.slice/crio-a7986d8d99c0e1a5efdcf2e1a24c8fa8d16df3a6dad836c5dd2b35eb9a3b6036 WatchSource:0}: Error finding container a7986d8d99c0e1a5efdcf2e1a24c8fa8d16df3a6dad836c5dd2b35eb9a3b6036: Status 404 returned error can't find the container with id a7986d8d99c0e1a5efdcf2e1a24c8fa8d16df3a6dad836c5dd2b35eb9a3b6036
Oct 30 01:09:45.593522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:45.593490    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg" event={"ID":"9638d640-9480-46a2-a031-03a7bcb81868","Type":"ContainerStarted","Data":"a7986d8d99c0e1a5efdcf2e1a24c8fa8d16df3a6dad836c5dd2b35eb9a3b6036"}
Oct 30 01:09:46.602520 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:46.602469    4425 generic.go:356] "Generic (PLEG): container finished" podID="9638d640-9480-46a2-a031-03a7bcb81868" containerID="05508247032e1b507b8d3bb29015e8f5499d18b11074838c994ef803ba77e3f3" exitCode=0
Oct 30 01:09:46.603081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:46.602723    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg" event={"ID":"9638d640-9480-46a2-a031-03a7bcb81868","Type":"ContainerDied","Data":"05508247032e1b507b8d3bb29015e8f5499d18b11074838c994ef803ba77e3f3"}
Oct 30 01:09:47.793175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:47.793135    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"
Oct 30 01:09:47.927784 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:47.927682    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9638d640-9480-46a2-a031-03a7bcb81868-kube-api-access-6sn2f\" (UniqueName: \"kubernetes.io/projected/9638d640-9480-46a2-a031-03a7bcb81868-kube-api-access-6sn2f\") pod \"9638d640-9480-46a2-a031-03a7bcb81868\" (UID: \"9638d640-9480-46a2-a031-03a7bcb81868\") "
Oct 30 01:09:47.927959 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:47.927881    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9638d640-9480-46a2-a031-03a7bcb81868-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9638d640-9480-46a2-a031-03a7bcb81868-maas-api-xks-ca\") pod \"9638d640-9480-46a2-a031-03a7bcb81868\" (UID: \"9638d640-9480-46a2-a031-03a7bcb81868\") "
Oct 30 01:09:47.927959 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:47.927951    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9638d640-9480-46a2-a031-03a7bcb81868-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9638d640-9480-46a2-a031-03a7bcb81868-maas-api-openshift-ca\") pod \"9638d640-9480-46a2-a031-03a7bcb81868\" (UID: \"9638d640-9480-46a2-a031-03a7bcb81868\") "
Oct 30 01:09:47.928375 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:47.928345    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9638d640-9480-46a2-a031-03a7bcb81868-maas-api-openshift-ca" pod "9638d640-9480-46a2-a031-03a7bcb81868" (UID: "9638d640-9480-46a2-a031-03a7bcb81868"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:47.930024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:47.929995    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9638d640-9480-46a2-a031-03a7bcb81868-maas-api-xks-ca" pod "9638d640-9480-46a2-a031-03a7bcb81868" (UID: "9638d640-9480-46a2-a031-03a7bcb81868"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:47.930132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:47.930001    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9638d640-9480-46a2-a031-03a7bcb81868-kube-api-access-6sn2f" pod "9638d640-9480-46a2-a031-03a7bcb81868" (UID: "9638d640-9480-46a2-a031-03a7bcb81868"). InnerVolumeSpecName "kube-api-access-6sn2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:48.029951 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:48.029903    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9638d640-9480-46a2-a031-03a7bcb81868-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:09:48.029951 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:48.029940    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9638d640-9480-46a2-a031-03a7bcb81868-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:09:48.029951 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:48.029954    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sn2f\" (UniqueName: \"kubernetes.io/projected/9638d640-9480-46a2-a031-03a7bcb81868-kube-api-access-6sn2f\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:09:48.618567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:48.618536    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"
Oct 30 01:09:48.618765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:48.618568    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg" event={"ID":"9638d640-9480-46a2-a031-03a7bcb81868","Type":"ContainerDied","Data":"a7986d8d99c0e1a5efdcf2e1a24c8fa8d16df3a6dad836c5dd2b35eb9a3b6036"}
Oct 30 01:09:48.618765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:48.618630    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7986d8d99c0e1a5efdcf2e1a24c8fa8d16df3a6dad836c5dd2b35eb9a3b6036"
Oct 30 01:09:53.277200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:53.277140    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"]
Oct 30 01:09:53.286249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:53.286197    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a8ab86-gj7t9"]
Oct 30 01:09:53.426459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:53.426412    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c335ebd-e66e-42e4-8555-c0d29f8e83a1" path="/var/lib/kubelet/pods/5c335ebd-e66e-42e4-8555-c0d29f8e83a1/volumes"
Oct 30 01:09:55.455934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:55.455890    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"]
Oct 30 01:09:55.495446 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:09:55.495405    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"]
Oct 30 01:10:00.460166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.460110    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c" podUID="a8e9db36-b363-4056-a290-8755b8dd91ec" containerName="payload-processing" containerID="cri-o://98d0282e1f8b669f7f94f0d49667c8ab0da35b83c551cab13cd7c5cffadf828c" gracePeriod=25
Oct 30 01:10:00.495935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.495885    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9" podUID="af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3" containerName="payload-pre-processing" containerID="cri-o://b09301be4d6ae125613d209f4e607db810a7b8604bf32d8b023792be2e46faec" gracePeriod=25
Oct 30 01:10:00.730449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.730187    4425 generic.go:356] "Generic (PLEG): container finished" podID="af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3" containerID="b09301be4d6ae125613d209f4e607db810a7b8604bf32d8b023792be2e46faec" exitCode=0
Oct 30 01:10:00.730449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.730316    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9" event={"ID":"af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3","Type":"ContainerDied","Data":"b09301be4d6ae125613d209f4e607db810a7b8604bf32d8b023792be2e46faec"}
Oct 30 01:10:00.732790 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.732594    4425 generic.go:356] "Generic (PLEG): container finished" podID="a8e9db36-b363-4056-a290-8755b8dd91ec" containerID="98d0282e1f8b669f7f94f0d49667c8ab0da35b83c551cab13cd7c5cffadf828c" exitCode=0
Oct 30 01:10:00.732790 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.732728    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c" event={"ID":"a8e9db36-b363-4056-a290-8755b8dd91ec","Type":"ContainerDied","Data":"98d0282e1f8b669f7f94f0d49667c8ab0da35b83c551cab13cd7c5cffadf828c"}
Oct 30 01:10:00.804042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.804010    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"
Oct 30 01:10:00.850130 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.849796    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"
Oct 30 01:10:00.928745 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.928709    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e9db36-b363-4056-a290-8755b8dd91ec-kube-api-access-rswv6\" (UniqueName: \"kubernetes.io/projected/a8e9db36-b363-4056-a290-8755b8dd91ec-kube-api-access-rswv6\") pod \"a8e9db36-b363-4056-a290-8755b8dd91ec\" (UID: \"a8e9db36-b363-4056-a290-8755b8dd91ec\") "
Oct 30 01:10:00.928909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.928826    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e9db36-b363-4056-a290-8755b8dd91ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e9db36-b363-4056-a290-8755b8dd91ec-plugins-config-volume\") pod \"a8e9db36-b363-4056-a290-8755b8dd91ec\" (UID: \"a8e9db36-b363-4056-a290-8755b8dd91ec\") "
Oct 30 01:10:00.928985 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.928926    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3-plugins-config-volume\") pod \"af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3\" (UID: \"af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3\") "
Oct 30 01:10:00.929048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.929031    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3-kube-api-access-vng5k\" (UniqueName: \"kubernetes.io/projected/af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3-kube-api-access-vng5k\") pod \"af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3\" (UID: \"af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3\") "
Oct 30 01:10:00.929462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.929356    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e9db36-b363-4056-a290-8755b8dd91ec-plugins-config-volume" pod "a8e9db36-b363-4056-a290-8755b8dd91ec" (UID: "a8e9db36-b363-4056-a290-8755b8dd91ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:00.929462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.929387    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3-plugins-config-volume" pod "af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3" (UID: "af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:00.929743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.929715    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e9db36-b363-4056-a290-8755b8dd91ec-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:00.929743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.929748    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:00.931741 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.931710    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e9db36-b363-4056-a290-8755b8dd91ec-kube-api-access-rswv6" pod "a8e9db36-b363-4056-a290-8755b8dd91ec" (UID: "a8e9db36-b363-4056-a290-8755b8dd91ec"). InnerVolumeSpecName "kube-api-access-rswv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:00.931882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:00.931809    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3-kube-api-access-vng5k" pod "af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3" (UID: "af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3"). InnerVolumeSpecName "kube-api-access-vng5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:01.030995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.030954    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vng5k\" (UniqueName: \"kubernetes.io/projected/af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3-kube-api-access-vng5k\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:01.030995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.030991    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rswv6\" (UniqueName: \"kubernetes.io/projected/a8e9db36-b363-4056-a290-8755b8dd91ec-kube-api-access-rswv6\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:01.741276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.741150    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"
Oct 30 01:10:01.741760 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.741184    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9" event={"ID":"af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3","Type":"ContainerDied","Data":"2c8cdf1ad5fbe64ea53b0be502e062219795b6147bd0c35c2afb1f5137249fc9"}
Oct 30 01:10:01.741760 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.741371    4425 scope.go:122] "RemoveContainer" containerID="b09301be4d6ae125613d209f4e607db810a7b8604bf32d8b023792be2e46faec"
Oct 30 01:10:01.742946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.742860    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"
Oct 30 01:10:01.742946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.742892    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c" event={"ID":"a8e9db36-b363-4056-a290-8755b8dd91ec","Type":"ContainerDied","Data":"e708c3da4f1a57d6d06e8aa26d41980b0fa2abf36eb7d5308366ce5b2490c6be"}
Oct 30 01:10:01.752473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.752447    4425 scope.go:122] "RemoveContainer" containerID="98d0282e1f8b669f7f94f0d49667c8ab0da35b83c551cab13cd7c5cffadf828c"
Oct 30 01:10:01.800542 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.800495    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"]
Oct 30 01:10:01.809766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.809730    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b43bff-756979f9d6-g766c"]
Oct 30 01:10:01.840918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.840879    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"]
Oct 30 01:10:01.844503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.844468    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b43bff-ccb4458f9-zjsf9"]
Oct 30 01:10:01.877270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.877230    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"]
Oct 30 01:10:01.877843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.877826    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9638d640-9480-46a2-a031-03a7bcb81868" containerName="revoke-keys"
Oct 30 01:10:01.877886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.877847    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9638d640-9480-46a2-a031-03a7bcb81868" containerName="revoke-keys"
Oct 30 01:10:01.877886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.877878    4425 cpu_manager.go:414] "containerMap: removing container" podUID="af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3" containerName="payload-pre-processing"
Oct 30 01:10:01.877886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.877883    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3" containerName="payload-pre-processing"
Oct 30 01:10:01.877990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.877904    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a8e9db36-b363-4056-a290-8755b8dd91ec" containerName="payload-processing"
Oct 30 01:10:01.877990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.877910    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e9db36-b363-4056-a290-8755b8dd91ec" containerName="payload-processing"
Oct 30 01:10:01.877990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.877987    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3" containerName="payload-pre-processing"
Oct 30 01:10:01.878084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.877999    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e9db36-b363-4056-a290-8755b8dd91ec" containerName="payload-processing"
Oct 30 01:10:01.878084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.878008    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9638d640-9480-46a2-a031-03a7bcb81868" containerName="revoke-keys"
Oct 30 01:10:01.882817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.882792    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"
Oct 30 01:10:01.899564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.899527    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:10:01.899564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.899527    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:10:01.902074 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.902043    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"]
Oct 30 01:10:01.941840 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.941802    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4js99\" (UniqueName: \"kubernetes.io/projected/23d957ad-df49-489b-b7ac-07a3427b01f5-kube-api-access-4js99\") pod \"maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc\" (UID: \"23d957ad-df49-489b-b7ac-07a3427b01f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"
Oct 30 01:10:01.942043 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.941863    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d957ad-df49-489b-b7ac-07a3427b01f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc\" (UID: \"23d957ad-df49-489b-b7ac-07a3427b01f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"
Oct 30 01:10:01.942103 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.942087    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d957ad-df49-489b-b7ac-07a3427b01f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc\" (UID: \"23d957ad-df49-489b-b7ac-07a3427b01f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"
Oct 30 01:10:01.963964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.963918    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:10:01.964213 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:01.963995    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:10:02.043551 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:02.043514    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4js99\" (UniqueName: \"kubernetes.io/projected/23d957ad-df49-489b-b7ac-07a3427b01f5-kube-api-access-4js99\") pod \"maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc\" (UID: \"23d957ad-df49-489b-b7ac-07a3427b01f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"
Oct 30 01:10:02.043786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:02.043575    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d957ad-df49-489b-b7ac-07a3427b01f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc\" (UID: \"23d957ad-df49-489b-b7ac-07a3427b01f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"
Oct 30 01:10:02.043856 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:02.043815    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d957ad-df49-489b-b7ac-07a3427b01f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc\" (UID: \"23d957ad-df49-489b-b7ac-07a3427b01f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"
Oct 30 01:10:02.044329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:02.044308    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d957ad-df49-489b-b7ac-07a3427b01f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc\" (UID: \"23d957ad-df49-489b-b7ac-07a3427b01f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"
Oct 30 01:10:02.046396 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:02.046369    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d957ad-df49-489b-b7ac-07a3427b01f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc\" (UID: \"23d957ad-df49-489b-b7ac-07a3427b01f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"
Oct 30 01:10:02.065797 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:02.065762    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4js99\" (UniqueName: \"kubernetes.io/projected/23d957ad-df49-489b-b7ac-07a3427b01f5-kube-api-access-4js99\") pod \"maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc\" (UID: \"23d957ad-df49-489b-b7ac-07a3427b01f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"
Oct 30 01:10:02.197864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:02.197801    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"
Oct 30 01:10:02.513826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:02.513793    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"]
Oct 30 01:10:02.516566 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:10:02.516530    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d957ad_df49_489b_b7ac_07a3427b01f5.slice/crio-5ffd738ad6e2fb2f8ddd05e95d1b3446fb5a8b872193b7b5d758c31a402cb0cb WatchSource:0}: Error finding container 5ffd738ad6e2fb2f8ddd05e95d1b3446fb5a8b872193b7b5d758c31a402cb0cb: Status 404 returned error can't find the container with id 5ffd738ad6e2fb2f8ddd05e95d1b3446fb5a8b872193b7b5d758c31a402cb0cb
Oct 30 01:10:02.751252 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:02.751210    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc" event={"ID":"23d957ad-df49-489b-b7ac-07a3427b01f5","Type":"ContainerStarted","Data":"5ffd738ad6e2fb2f8ddd05e95d1b3446fb5a8b872193b7b5d758c31a402cb0cb"}
Oct 30 01:10:03.424012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:03.423972    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e9db36-b363-4056-a290-8755b8dd91ec" path="/var/lib/kubelet/pods/a8e9db36-b363-4056-a290-8755b8dd91ec/volumes"
Oct 30 01:10:03.424598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:03.424575    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3" path="/var/lib/kubelet/pods/af25d92a-3a8e-40ae-b4d8-cfc9ee3287b3/volumes"
Oct 30 01:10:03.765989 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:03.765950    4425 generic.go:356] "Generic (PLEG): container finished" podID="23d957ad-df49-489b-b7ac-07a3427b01f5" containerID="b41603746ddf0d0d141fa4a00a1a47fe37a403e82851ccdc64ed5fdac2dbdf46" exitCode=0
Oct 30 01:10:03.766466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:03.766033    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc" event={"ID":"23d957ad-df49-489b-b7ac-07a3427b01f5","Type":"ContainerDied","Data":"b41603746ddf0d0d141fa4a00a1a47fe37a403e82851ccdc64ed5fdac2dbdf46"}
Oct 30 01:10:04.950805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:04.950777    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"
Oct 30 01:10:05.085538 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:05.085428    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d957ad-df49-489b-b7ac-07a3427b01f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d957ad-df49-489b-b7ac-07a3427b01f5-maas-api-xks-ca\") pod \"23d957ad-df49-489b-b7ac-07a3427b01f5\" (UID: \"23d957ad-df49-489b-b7ac-07a3427b01f5\") "
Oct 30 01:10:05.085722 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:05.085705    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d957ad-df49-489b-b7ac-07a3427b01f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d957ad-df49-489b-b7ac-07a3427b01f5-maas-api-openshift-ca\") pod \"23d957ad-df49-489b-b7ac-07a3427b01f5\" (UID: \"23d957ad-df49-489b-b7ac-07a3427b01f5\") "
Oct 30 01:10:05.085774 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:05.085752    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d957ad-df49-489b-b7ac-07a3427b01f5-kube-api-access-4js99\" (UniqueName: \"kubernetes.io/projected/23d957ad-df49-489b-b7ac-07a3427b01f5-kube-api-access-4js99\") pod \"23d957ad-df49-489b-b7ac-07a3427b01f5\" (UID: \"23d957ad-df49-489b-b7ac-07a3427b01f5\") "
Oct 30 01:10:05.086128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:05.086098    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d957ad-df49-489b-b7ac-07a3427b01f5-maas-api-openshift-ca" pod "23d957ad-df49-489b-b7ac-07a3427b01f5" (UID: "23d957ad-df49-489b-b7ac-07a3427b01f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:05.086479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:05.086455    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d957ad-df49-489b-b7ac-07a3427b01f5-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:05.087910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:05.087880    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d957ad-df49-489b-b7ac-07a3427b01f5-maas-api-xks-ca" pod "23d957ad-df49-489b-b7ac-07a3427b01f5" (UID: "23d957ad-df49-489b-b7ac-07a3427b01f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:10:05.088015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:05.087940    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d957ad-df49-489b-b7ac-07a3427b01f5-kube-api-access-4js99" pod "23d957ad-df49-489b-b7ac-07a3427b01f5" (UID: "23d957ad-df49-489b-b7ac-07a3427b01f5"). InnerVolumeSpecName "kube-api-access-4js99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:05.187771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:05.187725    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d957ad-df49-489b-b7ac-07a3427b01f5-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:05.187771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:05.187763    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4js99\" (UniqueName: \"kubernetes.io/projected/23d957ad-df49-489b-b7ac-07a3427b01f5-kube-api-access-4js99\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:05.781013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:05.780978    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc" event={"ID":"23d957ad-df49-489b-b7ac-07a3427b01f5","Type":"ContainerDied","Data":"5ffd738ad6e2fb2f8ddd05e95d1b3446fb5a8b872193b7b5d758c31a402cb0cb"}
Oct 30 01:10:05.781213 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:05.781033    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ffd738ad6e2fb2f8ddd05e95d1b3446fb5a8b872193b7b5d758c31a402cb0cb"
Oct 30 01:10:05.781213 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:05.780986    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"
Oct 30 01:10:07.558098 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:10:07.558057    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:10:07.609666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:07.609624    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"]
Oct 30 01:10:07.609888 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:07.609860    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f" podUID="cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" containerName="istio-proxy" containerID="cri-o://b75cf59de7e09c63251c75abadd6ecc32d6c5ee048c04035b3a1ba935d2e2113" gracePeriod=30
Oct 30 01:10:12.833391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:12.833190    4425 generic.go:356] "Generic (PLEG): container finished" podID="cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" containerID="b75cf59de7e09c63251c75abadd6ecc32d6c5ee048c04035b3a1ba935d2e2113" exitCode=0
Oct 30 01:10:12.833391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:12.833314    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f" event={"ID":"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29","Type":"ContainerDied","Data":"b75cf59de7e09c63251c75abadd6ecc32d6c5ee048c04035b3a1ba935d2e2113"}
Oct 30 01:10:12.914069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:12.914032    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 01:10:13.004206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.004147    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-workload-certs\") pod \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") "
Oct 30 01:10:13.004439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.004233    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-token\" (UniqueName: \"kubernetes.io/projected/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-token\") pod \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") "
Oct 30 01:10:13.004439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.004287    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-podinfo\") pod \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") "
Oct 30 01:10:13.004439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.004331    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-data\") pod \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") "
Oct 30 01:10:13.004632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.004445    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-workload-certs" pod "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" (UID: "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:13.004632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.004482    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-envoy\") pod \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") "
Oct 30 01:10:13.004632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.004533    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-kube-api-access-6wmf9\" (UniqueName: \"kubernetes.io/projected/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-kube-api-access-6wmf9\") pod \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") "
Oct 30 01:10:13.004632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.004608    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-workload-socket\") pod \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") "
Oct 30 01:10:13.004852 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.004638    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-credential-socket\") pod \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") "
Oct 30 01:10:13.004852 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.004730    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istiod-ca-cert\") pod \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\" (UID: \"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29\") "
Oct 30 01:10:13.005735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.005694    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-workload-socket" pod "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" (UID: "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:13.005945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.005891    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-credential-socket" pod "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" (UID: "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:13.006036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.005950    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-data" pod "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" (UID: "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:13.006129 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.006106    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istiod-ca-cert" pod "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" (UID: "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:13.006537 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.006513    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:13.006698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.006681    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:13.006813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.006798    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:13.006922 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.006909    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:13.007016 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.007005    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:13.007391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.007361    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-podinfo" pod "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" (UID: "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:10:13.008250 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.008214    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-envoy" pod "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" (UID: "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:13.008709 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.008687    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-kube-api-access-6wmf9" pod "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" (UID: "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29"). InnerVolumeSpecName "kube-api-access-6wmf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:13.009562 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.009533    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-token" pod "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" (UID: "cee90c29-d7d2-41cc-b02d-5cbcf06c9b29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:13.108940 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.108903    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:13.108940 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.108938    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wmf9\" (UniqueName: \"kubernetes.io/projected/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-kube-api-access-6wmf9\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:13.109213 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.108951    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:13.109213 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.108964    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:13.841391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.841356    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"
Oct 30 01:10:13.841391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.841378    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f" event={"ID":"cee90c29-d7d2-41cc-b02d-5cbcf06c9b29","Type":"ContainerDied","Data":"abb1e9febe37bae9e8d147aa6d0f1941e5c95c37ed676f910f7b1a532ca331aa"}
Oct 30 01:10:13.841935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.841458    4425 scope.go:122] "RemoveContainer" containerID="b75cf59de7e09c63251c75abadd6ecc32d6c5ee048c04035b3a1ba935d2e2113"
Oct 30 01:10:13.923575 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.923533    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"]
Oct 30 01:10:13.932994 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:13.932955    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-4q92f"]
Oct 30 01:10:15.424441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:15.424404    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" path="/var/lib/kubelet/pods/cee90c29-d7d2-41cc-b02d-5cbcf06c9b29/volumes"
Oct 30 01:10:16.067912 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:16.067870    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"]
Oct 30 01:10:16.068206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:16.068095    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j" podUID="dae18a2f-da25-474a-af36-6be757b43311" containerName="istio-proxy" containerID="cri-o://df0be71d9ef343d2168a976ab5974e6f22bfb29a0a44bb1d27dd4046d8d33b03" gracePeriod=30
Oct 30 01:10:17.412317 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:17.412278    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"]
Oct 30 01:10:17.424019 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:17.423981    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-429c67-wxtlc"]
Oct 30 01:10:19.423887 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:19.423850    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d957ad-df49-489b-b7ac-07a3427b01f5" path="/var/lib/kubelet/pods/23d957ad-df49-489b-b7ac-07a3427b01f5/volumes"
Oct 30 01:10:20.739684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:20.739646    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"]
Oct 30 01:10:20.749277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:20.749233    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b43bff-79jqg"]
Oct 30 01:10:21.352419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.352346    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:10:21.421090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.421051    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-workload-socket\") pod \"dae18a2f-da25-474a-af36-6be757b43311\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") "
Oct 30 01:10:21.421327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.421229    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-istio-data\") pod \"dae18a2f-da25-474a-af36-6be757b43311\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") "
Oct 30 01:10:21.421327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.421280    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae18a2f-da25-474a-af36-6be757b43311-istio-token\" (UniqueName: \"kubernetes.io/projected/dae18a2f-da25-474a-af36-6be757b43311-istio-token\") pod \"dae18a2f-da25-474a-af36-6be757b43311\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") "
Oct 30 01:10:21.421460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.421442    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae18a2f-da25-474a-af36-6be757b43311-kube-api-access-dtmhn\" (UniqueName: \"kubernetes.io/projected/dae18a2f-da25-474a-af36-6be757b43311-kube-api-access-dtmhn\") pod \"dae18a2f-da25-474a-af36-6be757b43311\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") "
Oct 30 01:10:21.421518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.421483    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-istio-envoy\") pod \"dae18a2f-da25-474a-af36-6be757b43311\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") "
Oct 30 01:10:21.421574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.421525    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae18a2f-da25-474a-af36-6be757b43311-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae18a2f-da25-474a-af36-6be757b43311-istiod-ca-cert\") pod \"dae18a2f-da25-474a-af36-6be757b43311\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") "
Oct 30 01:10:21.421574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.421568    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-credential-socket\") pod \"dae18a2f-da25-474a-af36-6be757b43311\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") "
Oct 30 01:10:21.421729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.421709    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-workload-certs\") pod \"dae18a2f-da25-474a-af36-6be757b43311\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") "
Oct 30 01:10:21.421789 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.421761    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dae18a2f-da25-474a-af36-6be757b43311-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae18a2f-da25-474a-af36-6be757b43311-istio-podinfo\") pod \"dae18a2f-da25-474a-af36-6be757b43311\" (UID: \"dae18a2f-da25-474a-af36-6be757b43311\") "
Oct 30 01:10:21.421991 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.421960    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-workload-socket" pod "dae18a2f-da25-474a-af36-6be757b43311" (UID: "dae18a2f-da25-474a-af36-6be757b43311"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:21.422284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.421985    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-istio-data" pod "dae18a2f-da25-474a-af36-6be757b43311" (UID: "dae18a2f-da25-474a-af36-6be757b43311"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:21.422408 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.422378    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-workload-certs" pod "dae18a2f-da25-474a-af36-6be757b43311" (UID: "dae18a2f-da25-474a-af36-6be757b43311"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:21.422408 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.422187    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-credential-socket" pod "dae18a2f-da25-474a-af36-6be757b43311" (UID: "dae18a2f-da25-474a-af36-6be757b43311"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:21.422612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.422474    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae18a2f-da25-474a-af36-6be757b43311-istiod-ca-cert" pod "dae18a2f-da25-474a-af36-6be757b43311" (UID: "dae18a2f-da25-474a-af36-6be757b43311"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:21.425092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.425055    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dae18a2f-da25-474a-af36-6be757b43311-istio-podinfo" pod "dae18a2f-da25-474a-af36-6be757b43311" (UID: "dae18a2f-da25-474a-af36-6be757b43311"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:10:21.425661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.425525    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae18a2f-da25-474a-af36-6be757b43311-kube-api-access-dtmhn" pod "dae18a2f-da25-474a-af36-6be757b43311" (UID: "dae18a2f-da25-474a-af36-6be757b43311"). InnerVolumeSpecName "kube-api-access-dtmhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:21.425661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.425619    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-istio-envoy" pod "dae18a2f-da25-474a-af36-6be757b43311" (UID: "dae18a2f-da25-474a-af36-6be757b43311"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:21.426681 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.426639    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9638d640-9480-46a2-a031-03a7bcb81868" path="/var/lib/kubelet/pods/9638d640-9480-46a2-a031-03a7bcb81868/volumes"
Oct 30 01:10:21.427100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.427073    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae18a2f-da25-474a-af36-6be757b43311-istio-token" pod "dae18a2f-da25-474a-af36-6be757b43311" (UID: "dae18a2f-da25-474a-af36-6be757b43311"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:21.524412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.524373    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.524412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.524410    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae18a2f-da25-474a-af36-6be757b43311-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.524663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.524424    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtmhn\" (UniqueName: \"kubernetes.io/projected/dae18a2f-da25-474a-af36-6be757b43311-kube-api-access-dtmhn\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.524663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.524438    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.524663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.524450    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae18a2f-da25-474a-af36-6be757b43311-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.524663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.524461    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.524663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.524473    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.524663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.524485    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae18a2f-da25-474a-af36-6be757b43311-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.524663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.524495    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae18a2f-da25-474a-af36-6be757b43311-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:21.941545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.941503    4425 generic.go:356] "Generic (PLEG): container finished" podID="dae18a2f-da25-474a-af36-6be757b43311" containerID="df0be71d9ef343d2168a976ab5974e6f22bfb29a0a44bb1d27dd4046d8d33b03" exitCode=0
Oct 30 01:10:21.942107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.941567    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"
Oct 30 01:10:21.942107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.941590    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j" event={"ID":"dae18a2f-da25-474a-af36-6be757b43311","Type":"ContainerDied","Data":"df0be71d9ef343d2168a976ab5974e6f22bfb29a0a44bb1d27dd4046d8d33b03"}
Oct 30 01:10:21.942107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.941667    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j" event={"ID":"dae18a2f-da25-474a-af36-6be757b43311","Type":"ContainerDied","Data":"fdc4529e78a174f383c5582b8c1c8cea050b26a6809421e69e8456777f29ad96"}
Oct 30 01:10:21.942107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.941708    4425 scope.go:122] "RemoveContainer" containerID="df0be71d9ef343d2168a976ab5974e6f22bfb29a0a44bb1d27dd4046d8d33b03"
Oct 30 01:10:21.953783 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.953757    4425 scope.go:122] "RemoveContainer" containerID="df0be71d9ef343d2168a976ab5974e6f22bfb29a0a44bb1d27dd4046d8d33b03"
Oct 30 01:10:21.954283 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:10:21.954251    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0be71d9ef343d2168a976ab5974e6f22bfb29a0a44bb1d27dd4046d8d33b03\": container with ID starting with df0be71d9ef343d2168a976ab5974e6f22bfb29a0a44bb1d27dd4046d8d33b03 not found: ID does not exist" containerID="df0be71d9ef343d2168a976ab5974e6f22bfb29a0a44bb1d27dd4046d8d33b03"
Oct 30 01:10:21.954413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.954290    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0be71d9ef343d2168a976ab5974e6f22bfb29a0a44bb1d27dd4046d8d33b03"} err="failed to get container status \"df0be71d9ef343d2168a976ab5974e6f22bfb29a0a44bb1d27dd4046d8d33b03\": rpc error: code = NotFound desc = could not find container \"df0be71d9ef343d2168a976ab5974e6f22bfb29a0a44bb1d27dd4046d8d33b03\": container with ID starting with df0be71d9ef343d2168a976ab5974e6f22bfb29a0a44bb1d27dd4046d8d33b03 not found: ID does not exist"
Oct 30 01:10:21.993654 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:21.993617    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"]
Oct 30 01:10:22.002910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:22.002877    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ltz8j"]
Oct 30 01:10:22.675436 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:22.675392    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"]
Oct 30 01:10:22.675699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:22.675671    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl" podUID="7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" containerName="istio-proxy" containerID="cri-o://63fb51d1789a4aa53e528ed5d079550ca126b5f77774cd9b93405eb79f753094" gracePeriod=30
Oct 30 01:10:23.423778 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:23.423736    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae18a2f-da25-474a-af36-6be757b43311" path="/var/lib/kubelet/pods/dae18a2f-da25-474a-af36-6be757b43311/volumes"
Oct 30 01:10:25.039488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.039450    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"]
Oct 30 01:10:25.039988 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.039703    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f" podUID="7b917a55-0178-422d-919a-c7499e12aff3" containerName="discovery" containerID="cri-o://4d82b3e069898bfac687f9baab0e391c4d1171835d0bf858cfc06b93177c3190" gracePeriod=30
Oct 30 01:10:25.130651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.130607    4425 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-krk8f container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:8080/ready\": dial tcp 10.133.0.106:8080: connect: connection refused" start-of-body=
Oct 30 01:10:25.130852 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.130681    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f" podUID="7b917a55-0178-422d-919a-c7499e12aff3" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.106:8080/ready\": dial tcp 10.133.0.106:8080: connect: connection refused"
Oct 30 01:10:25.400350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.400314    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:10:25.486878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.486842    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b917a55-0178-422d-919a-c7499e12aff3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b917a55-0178-422d-919a-c7499e12aff3-local-certs\") pod \"7b917a55-0178-422d-919a-c7499e12aff3\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") "
Oct 30 01:10:25.487095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.486950    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b917a55-0178-422d-919a-c7499e12aff3-kube-api-access-g6fsl\" (UniqueName: \"kubernetes.io/projected/7b917a55-0178-422d-919a-c7499e12aff3-kube-api-access-g6fsl\") pod \"7b917a55-0178-422d-919a-c7499e12aff3\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") "
Oct 30 01:10:25.487095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.487030    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b917a55-0178-422d-919a-c7499e12aff3-istio-token\" (UniqueName: \"kubernetes.io/projected/7b917a55-0178-422d-919a-c7499e12aff3-istio-token\") pod \"7b917a55-0178-422d-919a-c7499e12aff3\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") "
Oct 30 01:10:25.487264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.487124    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-istio-csr-dns-cert\") pod \"7b917a55-0178-422d-919a-c7499e12aff3\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") "
Oct 30 01:10:25.487264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.487145    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b917a55-0178-422d-919a-c7499e12aff3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b917a55-0178-422d-919a-c7499e12aff3-istio-csr-ca-configmap\") pod \"7b917a55-0178-422d-919a-c7499e12aff3\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") "
Oct 30 01:10:25.487264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.487225    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-istio-kubeconfig\") pod \"7b917a55-0178-422d-919a-c7499e12aff3\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") "
Oct 30 01:10:25.487264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.487258    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-cacerts\" (UniqueName: \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-cacerts\") pod \"7b917a55-0178-422d-919a-c7499e12aff3\" (UID: \"7b917a55-0178-422d-919a-c7499e12aff3\") "
Oct 30 01:10:25.487586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.487548    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b917a55-0178-422d-919a-c7499e12aff3-istio-csr-ca-configmap" pod "7b917a55-0178-422d-919a-c7499e12aff3" (UID: "7b917a55-0178-422d-919a-c7499e12aff3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:25.487945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.487922    4425 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b917a55-0178-422d-919a-c7499e12aff3-istio-csr-ca-configmap\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:25.489750 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.489720    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b917a55-0178-422d-919a-c7499e12aff3-istio-token" pod "7b917a55-0178-422d-919a-c7499e12aff3" (UID: "7b917a55-0178-422d-919a-c7499e12aff3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:25.489859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.489719    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b917a55-0178-422d-919a-c7499e12aff3-kube-api-access-g6fsl" pod "7b917a55-0178-422d-919a-c7499e12aff3" (UID: "7b917a55-0178-422d-919a-c7499e12aff3"). InnerVolumeSpecName "kube-api-access-g6fsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:25.489918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.489860    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-istio-csr-dns-cert" pod "7b917a55-0178-422d-919a-c7499e12aff3" (UID: "7b917a55-0178-422d-919a-c7499e12aff3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:10:25.489918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.489895    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b917a55-0178-422d-919a-c7499e12aff3-local-certs" pod "7b917a55-0178-422d-919a-c7499e12aff3" (UID: "7b917a55-0178-422d-919a-c7499e12aff3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:25.489995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.489960    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-istio-kubeconfig" pod "7b917a55-0178-422d-919a-c7499e12aff3" (UID: "7b917a55-0178-422d-919a-c7499e12aff3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:10:25.490251 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.490225    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-cacerts" pod "7b917a55-0178-422d-919a-c7499e12aff3" (UID: "7b917a55-0178-422d-919a-c7499e12aff3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:10:25.589754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.589708    4425 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-istio-csr-dns-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:25.589754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.589745    4425 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-istio-kubeconfig\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:25.589754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.589759    4425 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7b917a55-0178-422d-919a-c7499e12aff3-cacerts\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:25.590021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.589769    4425 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b917a55-0178-422d-919a-c7499e12aff3-local-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:25.590021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.589781    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6fsl\" (UniqueName: \"kubernetes.io/projected/7b917a55-0178-422d-919a-c7499e12aff3-kube-api-access-g6fsl\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:25.590021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.589793    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b917a55-0178-422d-919a-c7499e12aff3-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:25.971696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.971651    4425 generic.go:356] "Generic (PLEG): container finished" podID="7b917a55-0178-422d-919a-c7499e12aff3" containerID="4d82b3e069898bfac687f9baab0e391c4d1171835d0bf858cfc06b93177c3190" exitCode=0
Oct 30 01:10:25.971885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.971707    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"
Oct 30 01:10:25.971885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.971723    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f" event={"ID":"7b917a55-0178-422d-919a-c7499e12aff3","Type":"ContainerDied","Data":"4d82b3e069898bfac687f9baab0e391c4d1171835d0bf858cfc06b93177c3190"}
Oct 30 01:10:25.971885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.971794    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f" event={"ID":"7b917a55-0178-422d-919a-c7499e12aff3","Type":"ContainerDied","Data":"dd042bd683991308ae4d166c572f6c8033703fe7990a81db788ec56e82d01489"}
Oct 30 01:10:25.971885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.971819    4425 scope.go:122] "RemoveContainer" containerID="4d82b3e069898bfac687f9baab0e391c4d1171835d0bf858cfc06b93177c3190"
Oct 30 01:10:25.983293 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.983268    4425 scope.go:122] "RemoveContainer" containerID="4d82b3e069898bfac687f9baab0e391c4d1171835d0bf858cfc06b93177c3190"
Oct 30 01:10:25.983622 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:10:25.983602    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d82b3e069898bfac687f9baab0e391c4d1171835d0bf858cfc06b93177c3190\": container with ID starting with 4d82b3e069898bfac687f9baab0e391c4d1171835d0bf858cfc06b93177c3190 not found: ID does not exist" containerID="4d82b3e069898bfac687f9baab0e391c4d1171835d0bf858cfc06b93177c3190"
Oct 30 01:10:25.983704 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:25.983629    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d82b3e069898bfac687f9baab0e391c4d1171835d0bf858cfc06b93177c3190"} err="failed to get container status \"4d82b3e069898bfac687f9baab0e391c4d1171835d0bf858cfc06b93177c3190\": rpc error: code = NotFound desc = could not find container \"4d82b3e069898bfac687f9baab0e391c4d1171835d0bf858cfc06b93177c3190\": container with ID starting with 4d82b3e069898bfac687f9baab0e391c4d1171835d0bf858cfc06b93177c3190 not found: ID does not exist"
Oct 30 01:10:26.030322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:26.030276    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"]
Oct 30 01:10:26.039597 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:26.039561    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-krk8f"]
Oct 30 01:10:27.422922 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:27.422880    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b917a55-0178-422d-919a-c7499e12aff3" path="/var/lib/kubelet/pods/7b917a55-0178-422d-919a-c7499e12aff3/volumes"
Oct 30 01:10:27.978510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:27.978481    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 01:10:27.988765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:27.988729    4425 generic.go:356] "Generic (PLEG): container finished" podID="7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" containerID="63fb51d1789a4aa53e528ed5d079550ca126b5f77774cd9b93405eb79f753094" exitCode=0
Oct 30 01:10:27.988965 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:27.988781    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"
Oct 30 01:10:27.988965 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:27.988819    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl" event={"ID":"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b","Type":"ContainerDied","Data":"63fb51d1789a4aa53e528ed5d079550ca126b5f77774cd9b93405eb79f753094"}
Oct 30 01:10:27.988965 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:27.988892    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl" event={"ID":"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b","Type":"ContainerDied","Data":"75411d3469d4c80a2450230b271210c5e5a4738b73b6ba539292039a6ee79d70"}
Oct 30 01:10:27.988965 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:27.988937    4425 scope.go:122] "RemoveContainer" containerID="63fb51d1789a4aa53e528ed5d079550ca126b5f77774cd9b93405eb79f753094"
Oct 30 01:10:27.999496 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:27.999472    4425 scope.go:122] "RemoveContainer" containerID="63fb51d1789a4aa53e528ed5d079550ca126b5f77774cd9b93405eb79f753094"
Oct 30 01:10:27.999851 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:10:27.999828    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63fb51d1789a4aa53e528ed5d079550ca126b5f77774cd9b93405eb79f753094\": container with ID starting with 63fb51d1789a4aa53e528ed5d079550ca126b5f77774cd9b93405eb79f753094 not found: ID does not exist" containerID="63fb51d1789a4aa53e528ed5d079550ca126b5f77774cd9b93405eb79f753094"
Oct 30 01:10:27.999972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:27.999861    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63fb51d1789a4aa53e528ed5d079550ca126b5f77774cd9b93405eb79f753094"} err="failed to get container status \"63fb51d1789a4aa53e528ed5d079550ca126b5f77774cd9b93405eb79f753094\": rpc error: code = NotFound desc = could not find container \"63fb51d1789a4aa53e528ed5d079550ca126b5f77774cd9b93405eb79f753094\": container with ID starting with 63fb51d1789a4aa53e528ed5d079550ca126b5f77774cd9b93405eb79f753094 not found: ID does not exist"
Oct 30 01:10:28.020813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.020765    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-data\") pod \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") "
Oct 30 01:10:28.020813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.020823    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-token\" (UniqueName: \"kubernetes.io/projected/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-token\") pod \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") "
Oct 30 01:10:28.021101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.020878    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-workload-certs\") pod \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") "
Oct 30 01:10:28.021101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.020978    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-podinfo\") pod \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") "
Oct 30 01:10:28.021101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.021021    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-envoy\") pod \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") "
Oct 30 01:10:28.021101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.021066    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istiod-ca-cert\") pod \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") "
Oct 30 01:10:28.021353 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.021099    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-data" pod "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" (UID: "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:28.021353 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.021171    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-credential-socket\") pod \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") "
Oct 30 01:10:28.021353 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.021218    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-workload-socket\") pod \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") "
Oct 30 01:10:28.021353 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.021323    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-kube-api-access-64zhw\" (UniqueName: \"kubernetes.io/projected/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-kube-api-access-64zhw\") pod \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\" (UID: \"7f5a0c33-0c8e-45a8-9fcb-433a6132af2b\") "
Oct 30 01:10:28.021568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.021485    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istiod-ca-cert" pod "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" (UID: "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:28.021568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.021535    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-workload-certs" pod "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" (UID: "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:28.021684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.021558    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-credential-socket" pod "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" (UID: "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:28.021742 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.021712    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-workload-socket" pod "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" (UID: "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:28.022012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.021989    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:28.022105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.022016    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:28.022105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.022030    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:28.022105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.022043    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:28.022105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.022057    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:28.023945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.023918    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-podinfo" pod "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" (UID: "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:10:28.023945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.023935    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-token" pod "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" (UID: "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:28.024299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.024270    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-kube-api-access-64zhw" pod "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" (UID: "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b"). InnerVolumeSpecName "kube-api-access-64zhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:28.024481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.024463    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-envoy" pod "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" (UID: "7f5a0c33-0c8e-45a8-9fcb-433a6132af2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:28.123884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.123830    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:28.123884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.123880    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:28.124125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.123936    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64zhw\" (UniqueName: \"kubernetes.io/projected/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-kube-api-access-64zhw\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:28.124125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.123953    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:28.335889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.335849    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"]
Oct 30 01:10:28.345124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:28.345081    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-rk8hl"]
Oct 30 01:10:29.425176 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:29.425130    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" path="/var/lib/kubelet/pods/7f5a0c33-0c8e-45a8-9fcb-433a6132af2b/volumes"
Oct 30 01:10:32.553746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.553709    4425 scope.go:122] "RemoveContainer" containerID="9ed5508f286891875cfcbb8500f759b5aeeac5fd037bb5cf7312fb5fd995ee62"
Oct 30 01:10:32.564505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.564480    4425 scope.go:122] "RemoveContainer" containerID="d7fd2449972d87c17ccc68a6db36ed683c7ffda2703c98c6983171a720c97c28"
Oct 30 01:10:32.574586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.574564    4425 scope.go:122] "RemoveContainer" containerID="2f77a27ed2147c56d0511c5c93d97f0c5c9bbb01c39e78e5fb8799f877feb9db"
Oct 30 01:10:32.585435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.585403    4425 scope.go:122] "RemoveContainer" containerID="33cb77879e33a468e610c077647e648bfa92fe4a846862c2cb66def0b9bed699"
Oct 30 01:10:32.898962 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.898864    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"]
Oct 30 01:10:32.899454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.899437    4425 cpu_manager.go:414] "containerMap: removing container" podUID="23d957ad-df49-489b-b7ac-07a3427b01f5" containerName="revoke-keys"
Oct 30 01:10:32.899831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.899454    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d957ad-df49-489b-b7ac-07a3427b01f5" containerName="revoke-keys"
Oct 30 01:10:32.899831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.899476    4425 cpu_manager.go:414] "containerMap: removing container" podUID="dae18a2f-da25-474a-af36-6be757b43311" containerName="istio-proxy"
Oct 30 01:10:32.899831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.899483    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae18a2f-da25-474a-af36-6be757b43311" containerName="istio-proxy"
Oct 30 01:10:32.899831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.899494    4425 cpu_manager.go:414] "containerMap: removing container" podUID="cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" containerName="istio-proxy"
Oct 30 01:10:32.899831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.899499    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" containerName="istio-proxy"
Oct 30 01:10:32.899831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.899518    4425 cpu_manager.go:414] "containerMap: removing container" podUID="7b917a55-0178-422d-919a-c7499e12aff3" containerName="discovery"
Oct 30 01:10:32.899831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.899524    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b917a55-0178-422d-919a-c7499e12aff3" containerName="discovery"
Oct 30 01:10:32.899831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.899535    4425 cpu_manager.go:414] "containerMap: removing container" podUID="7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" containerName="istio-proxy"
Oct 30 01:10:32.899831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.899540    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" containerName="istio-proxy"
Oct 30 01:10:32.899831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.899650    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b917a55-0178-422d-919a-c7499e12aff3" containerName="discovery"
Oct 30 01:10:32.899831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.899663    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5a0c33-0c8e-45a8-9fcb-433a6132af2b" containerName="istio-proxy"
Oct 30 01:10:32.899831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.899676    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d957ad-df49-489b-b7ac-07a3427b01f5" containerName="revoke-keys"
Oct 30 01:10:32.899831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.899686    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae18a2f-da25-474a-af36-6be757b43311" containerName="istio-proxy"
Oct 30 01:10:32.899831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.899694    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee90c29-d7d2-41cc-b02d-5cbcf06c9b29" containerName="istio-proxy"
Oct 30 01:10:32.905264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.905189    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"
Oct 30 01:10:32.924467 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.924424    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"]
Oct 30 01:10:32.930857 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.930825    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-5f8cbc\""
Oct 30 01:10:32.977812 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.977769    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"]
Oct 30 01:10:32.981838 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.981812    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"
Oct 30 01:10:32.989422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.989392    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f16cc640-5e15-45f9-87ba-942cf8a21325-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v\" (UID: \"f16cc640-5e15-45f9-87ba-942cf8a21325\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"
Oct 30 01:10:32.989520 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.989486    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9n22\" (UniqueName: \"kubernetes.io/projected/f16cc640-5e15-45f9-87ba-942cf8a21325-kube-api-access-w9n22\") pod \"payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v\" (UID: \"f16cc640-5e15-45f9-87ba-942cf8a21325\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"
Oct 30 01:10:32.993893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:32.993862    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-5f8cbc-dockercfg-bhnqv\""
Oct 30 01:10:33.000667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:33.000636    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"]
Oct 30 01:10:33.096190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:33.093011    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cspqt\" (UniqueName: \"kubernetes.io/projected/9eee6597-2b6b-4760-9144-4901fc0c342b-kube-api-access-cspqt\") pod \"payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z\" (UID: \"9eee6597-2b6b-4760-9144-4901fc0c342b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"
Oct 30 01:10:33.105896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:33.105853    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f16cc640-5e15-45f9-87ba-942cf8a21325-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v\" (UID: \"f16cc640-5e15-45f9-87ba-942cf8a21325\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"
Oct 30 01:10:33.107292 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:33.107263    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9n22\" (UniqueName: \"kubernetes.io/projected/f16cc640-5e15-45f9-87ba-942cf8a21325-kube-api-access-w9n22\") pod \"payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v\" (UID: \"f16cc640-5e15-45f9-87ba-942cf8a21325\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"
Oct 30 01:10:33.107951 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:33.107927    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eee6597-2b6b-4760-9144-4901fc0c342b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z\" (UID: \"9eee6597-2b6b-4760-9144-4901fc0c342b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"
Oct 30 01:10:33.108322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:33.107093    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f16cc640-5e15-45f9-87ba-942cf8a21325-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v\" (UID: \"f16cc640-5e15-45f9-87ba-942cf8a21325\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"
Oct 30 01:10:33.130749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:33.130719    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9n22\" (UniqueName: \"kubernetes.io/projected/f16cc640-5e15-45f9-87ba-942cf8a21325-kube-api-access-w9n22\") pod \"payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v\" (UID: \"f16cc640-5e15-45f9-87ba-942cf8a21325\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"
Oct 30 01:10:33.209312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:33.209215    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cspqt\" (UniqueName: \"kubernetes.io/projected/9eee6597-2b6b-4760-9144-4901fc0c342b-kube-api-access-cspqt\") pod \"payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z\" (UID: \"9eee6597-2b6b-4760-9144-4901fc0c342b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"
Oct 30 01:10:33.209476 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:33.209398    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eee6597-2b6b-4760-9144-4901fc0c342b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z\" (UID: \"9eee6597-2b6b-4760-9144-4901fc0c342b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"
Oct 30 01:10:33.210125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:33.210103    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eee6597-2b6b-4760-9144-4901fc0c342b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z\" (UID: \"9eee6597-2b6b-4760-9144-4901fc0c342b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"
Oct 30 01:10:33.220501 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:33.220463    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"
Oct 30 01:10:33.231328 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:33.231291    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cspqt\" (UniqueName: \"kubernetes.io/projected/9eee6597-2b6b-4760-9144-4901fc0c342b-kube-api-access-cspqt\") pod \"payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z\" (UID: \"9eee6597-2b6b-4760-9144-4901fc0c342b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"
Oct 30 01:10:33.297510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:33.297200    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"
Oct 30 01:10:33.531691 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:33.531658    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"]
Oct 30 01:10:33.533651 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:10:33.533621    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16cc640_5e15_45f9_87ba_942cf8a21325.slice/crio-014c1bd4f0de8a60da38b161254b0a7de46c41d20e19d0ea3ac0d790395fcbc7 WatchSource:0}: Error finding container 014c1bd4f0de8a60da38b161254b0a7de46c41d20e19d0ea3ac0d790395fcbc7: Status 404 returned error can't find the container with id 014c1bd4f0de8a60da38b161254b0a7de46c41d20e19d0ea3ac0d790395fcbc7
Oct 30 01:10:33.610525 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:33.610498    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"]
Oct 30 01:10:33.611981 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:10:33.611943    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eee6597_2b6b_4760_9144_4901fc0c342b.slice/crio-177ebe244f4f0f0f5c5429335f1a81ce7208038279e35f877ef743425a34f0aa WatchSource:0}: Error finding container 177ebe244f4f0f0f5c5429335f1a81ce7208038279e35f877ef743425a34f0aa: Status 404 returned error can't find the container with id 177ebe244f4f0f0f5c5429335f1a81ce7208038279e35f877ef743425a34f0aa
Oct 30 01:10:34.036083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.036044    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v" event={"ID":"f16cc640-5e15-45f9-87ba-942cf8a21325","Type":"ContainerStarted","Data":"06b1734144dd952c8f6399f675b9c589f84ceaef0d6a1df1460b4f185b1efc68"}
Oct 30 01:10:34.036339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.036098    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v" event={"ID":"f16cc640-5e15-45f9-87ba-942cf8a21325","Type":"ContainerStarted","Data":"014c1bd4f0de8a60da38b161254b0a7de46c41d20e19d0ea3ac0d790395fcbc7"}
Oct 30 01:10:34.036339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.036299    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"
Oct 30 01:10:34.037858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.037832    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z" event={"ID":"9eee6597-2b6b-4760-9144-4901fc0c342b","Type":"ContainerStarted","Data":"6b3d3d9e1570205ae2873503bd0449139d6751fdb31f79d859085b164844f4f9"}
Oct 30 01:10:34.037990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.037878    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z" event={"ID":"9eee6597-2b6b-4760-9144-4901fc0c342b","Type":"ContainerStarted","Data":"177ebe244f4f0f0f5c5429335f1a81ce7208038279e35f877ef743425a34f0aa"}
Oct 30 01:10:34.037990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.037953    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"
Oct 30 01:10:34.074235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.074149    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v" podStartSLOduration=2.074122288 podStartE2EDuration="2.074122288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:34.071750201 +0000 UTC m=+1742.821405512" watchObservedRunningTime="2026-10-30 01:10:34.074122288 +0000 UTC m=+1742.823777588"
Oct 30 01:10:34.090343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.090302    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"]
Oct 30 01:10:34.094608 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.094580    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"
Oct 30 01:10:34.107617 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.107586    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:10:34.108528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.108480    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z" podStartSLOduration=2.1084674310000002 podStartE2EDuration="2.108467431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:34.107633239 +0000 UTC m=+1742.857288590" watchObservedRunningTime="2026-10-30 01:10:34.108467431 +0000 UTC m=+1742.858122825"
Oct 30 01:10:34.110431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.110402    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:10:34.114592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.114561    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"]
Oct 30 01:10:34.158095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.158049    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:10:34.158357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.158151    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:10:34.223041 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.222339    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d16bbcf-0633-4890-9e51-e83cfffb38d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq\" (UID: \"7d16bbcf-0633-4890-9e51-e83cfffb38d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"
Oct 30 01:10:34.223041 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.222456    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7z7r\" (UniqueName: \"kubernetes.io/projected/7d16bbcf-0633-4890-9e51-e83cfffb38d4-kube-api-access-l7z7r\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq\" (UID: \"7d16bbcf-0633-4890-9e51-e83cfffb38d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"
Oct 30 01:10:34.223041 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.222505    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d16bbcf-0633-4890-9e51-e83cfffb38d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq\" (UID: \"7d16bbcf-0633-4890-9e51-e83cfffb38d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"
Oct 30 01:10:34.323855 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.323759    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7z7r\" (UniqueName: \"kubernetes.io/projected/7d16bbcf-0633-4890-9e51-e83cfffb38d4-kube-api-access-l7z7r\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq\" (UID: \"7d16bbcf-0633-4890-9e51-e83cfffb38d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"
Oct 30 01:10:34.323855 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.323802    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d16bbcf-0633-4890-9e51-e83cfffb38d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq\" (UID: \"7d16bbcf-0633-4890-9e51-e83cfffb38d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"
Oct 30 01:10:34.324113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.323943    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d16bbcf-0633-4890-9e51-e83cfffb38d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq\" (UID: \"7d16bbcf-0633-4890-9e51-e83cfffb38d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"
Oct 30 01:10:34.324809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.324775    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d16bbcf-0633-4890-9e51-e83cfffb38d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq\" (UID: \"7d16bbcf-0633-4890-9e51-e83cfffb38d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"
Oct 30 01:10:34.326419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.326392    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d16bbcf-0633-4890-9e51-e83cfffb38d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq\" (UID: \"7d16bbcf-0633-4890-9e51-e83cfffb38d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"
Oct 30 01:10:34.349383 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.349350    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7z7r\" (UniqueName: \"kubernetes.io/projected/7d16bbcf-0633-4890-9e51-e83cfffb38d4-kube-api-access-l7z7r\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq\" (UID: \"7d16bbcf-0633-4890-9e51-e83cfffb38d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"
Oct 30 01:10:34.409882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.409842    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"
Oct 30 01:10:34.482504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.479643    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"]
Oct 30 01:10:34.488317 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.488285    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"
Oct 30 01:10:34.502391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.502349    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-88a39d\""
Oct 30 01:10:34.507204 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.507122    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"]
Oct 30 01:10:34.538148 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.538111    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"]
Oct 30 01:10:34.543102 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.543071    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"
Oct 30 01:10:34.559231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.559195    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-88a39d-dockercfg-s8dbt\""
Oct 30 01:10:34.562609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.562571    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"]
Oct 30 01:10:34.634475 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.634379    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e79b33e-4d54-4f80-8363-70da08b12fdc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn\" (UID: \"3e79b33e-4d54-4f80-8363-70da08b12fdc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"
Oct 30 01:10:34.634475 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.634428    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjfth\" (UniqueName: \"kubernetes.io/projected/3e79b33e-4d54-4f80-8363-70da08b12fdc-kube-api-access-xjfth\") pod \"payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn\" (UID: \"3e79b33e-4d54-4f80-8363-70da08b12fdc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"
Oct 30 01:10:34.634900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.634628    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfr8s\" (UniqueName: \"kubernetes.io/projected/a7aae476-59c5-4012-b346-ce106823d157-kube-api-access-sfr8s\") pod \"payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l\" (UID: \"a7aae476-59c5-4012-b346-ce106823d157\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"
Oct 30 01:10:34.634900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.634678    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7aae476-59c5-4012-b346-ce106823d157-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l\" (UID: \"a7aae476-59c5-4012-b346-ce106823d157\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"
Oct 30 01:10:34.736015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.735969    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e79b33e-4d54-4f80-8363-70da08b12fdc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn\" (UID: \"3e79b33e-4d54-4f80-8363-70da08b12fdc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"
Oct 30 01:10:34.736235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.736025    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjfth\" (UniqueName: \"kubernetes.io/projected/3e79b33e-4d54-4f80-8363-70da08b12fdc-kube-api-access-xjfth\") pod \"payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn\" (UID: \"3e79b33e-4d54-4f80-8363-70da08b12fdc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"
Oct 30 01:10:34.736235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.736138    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfr8s\" (UniqueName: \"kubernetes.io/projected/a7aae476-59c5-4012-b346-ce106823d157-kube-api-access-sfr8s\") pod \"payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l\" (UID: \"a7aae476-59c5-4012-b346-ce106823d157\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"
Oct 30 01:10:34.736235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.736200    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7aae476-59c5-4012-b346-ce106823d157-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l\" (UID: \"a7aae476-59c5-4012-b346-ce106823d157\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"
Oct 30 01:10:34.737011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.736976    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e79b33e-4d54-4f80-8363-70da08b12fdc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn\" (UID: \"3e79b33e-4d54-4f80-8363-70da08b12fdc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"
Oct 30 01:10:34.737202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.737119    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7aae476-59c5-4012-b346-ce106823d157-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l\" (UID: \"a7aae476-59c5-4012-b346-ce106823d157\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"
Oct 30 01:10:34.741450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.741420    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"]
Oct 30 01:10:34.742894 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:10:34.742868    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d16bbcf_0633_4890_9e51_e83cfffb38d4.slice/crio-a1a053bfbc0951699f666594d683f351ce4f5fbfda9b20f4f974b78a0416c303 WatchSource:0}: Error finding container a1a053bfbc0951699f666594d683f351ce4f5fbfda9b20f4f974b78a0416c303: Status 404 returned error can't find the container with id a1a053bfbc0951699f666594d683f351ce4f5fbfda9b20f4f974b78a0416c303
Oct 30 01:10:34.757650 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.757575    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfr8s\" (UniqueName: \"kubernetes.io/projected/a7aae476-59c5-4012-b346-ce106823d157-kube-api-access-sfr8s\") pod \"payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l\" (UID: \"a7aae476-59c5-4012-b346-ce106823d157\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"
Oct 30 01:10:34.757846 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.757718    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjfth\" (UniqueName: \"kubernetes.io/projected/3e79b33e-4d54-4f80-8363-70da08b12fdc-kube-api-access-xjfth\") pod \"payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn\" (UID: \"3e79b33e-4d54-4f80-8363-70da08b12fdc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"
Oct 30 01:10:34.807851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.807805    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"
Oct 30 01:10:34.861926 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:34.861883    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"
Oct 30 01:10:35.046442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:35.046404    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq" event={"ID":"7d16bbcf-0633-4890-9e51-e83cfffb38d4","Type":"ContainerStarted","Data":"a1a053bfbc0951699f666594d683f351ce4f5fbfda9b20f4f974b78a0416c303"}
Oct 30 01:10:35.123915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:35.123873    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"]
Oct 30 01:10:35.158849 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:10:35.157253    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e79b33e_4d54_4f80_8363_70da08b12fdc.slice/crio-996e343d117316913455ac315281c568bc8543b8c706ff252f8b1bc9d8f648ee WatchSource:0}: Error finding container 996e343d117316913455ac315281c568bc8543b8c706ff252f8b1bc9d8f648ee: Status 404 returned error can't find the container with id 996e343d117316913455ac315281c568bc8543b8c706ff252f8b1bc9d8f648ee
Oct 30 01:10:35.196194 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:35.196130    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"]
Oct 30 01:10:35.209849 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:10:35.209815    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7aae476_59c5_4012_b346_ce106823d157.slice/crio-649a2eef2105e4482b06302647c4608e57c84c3c900f4cabe2a58f3ff1036b08 WatchSource:0}: Error finding container 649a2eef2105e4482b06302647c4608e57c84c3c900f4cabe2a58f3ff1036b08: Status 404 returned error can't find the container with id 649a2eef2105e4482b06302647c4608e57c84c3c900f4cabe2a58f3ff1036b08
Oct 30 01:10:36.056149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:36.056102    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn" event={"ID":"3e79b33e-4d54-4f80-8363-70da08b12fdc","Type":"ContainerStarted","Data":"ede835f6d8cbd06ae7c5758fb6a5485a3e9f3c26f779dbc21e12bdcd325d82d5"}
Oct 30 01:10:36.056671 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:36.056192    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn" event={"ID":"3e79b33e-4d54-4f80-8363-70da08b12fdc","Type":"ContainerStarted","Data":"996e343d117316913455ac315281c568bc8543b8c706ff252f8b1bc9d8f648ee"}
Oct 30 01:10:36.056671 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:36.056217    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"
Oct 30 01:10:36.059833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:36.059790    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l" event={"ID":"a7aae476-59c5-4012-b346-ce106823d157","Type":"ContainerStarted","Data":"18c9d60fda30c8eaf610603666f47e30cd70aa7f19cb66a0a9a2f6bb0bfc8c83"}
Oct 30 01:10:36.060030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:36.059862    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l" event={"ID":"a7aae476-59c5-4012-b346-ce106823d157","Type":"ContainerStarted","Data":"649a2eef2105e4482b06302647c4608e57c84c3c900f4cabe2a58f3ff1036b08"}
Oct 30 01:10:36.060030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:36.059895    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"
Oct 30 01:10:36.061657 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:36.061626    4425 generic.go:356] "Generic (PLEG): container finished" podID="7d16bbcf-0633-4890-9e51-e83cfffb38d4" containerID="c17fc3babd856b59729fd9a0973647464e38252ef83fa011ec09e21cb348eb33" exitCode=0
Oct 30 01:10:36.061807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:36.061669    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq" event={"ID":"7d16bbcf-0633-4890-9e51-e83cfffb38d4","Type":"ContainerDied","Data":"c17fc3babd856b59729fd9a0973647464e38252ef83fa011ec09e21cb348eb33"}
Oct 30 01:10:36.115952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:36.115888    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn" podStartSLOduration=2.115868607 podStartE2EDuration="2.115868607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:36.112937871 +0000 UTC m=+1744.862593179" watchObservedRunningTime="2026-10-30 01:10:36.115868607 +0000 UTC m=+1744.865523916"
Oct 30 01:10:36.193932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:36.193876    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l" podStartSLOduration=2.193861298 podStartE2EDuration="2.193861298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:36.192945802 +0000 UTC m=+1744.942601113" watchObservedRunningTime="2026-10-30 01:10:36.193861298 +0000 UTC m=+1744.943516607"
Oct 30 01:10:37.228864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:37.228831    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"
Oct 30 01:10:37.371522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:37.371430    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d16bbcf-0633-4890-9e51-e83cfffb38d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d16bbcf-0633-4890-9e51-e83cfffb38d4-maas-api-xks-ca\") pod \"7d16bbcf-0633-4890-9e51-e83cfffb38d4\" (UID: \"7d16bbcf-0633-4890-9e51-e83cfffb38d4\") "
Oct 30 01:10:37.371522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:37.371495    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d16bbcf-0633-4890-9e51-e83cfffb38d4-kube-api-access-l7z7r\" (UniqueName: \"kubernetes.io/projected/7d16bbcf-0633-4890-9e51-e83cfffb38d4-kube-api-access-l7z7r\") pod \"7d16bbcf-0633-4890-9e51-e83cfffb38d4\" (UID: \"7d16bbcf-0633-4890-9e51-e83cfffb38d4\") "
Oct 30 01:10:37.371734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:37.371559    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d16bbcf-0633-4890-9e51-e83cfffb38d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d16bbcf-0633-4890-9e51-e83cfffb38d4-maas-api-openshift-ca\") pod \"7d16bbcf-0633-4890-9e51-e83cfffb38d4\" (UID: \"7d16bbcf-0633-4890-9e51-e83cfffb38d4\") "
Oct 30 01:10:37.371964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:37.371939    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d16bbcf-0633-4890-9e51-e83cfffb38d4-maas-api-openshift-ca" pod "7d16bbcf-0633-4890-9e51-e83cfffb38d4" (UID: "7d16bbcf-0633-4890-9e51-e83cfffb38d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:37.373779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:37.373741    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d16bbcf-0633-4890-9e51-e83cfffb38d4-maas-api-xks-ca" pod "7d16bbcf-0633-4890-9e51-e83cfffb38d4" (UID: "7d16bbcf-0633-4890-9e51-e83cfffb38d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:10:37.373888 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:37.373850    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d16bbcf-0633-4890-9e51-e83cfffb38d4-kube-api-access-l7z7r" pod "7d16bbcf-0633-4890-9e51-e83cfffb38d4" (UID: "7d16bbcf-0633-4890-9e51-e83cfffb38d4"). InnerVolumeSpecName "kube-api-access-l7z7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:37.473311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:37.473267    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d16bbcf-0633-4890-9e51-e83cfffb38d4-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:37.473311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:37.473302    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7z7r\" (UniqueName: \"kubernetes.io/projected/7d16bbcf-0633-4890-9e51-e83cfffb38d4-kube-api-access-l7z7r\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:37.473311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:37.473317    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d16bbcf-0633-4890-9e51-e83cfffb38d4-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:10:37.556832 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:10:37.556787    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:10:38.078142 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:38.078112    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"
Oct 30 01:10:38.078370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:38.078151    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq" event={"ID":"7d16bbcf-0633-4890-9e51-e83cfffb38d4","Type":"ContainerDied","Data":"a1a053bfbc0951699f666594d683f351ce4f5fbfda9b20f4f974b78a0416c303"}
Oct 30 01:10:38.078370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:38.078238    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1a053bfbc0951699f666594d683f351ce4f5fbfda9b20f4f974b78a0416c303"
Oct 30 01:10:39.200375 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:39.200332    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:10:45.059002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:45.057866    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"
Oct 30 01:10:45.061563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:45.061534    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"
Oct 30 01:10:47.072835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:47.072804    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"
Oct 30 01:10:47.073466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:47.072894    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"
Oct 30 01:10:53.188674 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:53.188633    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"]
Oct 30 01:10:53.197996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:53.197963    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d4ea-dqpzq"]
Oct 30 01:10:53.424878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:10:53.424842    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d16bbcf-0633-4890-9e51-e83cfffb38d4" path="/var/lib/kubelet/pods/7d16bbcf-0633-4890-9e51-e83cfffb38d4/volumes"
Oct 30 01:11:05.288569 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.288522    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"]
Oct 30 01:11:05.289287 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.289268    4425 cpu_manager.go:414] "containerMap: removing container" podUID="7d16bbcf-0633-4890-9e51-e83cfffb38d4" containerName="revoke-keys"
Oct 30 01:11:05.289353 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.289288    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d16bbcf-0633-4890-9e51-e83cfffb38d4" containerName="revoke-keys"
Oct 30 01:11:05.289409 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.289400    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d16bbcf-0633-4890-9e51-e83cfffb38d4" containerName="revoke-keys"
Oct 30 01:11:05.293251 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.293229    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"
Oct 30 01:11:05.306067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.306038    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:05.315670 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.315632    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:11:05.315670 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.315637    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:11:05.315887 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.315726    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:11:05.319316 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.319278    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"]
Oct 30 01:11:05.466029 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.465975    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7l5p\" (UniqueName: \"kubernetes.io/projected/2e225afe-9707-464a-8d42-20c33b7c5390-kube-api-access-r7l5p\") pod \"maas-api-revoke-sub-be79cd53-nkbs5\" (UID: \"2e225afe-9707-464a-8d42-20c33b7c5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"
Oct 30 01:11:05.466029 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.466048    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e225afe-9707-464a-8d42-20c33b7c5390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be79cd53-nkbs5\" (UID: \"2e225afe-9707-464a-8d42-20c33b7c5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"
Oct 30 01:11:05.466406 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.466339    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e225afe-9707-464a-8d42-20c33b7c5390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be79cd53-nkbs5\" (UID: \"2e225afe-9707-464a-8d42-20c33b7c5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"
Oct 30 01:11:05.568134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.568029    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e225afe-9707-464a-8d42-20c33b7c5390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be79cd53-nkbs5\" (UID: \"2e225afe-9707-464a-8d42-20c33b7c5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"
Oct 30 01:11:05.568134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.568106    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7l5p\" (UniqueName: \"kubernetes.io/projected/2e225afe-9707-464a-8d42-20c33b7c5390-kube-api-access-r7l5p\") pod \"maas-api-revoke-sub-be79cd53-nkbs5\" (UID: \"2e225afe-9707-464a-8d42-20c33b7c5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"
Oct 30 01:11:05.568441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.568148    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e225afe-9707-464a-8d42-20c33b7c5390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be79cd53-nkbs5\" (UID: \"2e225afe-9707-464a-8d42-20c33b7c5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"
Oct 30 01:11:05.568891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.568867    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e225afe-9707-464a-8d42-20c33b7c5390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be79cd53-nkbs5\" (UID: \"2e225afe-9707-464a-8d42-20c33b7c5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"
Oct 30 01:11:05.570703 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.570676    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e225afe-9707-464a-8d42-20c33b7c5390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be79cd53-nkbs5\" (UID: \"2e225afe-9707-464a-8d42-20c33b7c5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"
Oct 30 01:11:05.589455 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.589422    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7l5p\" (UniqueName: \"kubernetes.io/projected/2e225afe-9707-464a-8d42-20c33b7c5390-kube-api-access-r7l5p\") pod \"maas-api-revoke-sub-be79cd53-nkbs5\" (UID: \"2e225afe-9707-464a-8d42-20c33b7c5390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"
Oct 30 01:11:05.608252 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.608212    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"
Oct 30 01:11:05.917010 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:05.916972    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"]
Oct 30 01:11:05.918956 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:11:05.918921    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e225afe_9707_464a_8d42_20c33b7c5390.slice/crio-dc5f3ab8da3c8b334a59d69cb0f49de683f570685a588131591b95ed2b1383ea WatchSource:0}: Error finding container dc5f3ab8da3c8b334a59d69cb0f49de683f570685a588131591b95ed2b1383ea: Status 404 returned error can't find the container with id dc5f3ab8da3c8b334a59d69cb0f49de683f570685a588131591b95ed2b1383ea
Oct 30 01:11:06.309501 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:06.309465    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5" event={"ID":"2e225afe-9707-464a-8d42-20c33b7c5390","Type":"ContainerStarted","Data":"dc5f3ab8da3c8b334a59d69cb0f49de683f570685a588131591b95ed2b1383ea"}
Oct 30 01:11:07.317811 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:07.317771    4425 generic.go:356] "Generic (PLEG): container finished" podID="2e225afe-9707-464a-8d42-20c33b7c5390" containerID="32987fa66b3f1a3bf300f14cf9472d3d2c465fc7ac8106cb494136f488ac5c81" exitCode=0
Oct 30 01:11:07.318281 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:07.317828    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5" event={"ID":"2e225afe-9707-464a-8d42-20c33b7c5390","Type":"ContainerDied","Data":"32987fa66b3f1a3bf300f14cf9472d3d2c465fc7ac8106cb494136f488ac5c81"}
Oct 30 01:11:07.557034 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:11:07.556996    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:11:08.496627 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:08.496588    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"
Oct 30 01:11:08.500758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:08.500720    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e225afe-9707-464a-8d42-20c33b7c5390-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e225afe-9707-464a-8d42-20c33b7c5390-maas-api-xks-ca\") pod \"2e225afe-9707-464a-8d42-20c33b7c5390\" (UID: \"2e225afe-9707-464a-8d42-20c33b7c5390\") "
Oct 30 01:11:08.500968 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:08.500819    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e225afe-9707-464a-8d42-20c33b7c5390-kube-api-access-r7l5p\" (UniqueName: \"kubernetes.io/projected/2e225afe-9707-464a-8d42-20c33b7c5390-kube-api-access-r7l5p\") pod \"2e225afe-9707-464a-8d42-20c33b7c5390\" (UID: \"2e225afe-9707-464a-8d42-20c33b7c5390\") "
Oct 30 01:11:08.501041 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:08.501020    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e225afe-9707-464a-8d42-20c33b7c5390-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e225afe-9707-464a-8d42-20c33b7c5390-maas-api-openshift-ca\") pod \"2e225afe-9707-464a-8d42-20c33b7c5390\" (UID: \"2e225afe-9707-464a-8d42-20c33b7c5390\") "
Oct 30 01:11:08.501508 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:08.501473    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e225afe-9707-464a-8d42-20c33b7c5390-maas-api-openshift-ca" pod "2e225afe-9707-464a-8d42-20c33b7c5390" (UID: "2e225afe-9707-464a-8d42-20c33b7c5390"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:08.501753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:08.501723    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e225afe-9707-464a-8d42-20c33b7c5390-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:08.504209 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:08.504168    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e225afe-9707-464a-8d42-20c33b7c5390-maas-api-xks-ca" pod "2e225afe-9707-464a-8d42-20c33b7c5390" (UID: "2e225afe-9707-464a-8d42-20c33b7c5390"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:08.504521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:08.504480    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e225afe-9707-464a-8d42-20c33b7c5390-kube-api-access-r7l5p" pod "2e225afe-9707-464a-8d42-20c33b7c5390" (UID: "2e225afe-9707-464a-8d42-20c33b7c5390"). InnerVolumeSpecName "kube-api-access-r7l5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:08.602996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:08.602900    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e225afe-9707-464a-8d42-20c33b7c5390-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:08.602996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:08.602939    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7l5p\" (UniqueName: \"kubernetes.io/projected/2e225afe-9707-464a-8d42-20c33b7c5390-kube-api-access-r7l5p\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:09.333301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:09.333268    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"
Oct 30 01:11:09.335387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:09.333306    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5" event={"ID":"2e225afe-9707-464a-8d42-20c33b7c5390","Type":"ContainerDied","Data":"dc5f3ab8da3c8b334a59d69cb0f49de683f570685a588131591b95ed2b1383ea"}
Oct 30 01:11:09.335387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:09.333386    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc5f3ab8da3c8b334a59d69cb0f49de683f570685a588131591b95ed2b1383ea"
Oct 30 01:11:17.630142 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.630101    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"]
Oct 30 01:11:17.630799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.630777    4425 cpu_manager.go:414] "containerMap: removing container" podUID="2e225afe-9707-464a-8d42-20c33b7c5390" containerName="revoke-keys"
Oct 30 01:11:17.630882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.630800    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e225afe-9707-464a-8d42-20c33b7c5390" containerName="revoke-keys"
Oct 30 01:11:17.631011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.630998    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e225afe-9707-464a-8d42-20c33b7c5390" containerName="revoke-keys"
Oct 30 01:11:17.635741 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.635714    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"
Oct 30 01:11:17.660825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.660782    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:11:17.669431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.669390    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:17.669841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.669820    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:11:17.670838 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.670806    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:11:17.680621 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.680581    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"]
Oct 30 01:11:17.722835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.722794    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcdfn\" (UniqueName: \"kubernetes.io/projected/e2a4abd5-eb3e-425d-9faa-d16f91876af3-kube-api-access-tcdfn\") pod \"maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd\" (UID: \"e2a4abd5-eb3e-425d-9faa-d16f91876af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"
Oct 30 01:11:17.723023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.722885    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a4abd5-eb3e-425d-9faa-d16f91876af3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd\" (UID: \"e2a4abd5-eb3e-425d-9faa-d16f91876af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"
Oct 30 01:11:17.723023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.722996    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a4abd5-eb3e-425d-9faa-d16f91876af3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd\" (UID: \"e2a4abd5-eb3e-425d-9faa-d16f91876af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"
Oct 30 01:11:17.824017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.823976    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcdfn\" (UniqueName: \"kubernetes.io/projected/e2a4abd5-eb3e-425d-9faa-d16f91876af3-kube-api-access-tcdfn\") pod \"maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd\" (UID: \"e2a4abd5-eb3e-425d-9faa-d16f91876af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"
Oct 30 01:11:17.824263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.824097    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a4abd5-eb3e-425d-9faa-d16f91876af3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd\" (UID: \"e2a4abd5-eb3e-425d-9faa-d16f91876af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"
Oct 30 01:11:17.824263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.824177    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a4abd5-eb3e-425d-9faa-d16f91876af3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd\" (UID: \"e2a4abd5-eb3e-425d-9faa-d16f91876af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"
Oct 30 01:11:17.825031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.825003    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a4abd5-eb3e-425d-9faa-d16f91876af3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd\" (UID: \"e2a4abd5-eb3e-425d-9faa-d16f91876af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"
Oct 30 01:11:17.826860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.826831    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a4abd5-eb3e-425d-9faa-d16f91876af3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd\" (UID: \"e2a4abd5-eb3e-425d-9faa-d16f91876af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"
Oct 30 01:11:17.846785 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.846749    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcdfn\" (UniqueName: \"kubernetes.io/projected/e2a4abd5-eb3e-425d-9faa-d16f91876af3-kube-api-access-tcdfn\") pod \"maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd\" (UID: \"e2a4abd5-eb3e-425d-9faa-d16f91876af3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"
Oct 30 01:11:17.972489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:17.972405    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"
Oct 30 01:11:18.283242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:18.283203    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"]
Oct 30 01:11:18.285930 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:11:18.285899    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a4abd5_eb3e_425d_9faa_d16f91876af3.slice/crio-ca23f87c8c5407f6528c3a0729de5f3bc98b7089d14189e2e4c9649637974623 WatchSource:0}: Error finding container ca23f87c8c5407f6528c3a0729de5f3bc98b7089d14189e2e4c9649637974623: Status 404 returned error can't find the container with id ca23f87c8c5407f6528c3a0729de5f3bc98b7089d14189e2e4c9649637974623
Oct 30 01:11:18.395086 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:18.395045    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd" event={"ID":"e2a4abd5-eb3e-425d-9faa-d16f91876af3","Type":"ContainerStarted","Data":"ca23f87c8c5407f6528c3a0729de5f3bc98b7089d14189e2e4c9649637974623"}
Oct 30 01:11:19.404001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:19.403963    4425 generic.go:356] "Generic (PLEG): container finished" podID="e2a4abd5-eb3e-425d-9faa-d16f91876af3" containerID="47aac729f02a73dcfddfdf26db7461742338f79879bd03b325d9d8eb3243623e" exitCode=0
Oct 30 01:11:19.404488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:19.404048    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd" event={"ID":"e2a4abd5-eb3e-425d-9faa-d16f91876af3","Type":"ContainerDied","Data":"47aac729f02a73dcfddfdf26db7461742338f79879bd03b325d9d8eb3243623e"}
Oct 30 01:11:20.570400 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:20.570361    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"
Oct 30 01:11:20.658683 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:20.658641    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a4abd5-eb3e-425d-9faa-d16f91876af3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a4abd5-eb3e-425d-9faa-d16f91876af3-maas-api-xks-ca\") pod \"e2a4abd5-eb3e-425d-9faa-d16f91876af3\" (UID: \"e2a4abd5-eb3e-425d-9faa-d16f91876af3\") "
Oct 30 01:11:20.658896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:20.658720    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a4abd5-eb3e-425d-9faa-d16f91876af3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a4abd5-eb3e-425d-9faa-d16f91876af3-maas-api-openshift-ca\") pod \"e2a4abd5-eb3e-425d-9faa-d16f91876af3\" (UID: \"e2a4abd5-eb3e-425d-9faa-d16f91876af3\") "
Oct 30 01:11:20.658896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:20.658762    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a4abd5-eb3e-425d-9faa-d16f91876af3-kube-api-access-tcdfn\" (UniqueName: \"kubernetes.io/projected/e2a4abd5-eb3e-425d-9faa-d16f91876af3-kube-api-access-tcdfn\") pod \"e2a4abd5-eb3e-425d-9faa-d16f91876af3\" (UID: \"e2a4abd5-eb3e-425d-9faa-d16f91876af3\") "
Oct 30 01:11:20.659143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:20.659118    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a4abd5-eb3e-425d-9faa-d16f91876af3-maas-api-openshift-ca" pod "e2a4abd5-eb3e-425d-9faa-d16f91876af3" (UID: "e2a4abd5-eb3e-425d-9faa-d16f91876af3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:20.661016 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:20.660989    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a4abd5-eb3e-425d-9faa-d16f91876af3-maas-api-xks-ca" pod "e2a4abd5-eb3e-425d-9faa-d16f91876af3" (UID: "e2a4abd5-eb3e-425d-9faa-d16f91876af3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:20.661128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:20.660992    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a4abd5-eb3e-425d-9faa-d16f91876af3-kube-api-access-tcdfn" pod "e2a4abd5-eb3e-425d-9faa-d16f91876af3" (UID: "e2a4abd5-eb3e-425d-9faa-d16f91876af3"). InnerVolumeSpecName "kube-api-access-tcdfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:20.760322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:20.760280    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcdfn\" (UniqueName: \"kubernetes.io/projected/e2a4abd5-eb3e-425d-9faa-d16f91876af3-kube-api-access-tcdfn\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:20.760322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:20.760316    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2a4abd5-eb3e-425d-9faa-d16f91876af3-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:20.760322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:20.760326    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2a4abd5-eb3e-425d-9faa-d16f91876af3-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:21.420287 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:21.420242    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"
Oct 30 01:11:21.424760 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:21.424714    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd" event={"ID":"e2a4abd5-eb3e-425d-9faa-d16f91876af3","Type":"ContainerDied","Data":"ca23f87c8c5407f6528c3a0729de5f3bc98b7089d14189e2e4c9649637974623"}
Oct 30 01:11:21.424966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:21.424790    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca23f87c8c5407f6528c3a0729de5f3bc98b7089d14189e2e4c9649637974623"
Oct 30 01:11:22.265792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.265754    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"]
Oct 30 01:11:22.266578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.266553    4425 cpu_manager.go:414] "containerMap: removing container" podUID="e2a4abd5-eb3e-425d-9faa-d16f91876af3" containerName="revoke-keys"
Oct 30 01:11:22.266693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.266580    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a4abd5-eb3e-425d-9faa-d16f91876af3" containerName="revoke-keys"
Oct 30 01:11:22.266765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.266713    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a4abd5-eb3e-425d-9faa-d16f91876af3" containerName="revoke-keys"
Oct 30 01:11:22.270733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.270699    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"
Oct 30 01:11:22.296192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.296130    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:11:22.296192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.296130    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:22.296422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.296148    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:11:22.296422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.296148    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:11:22.299202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.299170    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"]
Oct 30 01:11:22.383309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.383269    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnh54\" (UniqueName: \"kubernetes.io/projected/0907da7b-464b-4ad4-949c-d956330e6477-kube-api-access-lnh54\") pod \"maas-api-revoke-sub-006f7362-kl77s\" (UID: \"0907da7b-464b-4ad4-949c-d956330e6477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"
Oct 30 01:11:22.383537 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.383458    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0907da7b-464b-4ad4-949c-d956330e6477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-006f7362-kl77s\" (UID: \"0907da7b-464b-4ad4-949c-d956330e6477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"
Oct 30 01:11:22.383599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.383583    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0907da7b-464b-4ad4-949c-d956330e6477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-006f7362-kl77s\" (UID: \"0907da7b-464b-4ad4-949c-d956330e6477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"
Oct 30 01:11:22.485294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.485245    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0907da7b-464b-4ad4-949c-d956330e6477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-006f7362-kl77s\" (UID: \"0907da7b-464b-4ad4-949c-d956330e6477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"
Oct 30 01:11:22.485504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.485356    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnh54\" (UniqueName: \"kubernetes.io/projected/0907da7b-464b-4ad4-949c-d956330e6477-kube-api-access-lnh54\") pod \"maas-api-revoke-sub-006f7362-kl77s\" (UID: \"0907da7b-464b-4ad4-949c-d956330e6477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"
Oct 30 01:11:22.485504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.485471    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0907da7b-464b-4ad4-949c-d956330e6477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-006f7362-kl77s\" (UID: \"0907da7b-464b-4ad4-949c-d956330e6477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"
Oct 30 01:11:22.485947 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.485919    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0907da7b-464b-4ad4-949c-d956330e6477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-006f7362-kl77s\" (UID: \"0907da7b-464b-4ad4-949c-d956330e6477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"
Oct 30 01:11:22.488192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.488146    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0907da7b-464b-4ad4-949c-d956330e6477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-006f7362-kl77s\" (UID: \"0907da7b-464b-4ad4-949c-d956330e6477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"
Oct 30 01:11:22.506753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.506720    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnh54\" (UniqueName: \"kubernetes.io/projected/0907da7b-464b-4ad4-949c-d956330e6477-kube-api-access-lnh54\") pod \"maas-api-revoke-sub-006f7362-kl77s\" (UID: \"0907da7b-464b-4ad4-949c-d956330e6477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"
Oct 30 01:11:22.592644 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.592542    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"
Oct 30 01:11:22.915903 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:22.915871    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"]
Oct 30 01:11:22.916823 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:11:22.916796    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0907da7b_464b_4ad4_949c_d956330e6477.slice/crio-fc6566aa43d18107db011889e29ae299963416aac301436cb19bfcb7eede7432 WatchSource:0}: Error finding container fc6566aa43d18107db011889e29ae299963416aac301436cb19bfcb7eede7432: Status 404 returned error can't find the container with id fc6566aa43d18107db011889e29ae299963416aac301436cb19bfcb7eede7432
Oct 30 01:11:23.438799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:23.438685    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s" event={"ID":"0907da7b-464b-4ad4-949c-d956330e6477","Type":"ContainerStarted","Data":"e5dc512815f4224c42093b6ea0585333a32f6210e1926f253973498e0757ac11"}
Oct 30 01:11:23.438799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:23.438769    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s" event={"ID":"0907da7b-464b-4ad4-949c-d956330e6477","Type":"ContainerStarted","Data":"fc6566aa43d18107db011889e29ae299963416aac301436cb19bfcb7eede7432"}
Oct 30 01:11:24.447592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:24.447555    4425 generic.go:356] "Generic (PLEG): container finished" podID="0907da7b-464b-4ad4-949c-d956330e6477" containerID="e5dc512815f4224c42093b6ea0585333a32f6210e1926f253973498e0757ac11" exitCode=0
Oct 30 01:11:24.448020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:24.447650    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s" event={"ID":"0907da7b-464b-4ad4-949c-d956330e6477","Type":"ContainerDied","Data":"e5dc512815f4224c42093b6ea0585333a32f6210e1926f253973498e0757ac11"}
Oct 30 01:11:25.611384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:25.611356    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"
Oct 30 01:11:25.734678 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:25.734580    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0907da7b-464b-4ad4-949c-d956330e6477-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0907da7b-464b-4ad4-949c-d956330e6477-maas-api-xks-ca\") pod \"0907da7b-464b-4ad4-949c-d956330e6477\" (UID: \"0907da7b-464b-4ad4-949c-d956330e6477\") "
Oct 30 01:11:25.734678 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:25.734671    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0907da7b-464b-4ad4-949c-d956330e6477-kube-api-access-lnh54\" (UniqueName: \"kubernetes.io/projected/0907da7b-464b-4ad4-949c-d956330e6477-kube-api-access-lnh54\") pod \"0907da7b-464b-4ad4-949c-d956330e6477\" (UID: \"0907da7b-464b-4ad4-949c-d956330e6477\") "
Oct 30 01:11:25.734894 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:25.734731    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0907da7b-464b-4ad4-949c-d956330e6477-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0907da7b-464b-4ad4-949c-d956330e6477-maas-api-openshift-ca\") pod \"0907da7b-464b-4ad4-949c-d956330e6477\" (UID: \"0907da7b-464b-4ad4-949c-d956330e6477\") "
Oct 30 01:11:25.735145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:25.735120    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0907da7b-464b-4ad4-949c-d956330e6477-maas-api-openshift-ca" pod "0907da7b-464b-4ad4-949c-d956330e6477" (UID: "0907da7b-464b-4ad4-949c-d956330e6477"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:25.736966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:25.736939    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0907da7b-464b-4ad4-949c-d956330e6477-kube-api-access-lnh54" pod "0907da7b-464b-4ad4-949c-d956330e6477" (UID: "0907da7b-464b-4ad4-949c-d956330e6477"). InnerVolumeSpecName "kube-api-access-lnh54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:25.736966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:25.736947    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0907da7b-464b-4ad4-949c-d956330e6477-maas-api-xks-ca" pod "0907da7b-464b-4ad4-949c-d956330e6477" (UID: "0907da7b-464b-4ad4-949c-d956330e6477"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:25.836413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:25.836370    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnh54\" (UniqueName: \"kubernetes.io/projected/0907da7b-464b-4ad4-949c-d956330e6477-kube-api-access-lnh54\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:25.836413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:25.836406    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0907da7b-464b-4ad4-949c-d956330e6477-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:25.836413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:25.836418    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0907da7b-464b-4ad4-949c-d956330e6477-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:26.467262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:26.467230    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"
Oct 30 01:11:26.467548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:26.467245    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s" event={"ID":"0907da7b-464b-4ad4-949c-d956330e6477","Type":"ContainerDied","Data":"fc6566aa43d18107db011889e29ae299963416aac301436cb19bfcb7eede7432"}
Oct 30 01:11:26.467638 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:26.467579    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc6566aa43d18107db011889e29ae299963416aac301436cb19bfcb7eede7432"
Oct 30 01:11:27.436052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.436008    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"]
Oct 30 01:11:27.436851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.436828    4425 cpu_manager.go:414] "containerMap: removing container" podUID="0907da7b-464b-4ad4-949c-d956330e6477" containerName="revoke-keys"
Oct 30 01:11:27.436963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.436854    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0907da7b-464b-4ad4-949c-d956330e6477" containerName="revoke-keys"
Oct 30 01:11:27.437030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.437012    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0907da7b-464b-4ad4-949c-d956330e6477" containerName="revoke-keys"
Oct 30 01:11:27.440526 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.440501    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.453259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.453222    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-ce5996\"/\"openshift-service-ca.crt\""
Oct 30 01:11:27.453428 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.453246    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-ce5996\"/\"kube-root-ca.crt\""
Oct 30 01:11:27.460226 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.460145    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-tmp-dir\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.460759 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.460248    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-home\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.460759 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.460740    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d6l5\" (UniqueName: \"kubernetes.io/projected/99499942-631e-4844-b1c4-1c7059fb5ece-kube-api-access-2d6l5\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.460976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.460929    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99499942-631e-4844-b1c4-1c7059fb5ece-tls-certs\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.461062 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.460976    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-dshm\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.461062 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.461013    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-model-cache\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.464432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.464389    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-ce5996\"/\"default-dockercfg-h6rjb\""
Oct 30 01:11:27.464633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.464594    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-ce5996\"/\"praxis-route-ce5996-kserve-self-signed-certs\""
Oct 30 01:11:27.469019 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.468982    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"]
Oct 30 01:11:27.562610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.562536    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99499942-631e-4844-b1c4-1c7059fb5ece-tls-certs\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.562841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.562688    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-dshm\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.562841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.562726    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-model-cache\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.562976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.562839    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-tmp-dir\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.562976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.562881    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-home\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.562976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.562906    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d6l5\" (UniqueName: \"kubernetes.io/projected/99499942-631e-4844-b1c4-1c7059fb5ece-kube-api-access-2d6l5\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.563393 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.563361    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-tmp-dir\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.563647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.563619    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-home\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.563744 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.563668    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-model-cache\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.565259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.565234    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-dshm\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.565701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.565678    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99499942-631e-4844-b1c4-1c7059fb5ece-tls-certs\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.585011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.584978    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d6l5\" (UniqueName: \"kubernetes.io/projected/99499942-631e-4844-b1c4-1c7059fb5ece-kube-api-access-2d6l5\") pod \"praxis-route-ce5996-kserve-6b7d54c547-84lv8\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") " pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:27.759517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:27.759407    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:28.078993 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:28.078967    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"]
Oct 30 01:11:28.080997 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:11:28.080964    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99499942_631e_4844_b1c4_1c7059fb5ece.slice/crio-600a3962de9cc690f8d07dd620b1abb038dcbd8a212f56bc9b3fca552e132b25 WatchSource:0}: Error finding container 600a3962de9cc690f8d07dd620b1abb038dcbd8a212f56bc9b3fca552e132b25: Status 404 returned error can't find the container with id 600a3962de9cc690f8d07dd620b1abb038dcbd8a212f56bc9b3fca552e132b25
Oct 30 01:11:28.492614 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:28.492513    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8" event={"ID":"99499942-631e-4844-b1c4-1c7059fb5ece","Type":"ContainerStarted","Data":"c2039bbf8b8223e6c3d2af287175b625c1b9afdf67e62c78f8aca95c98f08566"}
Oct 30 01:11:28.492614 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:28.492563    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8" event={"ID":"99499942-631e-4844-b1c4-1c7059fb5ece","Type":"ContainerStarted","Data":"600a3962de9cc690f8d07dd620b1abb038dcbd8a212f56bc9b3fca552e132b25"}
Oct 30 01:11:28.531485 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:28.531399    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8" podStartSLOduration=1.53138005 podStartE2EDuration="1.53138005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:28.529039826 +0000 UTC m=+1797.278695136" watchObservedRunningTime="2026-10-30 01:11:28.53138005 +0000 UTC m=+1797.281035361"
Oct 30 01:11:28.760021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:28.759991    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:29.355384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.355346    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"]
Oct 30 01:11:29.359783 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.359754    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"
Oct 30 01:11:29.381551 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.381523    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-1d6d47\""
Oct 30 01:11:29.384521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.384489    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzpkv\" (UniqueName: \"kubernetes.io/projected/c820790b-21c4-44d7-b2e8-7a7d1c6af82d-kube-api-access-wzpkv\") pod \"payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh\" (UID: \"c820790b-21c4-44d7-b2e8-7a7d1c6af82d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"
Oct 30 01:11:29.384743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.384719    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c820790b-21c4-44d7-b2e8-7a7d1c6af82d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh\" (UID: \"c820790b-21c4-44d7-b2e8-7a7d1c6af82d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"
Oct 30 01:11:29.393115 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.393068    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"]
Oct 30 01:11:29.401904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.401865    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-1d6d47-dockercfg-9fnfd\""
Oct 30 01:11:29.412405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.412366    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"]
Oct 30 01:11:29.417028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.416991    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"
Oct 30 01:11:29.437724 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.436782    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"]
Oct 30 01:11:29.489475 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.489435    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c820790b-21c4-44d7-b2e8-7a7d1c6af82d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh\" (UID: \"c820790b-21c4-44d7-b2e8-7a7d1c6af82d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"
Oct 30 01:11:29.489691 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.489598    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzpkv\" (UniqueName: \"kubernetes.io/projected/c820790b-21c4-44d7-b2e8-7a7d1c6af82d-kube-api-access-wzpkv\") pod \"payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh\" (UID: \"c820790b-21c4-44d7-b2e8-7a7d1c6af82d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"
Oct 30 01:11:29.492141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.491408    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c820790b-21c4-44d7-b2e8-7a7d1c6af82d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh\" (UID: \"c820790b-21c4-44d7-b2e8-7a7d1c6af82d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"
Oct 30 01:11:29.512373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.512339    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzpkv\" (UniqueName: \"kubernetes.io/projected/c820790b-21c4-44d7-b2e8-7a7d1c6af82d-kube-api-access-wzpkv\") pod \"payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh\" (UID: \"c820790b-21c4-44d7-b2e8-7a7d1c6af82d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"
Oct 30 01:11:29.590459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.590411    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ptf\" (UniqueName: \"kubernetes.io/projected/888c42f2-9961-49e5-a8e8-30af506bfe25-kube-api-access-v6ptf\") pod \"payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d\" (UID: \"888c42f2-9961-49e5-a8e8-30af506bfe25\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"
Oct 30 01:11:29.590732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.590709    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888c42f2-9961-49e5-a8e8-30af506bfe25-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d\" (UID: \"888c42f2-9961-49e5-a8e8-30af506bfe25\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"
Oct 30 01:11:29.642266 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.642167    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"]
Oct 30 01:11:29.674773 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.674712    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"
Oct 30 01:11:29.676391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.676362    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"]
Oct 30 01:11:29.694010 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.693971    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888c42f2-9961-49e5-a8e8-30af506bfe25-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d\" (UID: \"888c42f2-9961-49e5-a8e8-30af506bfe25\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"
Oct 30 01:11:29.694320 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.694287    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ptf\" (UniqueName: \"kubernetes.io/projected/888c42f2-9961-49e5-a8e8-30af506bfe25-kube-api-access-v6ptf\") pod \"payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d\" (UID: \"888c42f2-9961-49e5-a8e8-30af506bfe25\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"
Oct 30 01:11:29.695144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.695096    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888c42f2-9961-49e5-a8e8-30af506bfe25-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d\" (UID: \"888c42f2-9961-49e5-a8e8-30af506bfe25\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"
Oct 30 01:11:29.717915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.717867    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ptf\" (UniqueName: \"kubernetes.io/projected/888c42f2-9961-49e5-a8e8-30af506bfe25-kube-api-access-v6ptf\") pod \"payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d\" (UID: \"888c42f2-9961-49e5-a8e8-30af506bfe25\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"
Oct 30 01:11:29.741883 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:29.741313    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"
Oct 30 01:11:30.004342 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:30.004314    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"]
Oct 30 01:11:30.005882 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:11:30.005846    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc820790b_21c4_44d7_b2e8_7a7d1c6af82d.slice/crio-436ed0fa3aa6458dbcf6540b5ae59d8e9f1c5571f0b98205e8c06b05db36f28a WatchSource:0}: Error finding container 436ed0fa3aa6458dbcf6540b5ae59d8e9f1c5571f0b98205e8c06b05db36f28a: Status 404 returned error can't find the container with id 436ed0fa3aa6458dbcf6540b5ae59d8e9f1c5571f0b98205e8c06b05db36f28a
Oct 30 01:11:30.087796 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:30.087768    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"]
Oct 30 01:11:30.088550 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:11:30.088520    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888c42f2_9961_49e5_a8e8_30af506bfe25.slice/crio-6defe6d49cfb80a88b059b283ee84d1b6d0b142e69fb18f1b553cafb2627bfa4 WatchSource:0}: Error finding container 6defe6d49cfb80a88b059b283ee84d1b6d0b142e69fb18f1b553cafb2627bfa4: Status 404 returned error can't find the container with id 6defe6d49cfb80a88b059b283ee84d1b6d0b142e69fb18f1b553cafb2627bfa4
Oct 30 01:11:30.236839 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:30.236798    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:11:30.514228 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:30.514189    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d" event={"ID":"888c42f2-9961-49e5-a8e8-30af506bfe25","Type":"ContainerStarted","Data":"12f6ce8987215b3f25a6d43284c7f715630a1de690c99abde477f85ecd7d1699"}
Oct 30 01:11:30.514737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:30.514259    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d" event={"ID":"888c42f2-9961-49e5-a8e8-30af506bfe25","Type":"ContainerStarted","Data":"6defe6d49cfb80a88b059b283ee84d1b6d0b142e69fb18f1b553cafb2627bfa4"}
Oct 30 01:11:30.514737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:30.514403    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"
Oct 30 01:11:30.516228 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:30.516200    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh" event={"ID":"c820790b-21c4-44d7-b2e8-7a7d1c6af82d","Type":"ContainerStarted","Data":"64b380544adaa80c632cf47b6661ea731aadf4d23d4d7d6b5bc091dbc6b16f4d"}
Oct 30 01:11:30.516373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:30.516248    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"
Oct 30 01:11:30.516373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:30.516268    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh" event={"ID":"c820790b-21c4-44d7-b2e8-7a7d1c6af82d","Type":"ContainerStarted","Data":"436ed0fa3aa6458dbcf6540b5ae59d8e9f1c5571f0b98205e8c06b05db36f28a"}
Oct 30 01:11:30.552049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:30.551967    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d" podStartSLOduration=1.551949654 podStartE2EDuration="1.551949654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:30.549807708 +0000 UTC m=+1799.299463022" watchObservedRunningTime="2026-10-30 01:11:30.551949654 +0000 UTC m=+1799.301605000"
Oct 30 01:11:30.614079 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:30.614004    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh" podStartSLOduration=1.613983078 podStartE2EDuration="1.613983078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:30.610327488 +0000 UTC m=+1799.359982797" watchObservedRunningTime="2026-10-30 01:11:30.613983078 +0000 UTC m=+1799.363638388"
Oct 30 01:11:32.682381 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:32.682339    4425 scope.go:122] "RemoveContainer" containerID="44bfd636bbe46cea34a681da03adbdad82f7e514479d6e9cb4d142839778a683"
Oct 30 01:11:32.696001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:32.695970    4425 scope.go:122] "RemoveContainer" containerID="0ff7538697963dfce03bc1c5c6a7a763f70f32e2739775c77c58a97f3fc5cb33"
Oct 30 01:11:34.643373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:34.643302    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr" podUID="a50d15c6-f321-40a9-96fa-c70c6d1509fc" containerName="payload-processing" containerID="cri-o://b33fce9af725db8bd92e76265a8f39504b6210a1b0821acf370786e4a9bfbb7f" gracePeriod=25
Oct 30 01:11:34.676892 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:34.676843    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f" podUID="414e0450-fc6f-4380-9b46-381fc11cc95e" containerName="payload-pre-processing" containerID="cri-o://f83181df34fefbac740cda3c6d5d6c53374510cddbe9a1ba094f408bcdb482b1" gracePeriod=25
Oct 30 01:11:34.953718 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:34.953690    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"
Oct 30 01:11:35.003144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.003113    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"
Oct 30 01:11:35.090924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.090890    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a50d15c6-f321-40a9-96fa-c70c6d1509fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50d15c6-f321-40a9-96fa-c70c6d1509fc-plugins-config-volume\") pod \"a50d15c6-f321-40a9-96fa-c70c6d1509fc\" (UID: \"a50d15c6-f321-40a9-96fa-c70c6d1509fc\") "
Oct 30 01:11:35.091145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.090955    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414e0450-fc6f-4380-9b46-381fc11cc95e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/414e0450-fc6f-4380-9b46-381fc11cc95e-plugins-config-volume\") pod \"414e0450-fc6f-4380-9b46-381fc11cc95e\" (UID: \"414e0450-fc6f-4380-9b46-381fc11cc95e\") "
Oct 30 01:11:35.091145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.091004    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414e0450-fc6f-4380-9b46-381fc11cc95e-kube-api-access-99vkl\" (UniqueName: \"kubernetes.io/projected/414e0450-fc6f-4380-9b46-381fc11cc95e-kube-api-access-99vkl\") pod \"414e0450-fc6f-4380-9b46-381fc11cc95e\" (UID: \"414e0450-fc6f-4380-9b46-381fc11cc95e\") "
Oct 30 01:11:35.091145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.091104    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a50d15c6-f321-40a9-96fa-c70c6d1509fc-kube-api-access-26txx\" (UniqueName: \"kubernetes.io/projected/a50d15c6-f321-40a9-96fa-c70c6d1509fc-kube-api-access-26txx\") pod \"a50d15c6-f321-40a9-96fa-c70c6d1509fc\" (UID: \"a50d15c6-f321-40a9-96fa-c70c6d1509fc\") "
Oct 30 01:11:35.091431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.091405    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414e0450-fc6f-4380-9b46-381fc11cc95e-plugins-config-volume" pod "414e0450-fc6f-4380-9b46-381fc11cc95e" (UID: "414e0450-fc6f-4380-9b46-381fc11cc95e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:35.091502 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.091432    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50d15c6-f321-40a9-96fa-c70c6d1509fc-plugins-config-volume" pod "a50d15c6-f321-40a9-96fa-c70c6d1509fc" (UID: "a50d15c6-f321-40a9-96fa-c70c6d1509fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:35.091760 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.091740    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a50d15c6-f321-40a9-96fa-c70c6d1509fc-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:35.091821 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.091765    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/414e0450-fc6f-4380-9b46-381fc11cc95e-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:35.093397 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.093367    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414e0450-fc6f-4380-9b46-381fc11cc95e-kube-api-access-99vkl" pod "414e0450-fc6f-4380-9b46-381fc11cc95e" (UID: "414e0450-fc6f-4380-9b46-381fc11cc95e"). InnerVolumeSpecName "kube-api-access-99vkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:35.093577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.093552    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50d15c6-f321-40a9-96fa-c70c6d1509fc-kube-api-access-26txx" pod "a50d15c6-f321-40a9-96fa-c70c6d1509fc" (UID: "a50d15c6-f321-40a9-96fa-c70c6d1509fc"). InnerVolumeSpecName "kube-api-access-26txx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:35.192970 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.192856    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99vkl\" (UniqueName: \"kubernetes.io/projected/414e0450-fc6f-4380-9b46-381fc11cc95e-kube-api-access-99vkl\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:35.192970 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.192901    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26txx\" (UniqueName: \"kubernetes.io/projected/a50d15c6-f321-40a9-96fa-c70c6d1509fc-kube-api-access-26txx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:35.531177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.531124    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"]
Oct 30 01:11:35.531734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.531717    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a50d15c6-f321-40a9-96fa-c70c6d1509fc" containerName="payload-processing"
Oct 30 01:11:35.531801 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.531735    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50d15c6-f321-40a9-96fa-c70c6d1509fc" containerName="payload-processing"
Oct 30 01:11:35.531801 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.531781    4425 cpu_manager.go:414] "containerMap: removing container" podUID="414e0450-fc6f-4380-9b46-381fc11cc95e" containerName="payload-pre-processing"
Oct 30 01:11:35.531801 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.531786    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414e0450-fc6f-4380-9b46-381fc11cc95e" containerName="payload-pre-processing"
Oct 30 01:11:35.531956 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.531873    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="414e0450-fc6f-4380-9b46-381fc11cc95e" containerName="payload-pre-processing"
Oct 30 01:11:35.531956 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.531881    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50d15c6-f321-40a9-96fa-c70c6d1509fc" containerName="payload-processing"
Oct 30 01:11:35.535715 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.535691    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"
Oct 30 01:11:35.560765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.560599    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:11:35.560966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.560782    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:11:35.560966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.560888    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:11:35.560966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.560964    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:35.571221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.565606    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"]
Oct 30 01:11:35.571221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.567650    4425 generic.go:356] "Generic (PLEG): container finished" podID="a50d15c6-f321-40a9-96fa-c70c6d1509fc" containerID="b33fce9af725db8bd92e76265a8f39504b6210a1b0821acf370786e4a9bfbb7f" exitCode=0
Oct 30 01:11:35.571221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.567796    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr" event={"ID":"a50d15c6-f321-40a9-96fa-c70c6d1509fc","Type":"ContainerDied","Data":"b33fce9af725db8bd92e76265a8f39504b6210a1b0821acf370786e4a9bfbb7f"}
Oct 30 01:11:35.571221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.567852    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr" event={"ID":"a50d15c6-f321-40a9-96fa-c70c6d1509fc","Type":"ContainerDied","Data":"aad1accdbaa958beba02f6855f408afda0559ad1695da72f6392a2d52547ce4e"}
Oct 30 01:11:35.571221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.567894    4425 scope.go:122] "RemoveContainer" containerID="b33fce9af725db8bd92e76265a8f39504b6210a1b0821acf370786e4a9bfbb7f"
Oct 30 01:11:35.571221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.568113    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"
Oct 30 01:11:35.571221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.570970    4425 generic.go:356] "Generic (PLEG): container finished" podID="414e0450-fc6f-4380-9b46-381fc11cc95e" containerID="f83181df34fefbac740cda3c6d5d6c53374510cddbe9a1ba094f408bcdb482b1" exitCode=0
Oct 30 01:11:35.571221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.571055    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f" event={"ID":"414e0450-fc6f-4380-9b46-381fc11cc95e","Type":"ContainerDied","Data":"f83181df34fefbac740cda3c6d5d6c53374510cddbe9a1ba094f408bcdb482b1"}
Oct 30 01:11:35.571221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.571098    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f" event={"ID":"414e0450-fc6f-4380-9b46-381fc11cc95e","Type":"ContainerDied","Data":"45233d9e20d8a4c95a9da41ef04f4f22cff4ab488f74157806cd9d4c065d1535"}
Oct 30 01:11:35.571221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.571233    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"
Oct 30 01:11:35.582436 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.582413    4425 scope.go:122] "RemoveContainer" containerID="b33fce9af725db8bd92e76265a8f39504b6210a1b0821acf370786e4a9bfbb7f"
Oct 30 01:11:35.582912 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:11:35.582888    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33fce9af725db8bd92e76265a8f39504b6210a1b0821acf370786e4a9bfbb7f\": container with ID starting with b33fce9af725db8bd92e76265a8f39504b6210a1b0821acf370786e4a9bfbb7f not found: ID does not exist" containerID="b33fce9af725db8bd92e76265a8f39504b6210a1b0821acf370786e4a9bfbb7f"
Oct 30 01:11:35.583028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.582923    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33fce9af725db8bd92e76265a8f39504b6210a1b0821acf370786e4a9bfbb7f"} err="failed to get container status \"b33fce9af725db8bd92e76265a8f39504b6210a1b0821acf370786e4a9bfbb7f\": rpc error: code = NotFound desc = could not find container \"b33fce9af725db8bd92e76265a8f39504b6210a1b0821acf370786e4a9bfbb7f\": container with ID starting with b33fce9af725db8bd92e76265a8f39504b6210a1b0821acf370786e4a9bfbb7f not found: ID does not exist"
Oct 30 01:11:35.583028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.582948    4425 scope.go:122] "RemoveContainer" containerID="f83181df34fefbac740cda3c6d5d6c53374510cddbe9a1ba094f408bcdb482b1"
Oct 30 01:11:35.594401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.594371    4425 scope.go:122] "RemoveContainer" containerID="f83181df34fefbac740cda3c6d5d6c53374510cddbe9a1ba094f408bcdb482b1"
Oct 30 01:11:35.594816 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:11:35.594785    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83181df34fefbac740cda3c6d5d6c53374510cddbe9a1ba094f408bcdb482b1\": container with ID starting with f83181df34fefbac740cda3c6d5d6c53374510cddbe9a1ba094f408bcdb482b1 not found: ID does not exist" containerID="f83181df34fefbac740cda3c6d5d6c53374510cddbe9a1ba094f408bcdb482b1"
Oct 30 01:11:35.594897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.594826    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83181df34fefbac740cda3c6d5d6c53374510cddbe9a1ba094f408bcdb482b1"} err="failed to get container status \"f83181df34fefbac740cda3c6d5d6c53374510cddbe9a1ba094f408bcdb482b1\": rpc error: code = NotFound desc = could not find container \"f83181df34fefbac740cda3c6d5d6c53374510cddbe9a1ba094f408bcdb482b1\": container with ID starting with f83181df34fefbac740cda3c6d5d6c53374510cddbe9a1ba094f408bcdb482b1 not found: ID does not exist"
Oct 30 01:11:35.598069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.598039    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c71f4c8-8e3d-4ba7-8307-b56719826937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0398835f-jjgsn\" (UID: \"8c71f4c8-8e3d-4ba7-8307-b56719826937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"
Oct 30 01:11:35.598296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.598275    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg9xx\" (UniqueName: \"kubernetes.io/projected/8c71f4c8-8e3d-4ba7-8307-b56719826937-kube-api-access-xg9xx\") pod \"maas-api-revoke-sub-0398835f-jjgsn\" (UID: \"8c71f4c8-8e3d-4ba7-8307-b56719826937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"
Oct 30 01:11:35.598412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.598392    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c71f4c8-8e3d-4ba7-8307-b56719826937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0398835f-jjgsn\" (UID: \"8c71f4c8-8e3d-4ba7-8307-b56719826937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"
Oct 30 01:11:35.618258 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.618213    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"]
Oct 30 01:11:35.622408 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.622371    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4ec143-84f774bf75-qtp4f"]
Oct 30 01:11:35.652402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.652364    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"]
Oct 30 01:11:35.658105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.658070    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4ec143-59bff55668-cf8hr"]
Oct 30 01:11:35.699643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.699612    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg9xx\" (UniqueName: \"kubernetes.io/projected/8c71f4c8-8e3d-4ba7-8307-b56719826937-kube-api-access-xg9xx\") pod \"maas-api-revoke-sub-0398835f-jjgsn\" (UID: \"8c71f4c8-8e3d-4ba7-8307-b56719826937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"
Oct 30 01:11:35.699832 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.699688    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c71f4c8-8e3d-4ba7-8307-b56719826937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0398835f-jjgsn\" (UID: \"8c71f4c8-8e3d-4ba7-8307-b56719826937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"
Oct 30 01:11:35.699832 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.699736    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c71f4c8-8e3d-4ba7-8307-b56719826937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0398835f-jjgsn\" (UID: \"8c71f4c8-8e3d-4ba7-8307-b56719826937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"
Oct 30 01:11:35.700491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.700458    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c71f4c8-8e3d-4ba7-8307-b56719826937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0398835f-jjgsn\" (UID: \"8c71f4c8-8e3d-4ba7-8307-b56719826937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"
Oct 30 01:11:35.702178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.702140    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c71f4c8-8e3d-4ba7-8307-b56719826937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0398835f-jjgsn\" (UID: \"8c71f4c8-8e3d-4ba7-8307-b56719826937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"
Oct 30 01:11:35.721618 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.721582    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg9xx\" (UniqueName: \"kubernetes.io/projected/8c71f4c8-8e3d-4ba7-8307-b56719826937-kube-api-access-xg9xx\") pod \"maas-api-revoke-sub-0398835f-jjgsn\" (UID: \"8c71f4c8-8e3d-4ba7-8307-b56719826937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"
Oct 30 01:11:35.849782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:35.849682    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"
Oct 30 01:11:36.157664 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:36.157630    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"]
Oct 30 01:11:36.159209 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:11:36.159176    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c71f4c8_8e3d_4ba7_8307_b56719826937.slice/crio-39c96e6c64f7798d7ba1fa9ef4179e8e3df1ba9f08430ddd89a0968c53e0b1fb WatchSource:0}: Error finding container 39c96e6c64f7798d7ba1fa9ef4179e8e3df1ba9f08430ddd89a0968c53e0b1fb: Status 404 returned error can't find the container with id 39c96e6c64f7798d7ba1fa9ef4179e8e3df1ba9f08430ddd89a0968c53e0b1fb
Oct 30 01:11:36.579498 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:36.579456    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn" event={"ID":"8c71f4c8-8e3d-4ba7-8307-b56719826937","Type":"ContainerStarted","Data":"e5d28b34a3f6585f9e8b7b2836b254aed4b9b6c48a19b61cfd377c61af7275ef"}
Oct 30 01:11:36.579665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:36.579517    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn" event={"ID":"8c71f4c8-8e3d-4ba7-8307-b56719826937","Type":"ContainerStarted","Data":"39c96e6c64f7798d7ba1fa9ef4179e8e3df1ba9f08430ddd89a0968c53e0b1fb"}
Oct 30 01:11:36.618705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:36.618648    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn" podStartSLOduration=1.618634285 podStartE2EDuration="1.618634285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:36.616245617 +0000 UTC m=+1805.365900927" watchObservedRunningTime="2026-10-30 01:11:36.618634285 +0000 UTC m=+1805.368289640"
Oct 30 01:11:37.423563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:37.423474    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414e0450-fc6f-4380-9b46-381fc11cc95e" path="/var/lib/kubelet/pods/414e0450-fc6f-4380-9b46-381fc11cc95e/volumes"
Oct 30 01:11:37.423955 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:37.423862    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a50d15c6-f321-40a9-96fa-c70c6d1509fc" path="/var/lib/kubelet/pods/a50d15c6-f321-40a9-96fa-c70c6d1509fc/volumes"
Oct 30 01:11:37.558060 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:11:37.558017    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:11:37.589341 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:37.589297    4425 generic.go:356] "Generic (PLEG): container finished" podID="8c71f4c8-8e3d-4ba7-8307-b56719826937" containerID="e5d28b34a3f6585f9e8b7b2836b254aed4b9b6c48a19b61cfd377c61af7275ef" exitCode=0
Oct 30 01:11:37.589532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:37.589369    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn" event={"ID":"8c71f4c8-8e3d-4ba7-8307-b56719826937","Type":"ContainerDied","Data":"e5d28b34a3f6585f9e8b7b2836b254aed4b9b6c48a19b61cfd377c61af7275ef"}
Oct 30 01:11:37.759550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:37.759519    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:37.773205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:37.773142    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:38.608517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:38.608480    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:11:38.771137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:38.771108    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"
Oct 30 01:11:38.840402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:38.840366    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c71f4c8-8e3d-4ba7-8307-b56719826937-kube-api-access-xg9xx\" (UniqueName: \"kubernetes.io/projected/8c71f4c8-8e3d-4ba7-8307-b56719826937-kube-api-access-xg9xx\") pod \"8c71f4c8-8e3d-4ba7-8307-b56719826937\" (UID: \"8c71f4c8-8e3d-4ba7-8307-b56719826937\") "
Oct 30 01:11:38.840637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:38.840569    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c71f4c8-8e3d-4ba7-8307-b56719826937-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c71f4c8-8e3d-4ba7-8307-b56719826937-maas-api-xks-ca\") pod \"8c71f4c8-8e3d-4ba7-8307-b56719826937\" (UID: \"8c71f4c8-8e3d-4ba7-8307-b56719826937\") "
Oct 30 01:11:38.840701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:38.840644    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c71f4c8-8e3d-4ba7-8307-b56719826937-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c71f4c8-8e3d-4ba7-8307-b56719826937-maas-api-openshift-ca\") pod \"8c71f4c8-8e3d-4ba7-8307-b56719826937\" (UID: \"8c71f4c8-8e3d-4ba7-8307-b56719826937\") "
Oct 30 01:11:38.841061 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:38.841033    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c71f4c8-8e3d-4ba7-8307-b56719826937-maas-api-openshift-ca" pod "8c71f4c8-8e3d-4ba7-8307-b56719826937" (UID: "8c71f4c8-8e3d-4ba7-8307-b56719826937"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:38.841233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:38.841209    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c71f4c8-8e3d-4ba7-8307-b56719826937-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:38.842775 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:38.842742    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c71f4c8-8e3d-4ba7-8307-b56719826937-kube-api-access-xg9xx" pod "8c71f4c8-8e3d-4ba7-8307-b56719826937" (UID: "8c71f4c8-8e3d-4ba7-8307-b56719826937"). InnerVolumeSpecName "kube-api-access-xg9xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:38.842903 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:38.842846    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c71f4c8-8e3d-4ba7-8307-b56719826937-maas-api-xks-ca" pod "8c71f4c8-8e3d-4ba7-8307-b56719826937" (UID: "8c71f4c8-8e3d-4ba7-8307-b56719826937"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:38.942897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:38.942852    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c71f4c8-8e3d-4ba7-8307-b56719826937-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:38.942897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:38.942888    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg9xx\" (UniqueName: \"kubernetes.io/projected/8c71f4c8-8e3d-4ba7-8307-b56719826937-kube-api-access-xg9xx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:39.605703 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:39.605661    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn" event={"ID":"8c71f4c8-8e3d-4ba7-8307-b56719826937","Type":"ContainerDied","Data":"39c96e6c64f7798d7ba1fa9ef4179e8e3df1ba9f08430ddd89a0968c53e0b1fb"}
Oct 30 01:11:39.605893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:39.605723    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39c96e6c64f7798d7ba1fa9ef4179e8e3df1ba9f08430ddd89a0968c53e0b1fb"
Oct 30 01:11:39.605893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:39.605831    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"
Oct 30 01:11:41.531215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:41.531183    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"
Oct 30 01:11:41.531657 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:41.531272    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"
Oct 30 01:11:48.339584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.339542    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"]
Oct 30 01:11:48.340126 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.340111    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8c71f4c8-8e3d-4ba7-8307-b56719826937" containerName="revoke-keys"
Oct 30 01:11:48.340126 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.340126    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c71f4c8-8e3d-4ba7-8307-b56719826937" containerName="revoke-keys"
Oct 30 01:11:48.340261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.340250    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c71f4c8-8e3d-4ba7-8307-b56719826937" containerName="revoke-keys"
Oct 30 01:11:48.343512 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.343487    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"
Oct 30 01:11:48.356260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.356222    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:11:48.356260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.356236    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:48.356451 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.356286    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:11:48.356451 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.356410    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:11:48.365117 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.365072    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"]
Oct 30 01:11:48.366241 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.366209    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp2sd\" (UniqueName: \"kubernetes.io/projected/0f6d78a2-1478-43fa-89e5-fd12a889f62f-kube-api-access-wp2sd\") pod \"maas-api-revoke-sub-16f1cb73-hg76h\" (UID: \"0f6d78a2-1478-43fa-89e5-fd12a889f62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"
Oct 30 01:11:48.366439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.366416    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6d78a2-1478-43fa-89e5-fd12a889f62f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16f1cb73-hg76h\" (UID: \"0f6d78a2-1478-43fa-89e5-fd12a889f62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"
Oct 30 01:11:48.366525 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.366480    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6d78a2-1478-43fa-89e5-fd12a889f62f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16f1cb73-hg76h\" (UID: \"0f6d78a2-1478-43fa-89e5-fd12a889f62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"
Oct 30 01:11:48.468379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.468331    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6d78a2-1478-43fa-89e5-fd12a889f62f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16f1cb73-hg76h\" (UID: \"0f6d78a2-1478-43fa-89e5-fd12a889f62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"
Oct 30 01:11:48.468600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.468400    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6d78a2-1478-43fa-89e5-fd12a889f62f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16f1cb73-hg76h\" (UID: \"0f6d78a2-1478-43fa-89e5-fd12a889f62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"
Oct 30 01:11:48.468600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.468505    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp2sd\" (UniqueName: \"kubernetes.io/projected/0f6d78a2-1478-43fa-89e5-fd12a889f62f-kube-api-access-wp2sd\") pod \"maas-api-revoke-sub-16f1cb73-hg76h\" (UID: \"0f6d78a2-1478-43fa-89e5-fd12a889f62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"
Oct 30 01:11:48.469189 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.469147    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6d78a2-1478-43fa-89e5-fd12a889f62f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16f1cb73-hg76h\" (UID: \"0f6d78a2-1478-43fa-89e5-fd12a889f62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"
Oct 30 01:11:48.470949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.470926    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6d78a2-1478-43fa-89e5-fd12a889f62f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16f1cb73-hg76h\" (UID: \"0f6d78a2-1478-43fa-89e5-fd12a889f62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"
Oct 30 01:11:48.490533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.490491    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp2sd\" (UniqueName: \"kubernetes.io/projected/0f6d78a2-1478-43fa-89e5-fd12a889f62f-kube-api-access-wp2sd\") pod \"maas-api-revoke-sub-16f1cb73-hg76h\" (UID: \"0f6d78a2-1478-43fa-89e5-fd12a889f62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"
Oct 30 01:11:48.659425 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.659318    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"
Oct 30 01:11:48.969846 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:48.969811    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"]
Oct 30 01:11:48.972885 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:11:48.972839    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6d78a2_1478_43fa_89e5_fd12a889f62f.slice/crio-996ecbf74ca1ad72121d8be251b9ad202fe49d4273a1dd9f0e1d53d4ef915b39 WatchSource:0}: Error finding container 996ecbf74ca1ad72121d8be251b9ad202fe49d4273a1dd9f0e1d53d4ef915b39: Status 404 returned error can't find the container with id 996ecbf74ca1ad72121d8be251b9ad202fe49d4273a1dd9f0e1d53d4ef915b39
Oct 30 01:11:49.611544 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:49.611436    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"]
Oct 30 01:11:49.620786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:49.620747    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-4ec143-l65kd"]
Oct 30 01:11:49.680657 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:49.680617    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h" event={"ID":"0f6d78a2-1478-43fa-89e5-fd12a889f62f","Type":"ContainerStarted","Data":"6f1c91677138fae3af43f7eb528d30995db717240ef9c9816adc8e93d7be2367"}
Oct 30 01:11:49.680877 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:49.680687    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h" event={"ID":"0f6d78a2-1478-43fa-89e5-fd12a889f62f","Type":"ContainerStarted","Data":"996ecbf74ca1ad72121d8be251b9ad202fe49d4273a1dd9f0e1d53d4ef915b39"}
Oct 30 01:11:49.723877 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:49.723805    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h" podStartSLOduration=1.723782453 podStartE2EDuration="1.723782453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:49.717893227 +0000 UTC m=+1818.467548537" watchObservedRunningTime="2026-10-30 01:11:49.723782453 +0000 UTC m=+1818.473437763"
Oct 30 01:11:50.691460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:50.691405    4425 generic.go:356] "Generic (PLEG): container finished" podID="0f6d78a2-1478-43fa-89e5-fd12a889f62f" containerID="6f1c91677138fae3af43f7eb528d30995db717240ef9c9816adc8e93d7be2367" exitCode=0
Oct 30 01:11:50.691945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:50.691501    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h" event={"ID":"0f6d78a2-1478-43fa-89e5-fd12a889f62f","Type":"ContainerDied","Data":"6f1c91677138fae3af43f7eb528d30995db717240ef9c9816adc8e93d7be2367"}
Oct 30 01:11:51.422445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:51.422405    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a4abd5-eb3e-425d-9faa-d16f91876af3" path="/var/lib/kubelet/pods/e2a4abd5-eb3e-425d-9faa-d16f91876af3/volumes"
Oct 30 01:11:51.886553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:51.886518    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"
Oct 30 01:11:51.923845 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:51.923807    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f6d78a2-1478-43fa-89e5-fd12a889f62f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6d78a2-1478-43fa-89e5-fd12a889f62f-maas-api-xks-ca\") pod \"0f6d78a2-1478-43fa-89e5-fd12a889f62f\" (UID: \"0f6d78a2-1478-43fa-89e5-fd12a889f62f\") "
Oct 30 01:11:51.924034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:51.923861    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f6d78a2-1478-43fa-89e5-fd12a889f62f-kube-api-access-wp2sd\" (UniqueName: \"kubernetes.io/projected/0f6d78a2-1478-43fa-89e5-fd12a889f62f-kube-api-access-wp2sd\") pod \"0f6d78a2-1478-43fa-89e5-fd12a889f62f\" (UID: \"0f6d78a2-1478-43fa-89e5-fd12a889f62f\") "
Oct 30 01:11:51.924233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:51.924046    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f6d78a2-1478-43fa-89e5-fd12a889f62f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6d78a2-1478-43fa-89e5-fd12a889f62f-maas-api-openshift-ca\") pod \"0f6d78a2-1478-43fa-89e5-fd12a889f62f\" (UID: \"0f6d78a2-1478-43fa-89e5-fd12a889f62f\") "
Oct 30 01:11:51.924408 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:51.924377    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6d78a2-1478-43fa-89e5-fd12a889f62f-maas-api-openshift-ca" pod "0f6d78a2-1478-43fa-89e5-fd12a889f62f" (UID: "0f6d78a2-1478-43fa-89e5-fd12a889f62f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:11:51.924669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:51.924623    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f6d78a2-1478-43fa-89e5-fd12a889f62f-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:51.926950 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:51.926921    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6d78a2-1478-43fa-89e5-fd12a889f62f-kube-api-access-wp2sd" pod "0f6d78a2-1478-43fa-89e5-fd12a889f62f" (UID: "0f6d78a2-1478-43fa-89e5-fd12a889f62f"). InnerVolumeSpecName "kube-api-access-wp2sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:51.930322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:51.930287    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6d78a2-1478-43fa-89e5-fd12a889f62f-maas-api-xks-ca" pod "0f6d78a2-1478-43fa-89e5-fd12a889f62f" (UID: "0f6d78a2-1478-43fa-89e5-fd12a889f62f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:52.025800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:52.025758    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f6d78a2-1478-43fa-89e5-fd12a889f62f-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:52.025800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:52.025798    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp2sd\" (UniqueName: \"kubernetes.io/projected/0f6d78a2-1478-43fa-89e5-fd12a889f62f-kube-api-access-wp2sd\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:11:52.709391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:52.709346    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h" event={"ID":"0f6d78a2-1478-43fa-89e5-fd12a889f62f","Type":"ContainerDied","Data":"996ecbf74ca1ad72121d8be251b9ad202fe49d4273a1dd9f0e1d53d4ef915b39"}
Oct 30 01:11:52.709391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:52.709397    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="996ecbf74ca1ad72121d8be251b9ad202fe49d4273a1dd9f0e1d53d4ef915b39"
Oct 30 01:11:52.709657 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:52.709362    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"
Oct 30 01:11:56.335203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:56.335151    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt_336deb2f-912c-4e44-bbf2-356952a1cebf/payload-processing/0.log"
Oct 30 01:11:57.220438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.220388    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"]
Oct 30 01:11:57.221189 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.221172    4425 cpu_manager.go:414] "containerMap: removing container" podUID="0f6d78a2-1478-43fa-89e5-fd12a889f62f" containerName="revoke-keys"
Oct 30 01:11:57.221255 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.221192    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f6d78a2-1478-43fa-89e5-fd12a889f62f" containerName="revoke-keys"
Oct 30 01:11:57.221389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.221377    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f6d78a2-1478-43fa-89e5-fd12a889f62f" containerName="revoke-keys"
Oct 30 01:11:57.225270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.225243    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"
Oct 30 01:11:57.249385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.249347    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:11:57.249385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.249347    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:11:57.253274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.253232    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"]
Oct 30 01:11:57.292590 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.292548    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d483132-836d-458c-b3b6-e009fd1b1b2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz\" (UID: \"9d483132-836d-458c-b3b6-e009fd1b1b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"
Oct 30 01:11:57.292776 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.292600    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d483132-836d-458c-b3b6-e009fd1b1b2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz\" (UID: \"9d483132-836d-458c-b3b6-e009fd1b1b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"
Oct 30 01:11:57.292776 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.292700    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s8jk\" (UniqueName: \"kubernetes.io/projected/9d483132-836d-458c-b3b6-e009fd1b1b2d-kube-api-access-5s8jk\") pod \"maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz\" (UID: \"9d483132-836d-458c-b3b6-e009fd1b1b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"
Oct 30 01:11:57.325086 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.325029    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:11:57.325336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.325296    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:11:57.394610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.394555    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d483132-836d-458c-b3b6-e009fd1b1b2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz\" (UID: \"9d483132-836d-458c-b3b6-e009fd1b1b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"
Oct 30 01:11:57.395103 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.394648    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d483132-836d-458c-b3b6-e009fd1b1b2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz\" (UID: \"9d483132-836d-458c-b3b6-e009fd1b1b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"
Oct 30 01:11:57.395103 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.394688    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s8jk\" (UniqueName: \"kubernetes.io/projected/9d483132-836d-458c-b3b6-e009fd1b1b2d-kube-api-access-5s8jk\") pod \"maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz\" (UID: \"9d483132-836d-458c-b3b6-e009fd1b1b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"
Oct 30 01:11:57.395387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.395367    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d483132-836d-458c-b3b6-e009fd1b1b2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz\" (UID: \"9d483132-836d-458c-b3b6-e009fd1b1b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"
Oct 30 01:11:57.397356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.397319    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d483132-836d-458c-b3b6-e009fd1b1b2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz\" (UID: \"9d483132-836d-458c-b3b6-e009fd1b1b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"
Oct 30 01:11:57.416342 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.416309    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s8jk\" (UniqueName: \"kubernetes.io/projected/9d483132-836d-458c-b3b6-e009fd1b1b2d-kube-api-access-5s8jk\") pod \"maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz\" (UID: \"9d483132-836d-458c-b3b6-e009fd1b1b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"
Oct 30 01:11:57.540729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.540685    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"
Oct 30 01:11:57.847087 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.847057    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"]
Oct 30 01:11:57.848505 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:11:57.848480    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d483132_836d_458c_b3b6_e009fd1b1b2d.slice/crio-70c12975175310641a012683972a5bead5e44787d85a41c0db44585377efcf04 WatchSource:0}: Error finding container 70c12975175310641a012683972a5bead5e44787d85a41c0db44585377efcf04: Status 404 returned error can't find the container with id 70c12975175310641a012683972a5bead5e44787d85a41c0db44585377efcf04
Oct 30 01:11:57.881225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.881189    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"]
Oct 30 01:11:57.886749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.886719    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"
Oct 30 01:11:57.907565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:57.907530    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"]
Oct 30 01:11:58.002626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.002583    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw\" (UID: \"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"
Oct 30 01:11:58.002827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.002642    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g95vg\" (UniqueName: \"kubernetes.io/projected/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-kube-api-access-g95vg\") pod \"maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw\" (UID: \"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"
Oct 30 01:11:58.002827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.002743    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw\" (UID: \"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"
Oct 30 01:11:58.104081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.103977    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw\" (UID: \"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"
Oct 30 01:11:58.104081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.104050    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g95vg\" (UniqueName: \"kubernetes.io/projected/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-kube-api-access-g95vg\") pod \"maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw\" (UID: \"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"
Oct 30 01:11:58.104386 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.104124    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw\" (UID: \"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"
Oct 30 01:11:58.104776 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.104757    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw\" (UID: \"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"
Oct 30 01:11:58.106495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.106464    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw\" (UID: \"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"
Oct 30 01:11:58.126011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.125979    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g95vg\" (UniqueName: \"kubernetes.io/projected/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-kube-api-access-g95vg\") pod \"maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw\" (UID: \"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"
Oct 30 01:11:58.204402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.204358    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"
Oct 30 01:11:58.550628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.550591    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"]
Oct 30 01:11:58.552070 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:11:58.552045    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5e01ab_e8e6_4c06_a3cd_f3d112b3900a.slice/crio-99efc1b5490c67ac8cdb0048ef4243415d4319adfcb6f70ac642c8ed57a3cdc5 WatchSource:0}: Error finding container 99efc1b5490c67ac8cdb0048ef4243415d4319adfcb6f70ac642c8ed57a3cdc5: Status 404 returned error can't find the container with id 99efc1b5490c67ac8cdb0048ef4243415d4319adfcb6f70ac642c8ed57a3cdc5
Oct 30 01:11:58.714441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.714406    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"]
Oct 30 01:11:58.720047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.720021    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"
Oct 30 01:11:58.745662 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.745613    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"]
Oct 30 01:11:58.762854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.762808    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw" event={"ID":"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a","Type":"ContainerStarted","Data":"99efc1b5490c67ac8cdb0048ef4243415d4319adfcb6f70ac642c8ed57a3cdc5"}
Oct 30 01:11:58.765134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.765093    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz" event={"ID":"9d483132-836d-458c-b3b6-e009fd1b1b2d","Type":"ContainerStarted","Data":"20ed2b33855a36d004c0691ddb49b7ace4a98a6bdfa3e487680b8750bbc20589"}
Oct 30 01:11:58.765311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.765192    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz" event={"ID":"9d483132-836d-458c-b3b6-e009fd1b1b2d","Type":"ContainerStarted","Data":"70c12975175310641a012683972a5bead5e44787d85a41c0db44585377efcf04"}
Oct 30 01:11:58.801828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.801706    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz" podStartSLOduration=1.80169046 podStartE2EDuration="1.80169046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:58.800827983 +0000 UTC m=+1827.550483293" watchObservedRunningTime="2026-10-30 01:11:58.80169046 +0000 UTC m=+1827.551345768"
Oct 30 01:11:58.813522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.813482    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f370d8c-e615-445e-a16c-b0931d6c174a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-045742de-4rqcn\" (UID: \"5f370d8c-e615-445e-a16c-b0931d6c174a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"
Oct 30 01:11:58.813522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.813526    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f370d8c-e615-445e-a16c-b0931d6c174a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-045742de-4rqcn\" (UID: \"5f370d8c-e615-445e-a16c-b0931d6c174a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"
Oct 30 01:11:58.813755 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.813732    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btnkx\" (UniqueName: \"kubernetes.io/projected/5f370d8c-e615-445e-a16c-b0931d6c174a-kube-api-access-btnkx\") pod \"maas-api-revoke-sub-045742de-4rqcn\" (UID: \"5f370d8c-e615-445e-a16c-b0931d6c174a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"
Oct 30 01:11:58.915263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.915222    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btnkx\" (UniqueName: \"kubernetes.io/projected/5f370d8c-e615-445e-a16c-b0931d6c174a-kube-api-access-btnkx\") pod \"maas-api-revoke-sub-045742de-4rqcn\" (UID: \"5f370d8c-e615-445e-a16c-b0931d6c174a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"
Oct 30 01:11:58.915469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.915363    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f370d8c-e615-445e-a16c-b0931d6c174a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-045742de-4rqcn\" (UID: \"5f370d8c-e615-445e-a16c-b0931d6c174a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"
Oct 30 01:11:58.915469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.915395    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f370d8c-e615-445e-a16c-b0931d6c174a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-045742de-4rqcn\" (UID: \"5f370d8c-e615-445e-a16c-b0931d6c174a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"
Oct 30 01:11:58.916135 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.916109    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f370d8c-e615-445e-a16c-b0931d6c174a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-045742de-4rqcn\" (UID: \"5f370d8c-e615-445e-a16c-b0931d6c174a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"
Oct 30 01:11:58.917821 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.917800    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f370d8c-e615-445e-a16c-b0931d6c174a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-045742de-4rqcn\" (UID: \"5f370d8c-e615-445e-a16c-b0931d6c174a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"
Oct 30 01:11:58.936567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:58.936535    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btnkx\" (UniqueName: \"kubernetes.io/projected/5f370d8c-e615-445e-a16c-b0931d6c174a-kube-api-access-btnkx\") pod \"maas-api-revoke-sub-045742de-4rqcn\" (UID: \"5f370d8c-e615-445e-a16c-b0931d6c174a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"
Oct 30 01:11:59.035742 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:59.035697    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"
Oct 30 01:11:59.340744 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:59.340675    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"]
Oct 30 01:11:59.341950 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:11:59.341917    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f370d8c_e615_445e_a16c_b0931d6c174a.slice/crio-cc10fb5e6be2df8f17873726868b98dad41b09cc4c68ba3ad5d9437d63ad5dc0 WatchSource:0}: Error finding container cc10fb5e6be2df8f17873726868b98dad41b09cc4c68ba3ad5d9437d63ad5dc0: Status 404 returned error can't find the container with id cc10fb5e6be2df8f17873726868b98dad41b09cc4c68ba3ad5d9437d63ad5dc0
Oct 30 01:11:59.773143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:59.773099    4425 generic.go:356] "Generic (PLEG): container finished" podID="9d483132-836d-458c-b3b6-e009fd1b1b2d" containerID="20ed2b33855a36d004c0691ddb49b7ace4a98a6bdfa3e487680b8750bbc20589" exitCode=0
Oct 30 01:11:59.773698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:59.773140    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz" event={"ID":"9d483132-836d-458c-b3b6-e009fd1b1b2d","Type":"ContainerDied","Data":"20ed2b33855a36d004c0691ddb49b7ace4a98a6bdfa3e487680b8750bbc20589"}
Oct 30 01:11:59.775062 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:59.775032    4425 generic.go:356] "Generic (PLEG): container finished" podID="db5e01ab-e8e6-4c06-a3cd-f3d112b3900a" containerID="505be1d96779e56e4d19002c2ec74eb00bf336d6379795a00faf77b42a33240e" exitCode=0
Oct 30 01:11:59.775201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:59.775113    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw" event={"ID":"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a","Type":"ContainerDied","Data":"505be1d96779e56e4d19002c2ec74eb00bf336d6379795a00faf77b42a33240e"}
Oct 30 01:11:59.776440 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:11:59.776419    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn" event={"ID":"5f370d8c-e615-445e-a16c-b0931d6c174a","Type":"ContainerStarted","Data":"cc10fb5e6be2df8f17873726868b98dad41b09cc4c68ba3ad5d9437d63ad5dc0"}
Oct 30 01:12:00.785567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:00.785529    4425 generic.go:356] "Generic (PLEG): container finished" podID="5f370d8c-e615-445e-a16c-b0931d6c174a" containerID="8fe564defd38f9e5957414429a7fe23f48e3add8858e1982dabbc40e30f054f8" exitCode=0
Oct 30 01:12:00.786016 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:00.785621    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn" event={"ID":"5f370d8c-e615-445e-a16c-b0931d6c174a","Type":"ContainerDied","Data":"8fe564defd38f9e5957414429a7fe23f48e3add8858e1982dabbc40e30f054f8"}
Oct 30 01:12:00.969021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:00.968991    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"
Oct 30 01:12:01.005592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.005557    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"
Oct 30 01:12:01.042805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.042697    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-kube-api-access-g95vg\" (UniqueName: \"kubernetes.io/projected/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-kube-api-access-g95vg\") pod \"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a\" (UID: \"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a\") "
Oct 30 01:12:01.043051 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.042820    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-maas-api-xks-ca\") pod \"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a\" (UID: \"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a\") "
Oct 30 01:12:01.043051 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.043017    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-maas-api-openshift-ca\") pod \"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a\" (UID: \"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a\") "
Oct 30 01:12:01.043943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.043911    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-maas-api-openshift-ca" pod "db5e01ab-e8e6-4c06-a3cd-f3d112b3900a" (UID: "db5e01ab-e8e6-4c06-a3cd-f3d112b3900a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:01.046310 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.046261    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-maas-api-xks-ca" pod "db5e01ab-e8e6-4c06-a3cd-f3d112b3900a" (UID: "db5e01ab-e8e6-4c06-a3cd-f3d112b3900a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:01.046646 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.046620    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-kube-api-access-g95vg" pod "db5e01ab-e8e6-4c06-a3cd-f3d112b3900a" (UID: "db5e01ab-e8e6-4c06-a3cd-f3d112b3900a"). InnerVolumeSpecName "kube-api-access-g95vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:01.144538 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.144497    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d483132-836d-458c-b3b6-e009fd1b1b2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d483132-836d-458c-b3b6-e009fd1b1b2d-maas-api-openshift-ca\") pod \"9d483132-836d-458c-b3b6-e009fd1b1b2d\" (UID: \"9d483132-836d-458c-b3b6-e009fd1b1b2d\") "
Oct 30 01:12:01.144538 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.144543    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d483132-836d-458c-b3b6-e009fd1b1b2d-kube-api-access-5s8jk\" (UniqueName: \"kubernetes.io/projected/9d483132-836d-458c-b3b6-e009fd1b1b2d-kube-api-access-5s8jk\") pod \"9d483132-836d-458c-b3b6-e009fd1b1b2d\" (UID: \"9d483132-836d-458c-b3b6-e009fd1b1b2d\") "
Oct 30 01:12:01.144809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.144585    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d483132-836d-458c-b3b6-e009fd1b1b2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d483132-836d-458c-b3b6-e009fd1b1b2d-maas-api-xks-ca\") pod \"9d483132-836d-458c-b3b6-e009fd1b1b2d\" (UID: \"9d483132-836d-458c-b3b6-e009fd1b1b2d\") "
Oct 30 01:12:01.144902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.144880    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d483132-836d-458c-b3b6-e009fd1b1b2d-maas-api-openshift-ca" pod "9d483132-836d-458c-b3b6-e009fd1b1b2d" (UID: "9d483132-836d-458c-b3b6-e009fd1b1b2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:01.145139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.145105    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g95vg\" (UniqueName: \"kubernetes.io/projected/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-kube-api-access-g95vg\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:01.145139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.145140    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d483132-836d-458c-b3b6-e009fd1b1b2d-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:01.145345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.145194    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:01.145345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.145206    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:01.146840 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.146812    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d483132-836d-458c-b3b6-e009fd1b1b2d-maas-api-xks-ca" pod "9d483132-836d-458c-b3b6-e009fd1b1b2d" (UID: "9d483132-836d-458c-b3b6-e009fd1b1b2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:01.146840 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.146825    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d483132-836d-458c-b3b6-e009fd1b1b2d-kube-api-access-5s8jk" pod "9d483132-836d-458c-b3b6-e009fd1b1b2d" (UID: "9d483132-836d-458c-b3b6-e009fd1b1b2d"). InnerVolumeSpecName "kube-api-access-5s8jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:01.246560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.246520    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s8jk\" (UniqueName: \"kubernetes.io/projected/9d483132-836d-458c-b3b6-e009fd1b1b2d-kube-api-access-5s8jk\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:01.246560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.246558    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d483132-836d-458c-b3b6-e009fd1b1b2d-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:01.793952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.793908    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"
Oct 30 01:12:01.794478 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.793908    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz" event={"ID":"9d483132-836d-458c-b3b6-e009fd1b1b2d","Type":"ContainerDied","Data":"70c12975175310641a012683972a5bead5e44787d85a41c0db44585377efcf04"}
Oct 30 01:12:01.794478 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.794143    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70c12975175310641a012683972a5bead5e44787d85a41c0db44585377efcf04"
Oct 30 01:12:01.795529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.795508    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"
Oct 30 01:12:01.795672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.795547    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw" event={"ID":"db5e01ab-e8e6-4c06-a3cd-f3d112b3900a","Type":"ContainerDied","Data":"99efc1b5490c67ac8cdb0048ef4243415d4319adfcb6f70ac642c8ed57a3cdc5"}
Oct 30 01:12:01.795672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.795595    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99efc1b5490c67ac8cdb0048ef4243415d4319adfcb6f70ac642c8ed57a3cdc5"
Oct 30 01:12:01.970424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:01.970383    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"
Oct 30 01:12:02.057053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:02.056962    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f370d8c-e615-445e-a16c-b0931d6c174a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f370d8c-e615-445e-a16c-b0931d6c174a-maas-api-openshift-ca\") pod \"5f370d8c-e615-445e-a16c-b0931d6c174a\" (UID: \"5f370d8c-e615-445e-a16c-b0931d6c174a\") "
Oct 30 01:12:02.057277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:02.057097    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f370d8c-e615-445e-a16c-b0931d6c174a-kube-api-access-btnkx\" (UniqueName: \"kubernetes.io/projected/5f370d8c-e615-445e-a16c-b0931d6c174a-kube-api-access-btnkx\") pod \"5f370d8c-e615-445e-a16c-b0931d6c174a\" (UID: \"5f370d8c-e615-445e-a16c-b0931d6c174a\") "
Oct 30 01:12:02.057277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:02.057247    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f370d8c-e615-445e-a16c-b0931d6c174a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f370d8c-e615-445e-a16c-b0931d6c174a-maas-api-xks-ca\") pod \"5f370d8c-e615-445e-a16c-b0931d6c174a\" (UID: \"5f370d8c-e615-445e-a16c-b0931d6c174a\") "
Oct 30 01:12:02.057813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:02.057436    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f370d8c-e615-445e-a16c-b0931d6c174a-maas-api-openshift-ca" pod "5f370d8c-e615-445e-a16c-b0931d6c174a" (UID: "5f370d8c-e615-445e-a16c-b0931d6c174a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:02.058438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:02.058412    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f370d8c-e615-445e-a16c-b0931d6c174a-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:02.059581 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:02.059545    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f370d8c-e615-445e-a16c-b0931d6c174a-maas-api-xks-ca" pod "5f370d8c-e615-445e-a16c-b0931d6c174a" (UID: "5f370d8c-e615-445e-a16c-b0931d6c174a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:02.059971 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:02.059950    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f370d8c-e615-445e-a16c-b0931d6c174a-kube-api-access-btnkx" pod "5f370d8c-e615-445e-a16c-b0931d6c174a" (UID: "5f370d8c-e615-445e-a16c-b0931d6c174a"). InnerVolumeSpecName "kube-api-access-btnkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:02.159966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:02.159926    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btnkx\" (UniqueName: \"kubernetes.io/projected/5f370d8c-e615-445e-a16c-b0931d6c174a-kube-api-access-btnkx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:02.159966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:02.159961    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f370d8c-e615-445e-a16c-b0931d6c174a-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:02.804601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:02.804564    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"
Oct 30 01:12:02.804601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:02.804587    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn" event={"ID":"5f370d8c-e615-445e-a16c-b0931d6c174a","Type":"ContainerDied","Data":"cc10fb5e6be2df8f17873726868b98dad41b09cc4c68ba3ad5d9437d63ad5dc0"}
Oct 30 01:12:02.805018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:02.804635    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc10fb5e6be2df8f17873726868b98dad41b09cc4c68ba3ad5d9437d63ad5dc0"
Oct 30 01:12:07.556369 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:12:07.556313    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:12:11.491731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.491692    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"]
Oct 30 01:12:11.492296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.492280    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9d483132-836d-458c-b3b6-e009fd1b1b2d" containerName="revoke-keys"
Oct 30 01:12:11.492349 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.492297    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d483132-836d-458c-b3b6-e009fd1b1b2d" containerName="revoke-keys"
Oct 30 01:12:11.492349 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.492309    4425 cpu_manager.go:414] "containerMap: removing container" podUID="db5e01ab-e8e6-4c06-a3cd-f3d112b3900a" containerName="revoke-keys"
Oct 30 01:12:11.492349 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.492314    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5e01ab-e8e6-4c06-a3cd-f3d112b3900a" containerName="revoke-keys"
Oct 30 01:12:11.492349 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.492333    4425 cpu_manager.go:414] "containerMap: removing container" podUID="5f370d8c-e615-445e-a16c-b0931d6c174a" containerName="revoke-keys"
Oct 30 01:12:11.492349 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.492338    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f370d8c-e615-445e-a16c-b0931d6c174a" containerName="revoke-keys"
Oct 30 01:12:11.492504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.492431    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f370d8c-e615-445e-a16c-b0931d6c174a" containerName="revoke-keys"
Oct 30 01:12:11.492504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.492445    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d483132-836d-458c-b3b6-e009fd1b1b2d" containerName="revoke-keys"
Oct 30 01:12:11.492504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.492452    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5e01ab-e8e6-4c06-a3cd-f3d112b3900a" containerName="revoke-keys"
Oct 30 01:12:11.497208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.497151    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"
Oct 30 01:12:11.512472 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.512440    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:12:11.522343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.522302    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:12:11.522343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.522314    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:12:11.522581 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.522389    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:12:11.526669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.526631    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"]
Oct 30 01:12:11.575468 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.575429    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85202810-d4nn6\" (UID: \"dad28d88-d05f-4e08-9d34-ba8d61a42c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"
Oct 30 01:12:11.575672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.575606    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kxms\" (UniqueName: \"kubernetes.io/projected/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-kube-api-access-5kxms\") pod \"maas-api-revoke-sub-85202810-d4nn6\" (UID: \"dad28d88-d05f-4e08-9d34-ba8d61a42c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"
Oct 30 01:12:11.575719 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.575683    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85202810-d4nn6\" (UID: \"dad28d88-d05f-4e08-9d34-ba8d61a42c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"
Oct 30 01:12:11.676819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.676779    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kxms\" (UniqueName: \"kubernetes.io/projected/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-kube-api-access-5kxms\") pod \"maas-api-revoke-sub-85202810-d4nn6\" (UID: \"dad28d88-d05f-4e08-9d34-ba8d61a42c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"
Oct 30 01:12:11.677031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.676847    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85202810-d4nn6\" (UID: \"dad28d88-d05f-4e08-9d34-ba8d61a42c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"
Oct 30 01:12:11.677031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.676984    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85202810-d4nn6\" (UID: \"dad28d88-d05f-4e08-9d34-ba8d61a42c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"
Oct 30 01:12:11.677593 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.677564    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85202810-d4nn6\" (UID: \"dad28d88-d05f-4e08-9d34-ba8d61a42c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"
Oct 30 01:12:11.679545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.679523    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85202810-d4nn6\" (UID: \"dad28d88-d05f-4e08-9d34-ba8d61a42c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"
Oct 30 01:12:11.698075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.698038    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kxms\" (UniqueName: \"kubernetes.io/projected/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-kube-api-access-5kxms\") pod \"maas-api-revoke-sub-85202810-d4nn6\" (UID: \"dad28d88-d05f-4e08-9d34-ba8d61a42c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"
Oct 30 01:12:11.811715 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:11.811671    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"
Oct 30 01:12:12.118225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:12.118199    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"]
Oct 30 01:12:12.120142 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:12:12.120111    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad28d88_d05f_4e08_9d34_ba8d61a42c5c.slice/crio-fed6c1200650a548b991e19b3f1bc7d8591f2ecec60d14491c341856015d4004 WatchSource:0}: Error finding container fed6c1200650a548b991e19b3f1bc7d8591f2ecec60d14491c341856015d4004: Status 404 returned error can't find the container with id fed6c1200650a548b991e19b3f1bc7d8591f2ecec60d14491c341856015d4004
Oct 30 01:12:12.887629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:12.887582    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6" event={"ID":"dad28d88-d05f-4e08-9d34-ba8d61a42c5c","Type":"ContainerStarted","Data":"b07d35f99516d4cecd41625c52309035095733153a957266a73c0c70f3fd36e7"}
Oct 30 01:12:12.888098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:12.887655    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6" event={"ID":"dad28d88-d05f-4e08-9d34-ba8d61a42c5c","Type":"ContainerStarted","Data":"fed6c1200650a548b991e19b3f1bc7d8591f2ecec60d14491c341856015d4004"}
Oct 30 01:12:12.926972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:12.926916    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6" podStartSLOduration=1.926899232 podStartE2EDuration="1.926899232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:12.923558234 +0000 UTC m=+1841.673213542" watchObservedRunningTime="2026-10-30 01:12:12.926899232 +0000 UTC m=+1841.676554541"
Oct 30 01:12:13.240946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:13.240839    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:12:13.895960 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:13.895922    4425 generic.go:356] "Generic (PLEG): container finished" podID="dad28d88-d05f-4e08-9d34-ba8d61a42c5c" containerID="b07d35f99516d4cecd41625c52309035095733153a957266a73c0c70f3fd36e7" exitCode=0
Oct 30 01:12:13.896493 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:13.896010    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6" event={"ID":"dad28d88-d05f-4e08-9d34-ba8d61a42c5c","Type":"ContainerDied","Data":"b07d35f99516d4cecd41625c52309035095733153a957266a73c0c70f3fd36e7"}
Oct 30 01:12:15.084442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:15.084407    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"
Oct 30 01:12:15.126992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:15.126949    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-maas-api-xks-ca\") pod \"dad28d88-d05f-4e08-9d34-ba8d61a42c5c\" (UID: \"dad28d88-d05f-4e08-9d34-ba8d61a42c5c\") "
Oct 30 01:12:15.127220 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:15.127017    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-kube-api-access-5kxms\" (UniqueName: \"kubernetes.io/projected/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-kube-api-access-5kxms\") pod \"dad28d88-d05f-4e08-9d34-ba8d61a42c5c\" (UID: \"dad28d88-d05f-4e08-9d34-ba8d61a42c5c\") "
Oct 30 01:12:15.127308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:15.127258    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-maas-api-openshift-ca\") pod \"dad28d88-d05f-4e08-9d34-ba8d61a42c5c\" (UID: \"dad28d88-d05f-4e08-9d34-ba8d61a42c5c\") "
Oct 30 01:12:15.127910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:15.127875    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-maas-api-openshift-ca" pod "dad28d88-d05f-4e08-9d34-ba8d61a42c5c" (UID: "dad28d88-d05f-4e08-9d34-ba8d61a42c5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:15.129737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:15.129709    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-kube-api-access-5kxms" pod "dad28d88-d05f-4e08-9d34-ba8d61a42c5c" (UID: "dad28d88-d05f-4e08-9d34-ba8d61a42c5c"). InnerVolumeSpecName "kube-api-access-5kxms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:15.132704 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:15.132672    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-maas-api-xks-ca" pod "dad28d88-d05f-4e08-9d34-ba8d61a42c5c" (UID: "dad28d88-d05f-4e08-9d34-ba8d61a42c5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:15.229652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:15.229556    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:15.229652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:15.229591    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:15.229652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:15.229600    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kxms\" (UniqueName: \"kubernetes.io/projected/dad28d88-d05f-4e08-9d34-ba8d61a42c5c-kube-api-access-5kxms\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:15.916280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:15.916250    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"
Oct 30 01:12:15.916448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:15.916275    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6" event={"ID":"dad28d88-d05f-4e08-9d34-ba8d61a42c5c","Type":"ContainerDied","Data":"fed6c1200650a548b991e19b3f1bc7d8591f2ecec60d14491c341856015d4004"}
Oct 30 01:12:15.916448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:15.916326    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fed6c1200650a548b991e19b3f1bc7d8591f2ecec60d14491c341856015d4004"
Oct 30 01:12:19.055847 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.055800    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"]
Oct 30 01:12:19.056798 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.056777    4425 cpu_manager.go:414] "containerMap: removing container" podUID="dad28d88-d05f-4e08-9d34-ba8d61a42c5c" containerName="revoke-keys"
Oct 30 01:12:19.056844 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.056800    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad28d88-d05f-4e08-9d34-ba8d61a42c5c" containerName="revoke-keys"
Oct 30 01:12:19.056956 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.056945    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad28d88-d05f-4e08-9d34-ba8d61a42c5c" containerName="revoke-keys"
Oct 30 01:12:19.061236 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.061208    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"
Oct 30 01:12:19.084628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.084587    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:12:19.084628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.084587    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:12:19.084628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.084591    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:12:19.084884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.084638    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:12:19.088929 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.088891    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"]
Oct 30 01:12:19.182289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.182254    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h\" (UID: \"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"
Oct 30 01:12:19.182525 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.182422    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h\" (UID: \"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"
Oct 30 01:12:19.182525 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.182519    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2srj\" (UniqueName: \"kubernetes.io/projected/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-kube-api-access-q2srj\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h\" (UID: \"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"
Oct 30 01:12:19.284089 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.284040    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h\" (UID: \"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"
Oct 30 01:12:19.284291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.284170    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h\" (UID: \"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"
Oct 30 01:12:19.284291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.284268    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2srj\" (UniqueName: \"kubernetes.io/projected/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-kube-api-access-q2srj\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h\" (UID: \"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"
Oct 30 01:12:19.284762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.284737    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h\" (UID: \"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"
Oct 30 01:12:19.286758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.286726    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h\" (UID: \"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"
Oct 30 01:12:19.305511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.305472    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2srj\" (UniqueName: \"kubernetes.io/projected/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-kube-api-access-q2srj\") pod \"maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h\" (UID: \"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"
Oct 30 01:12:19.376023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.375929    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"
Oct 30 01:12:19.688044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.688005    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"]
Oct 30 01:12:19.689986 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:12:19.689945    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8ce56a_093a_41a8_a0f8_65a0bf9dfe3d.slice/crio-79dbc011f83855b35e8df21965b34c786c9ffc517b9e4d5ac3e7d254be78aad5 WatchSource:0}: Error finding container 79dbc011f83855b35e8df21965b34c786c9ffc517b9e4d5ac3e7d254be78aad5: Status 404 returned error can't find the container with id 79dbc011f83855b35e8df21965b34c786c9ffc517b9e4d5ac3e7d254be78aad5
Oct 30 01:12:19.945787 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:19.945689    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h" event={"ID":"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d","Type":"ContainerStarted","Data":"79dbc011f83855b35e8df21965b34c786c9ffc517b9e4d5ac3e7d254be78aad5"}
Oct 30 01:12:20.954584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:20.954474    4425 generic.go:356] "Generic (PLEG): container finished" podID="0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d" containerID="64dee34f57a29b2e80e449c6ab8fc25448026322fa21597771cbeeb5bd185e73" exitCode=0
Oct 30 01:12:20.954584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:20.954556    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h" event={"ID":"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d","Type":"ContainerDied","Data":"64dee34f57a29b2e80e449c6ab8fc25448026322fa21597771cbeeb5bd185e73"}
Oct 30 01:12:22.151879 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:22.151848    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"
Oct 30 01:12:22.224556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:22.224428    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-kube-api-access-q2srj\" (UniqueName: \"kubernetes.io/projected/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-kube-api-access-q2srj\") pod \"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d\" (UID: \"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d\") "
Oct 30 01:12:22.224556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:22.224548    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-maas-api-openshift-ca\") pod \"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d\" (UID: \"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d\") "
Oct 30 01:12:22.224840 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:22.224677    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-maas-api-xks-ca\") pod \"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d\" (UID: \"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d\") "
Oct 30 01:12:22.225200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:22.225141    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-maas-api-openshift-ca" pod "0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d" (UID: "0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:22.225333 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:22.225258    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:22.228095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:22.227779    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-maas-api-xks-ca" pod "0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d" (UID: "0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:22.232472 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:22.229548    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-kube-api-access-q2srj" pod "0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d" (UID: "0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d"). InnerVolumeSpecName "kube-api-access-q2srj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:22.326993 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:22.326951    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2srj\" (UniqueName: \"kubernetes.io/projected/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-kube-api-access-q2srj\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:22.326993 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:22.326986    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:22.970916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:22.970880    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"
Oct 30 01:12:22.970916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:22.970896    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h" event={"ID":"0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d","Type":"ContainerDied","Data":"79dbc011f83855b35e8df21965b34c786c9ffc517b9e4d5ac3e7d254be78aad5"}
Oct 30 01:12:22.971223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:22.970965    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79dbc011f83855b35e8df21965b34c786c9ffc517b9e4d5ac3e7d254be78aad5"
Oct 30 01:12:23.901273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:23.901223    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"]
Oct 30 01:12:23.902122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:23.902097    4425 cpu_manager.go:414] "containerMap: removing container" podUID="0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d" containerName="revoke-keys"
Oct 30 01:12:23.902201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:23.902126    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d" containerName="revoke-keys"
Oct 30 01:12:23.902341 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:23.902327    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d" containerName="revoke-keys"
Oct 30 01:12:23.906643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:23.906613    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"
Oct 30 01:12:23.924173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:23.924123    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"]
Oct 30 01:12:23.929606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:23.929571    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:12:23.929606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:23.929581    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:12:23.929831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:23.929581    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:12:23.929831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:23.929632    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:12:24.048495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:24.048442    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7l5n\" (UniqueName: \"kubernetes.io/projected/115c494a-00fd-4c37-a258-e9675dd36c02-kube-api-access-h7l5n\") pod \"maas-api-revoke-sub-c51e9993-lnv9p\" (UID: \"115c494a-00fd-4c37-a258-e9675dd36c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"
Oct 30 01:12:24.048724 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:24.048586    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115c494a-00fd-4c37-a258-e9675dd36c02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c51e9993-lnv9p\" (UID: \"115c494a-00fd-4c37-a258-e9675dd36c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"
Oct 30 01:12:24.048724 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:24.048670    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115c494a-00fd-4c37-a258-e9675dd36c02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c51e9993-lnv9p\" (UID: \"115c494a-00fd-4c37-a258-e9675dd36c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"
Oct 30 01:12:24.149906 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:24.149868    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115c494a-00fd-4c37-a258-e9675dd36c02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c51e9993-lnv9p\" (UID: \"115c494a-00fd-4c37-a258-e9675dd36c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"
Oct 30 01:12:24.150110 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:24.150004    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7l5n\" (UniqueName: \"kubernetes.io/projected/115c494a-00fd-4c37-a258-e9675dd36c02-kube-api-access-h7l5n\") pod \"maas-api-revoke-sub-c51e9993-lnv9p\" (UID: \"115c494a-00fd-4c37-a258-e9675dd36c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"
Oct 30 01:12:24.150110 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:24.150104    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115c494a-00fd-4c37-a258-e9675dd36c02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c51e9993-lnv9p\" (UID: \"115c494a-00fd-4c37-a258-e9675dd36c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"
Oct 30 01:12:24.150936 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:24.150903    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115c494a-00fd-4c37-a258-e9675dd36c02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c51e9993-lnv9p\" (UID: \"115c494a-00fd-4c37-a258-e9675dd36c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"
Oct 30 01:12:24.152504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:24.152445    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115c494a-00fd-4c37-a258-e9675dd36c02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c51e9993-lnv9p\" (UID: \"115c494a-00fd-4c37-a258-e9675dd36c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"
Oct 30 01:12:24.173346 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:24.173308    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7l5n\" (UniqueName: \"kubernetes.io/projected/115c494a-00fd-4c37-a258-e9675dd36c02-kube-api-access-h7l5n\") pod \"maas-api-revoke-sub-c51e9993-lnv9p\" (UID: \"115c494a-00fd-4c37-a258-e9675dd36c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"
Oct 30 01:12:24.222929 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:24.222886    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"
Oct 30 01:12:24.571936 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:24.571900    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"]
Oct 30 01:12:24.572463 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:12:24.572430    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115c494a_00fd_4c37_a258_e9675dd36c02.slice/crio-a0494845e6ae1a51d7cd202fed5166646649beb56608820667cc3dd00e9a15d0 WatchSource:0}: Error finding container a0494845e6ae1a51d7cd202fed5166646649beb56608820667cc3dd00e9a15d0: Status 404 returned error can't find the container with id a0494845e6ae1a51d7cd202fed5166646649beb56608820667cc3dd00e9a15d0
Oct 30 01:12:24.616354 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:24.616312    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"]
Oct 30 01:12:24.626856 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:24.626817    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-975807-5c9bz"]
Oct 30 01:12:24.990167 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:24.990121    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p" event={"ID":"115c494a-00fd-4c37-a258-e9675dd36c02","Type":"ContainerStarted","Data":"2fefc02d9d273169e59171067592ea881e7a7962290295507852fad27044062d"}
Oct 30 01:12:24.990607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:24.990230    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p" event={"ID":"115c494a-00fd-4c37-a258-e9675dd36c02","Type":"ContainerStarted","Data":"a0494845e6ae1a51d7cd202fed5166646649beb56608820667cc3dd00e9a15d0"}
Oct 30 01:12:25.027792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:25.027728    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p" podStartSLOduration=2.027713872 podStartE2EDuration="2.027713872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:25.025641545 +0000 UTC m=+1853.775296855" watchObservedRunningTime="2026-10-30 01:12:25.027713872 +0000 UTC m=+1853.777369180"
Oct 30 01:12:25.428245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:25.428209    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d483132-836d-458c-b3b6-e009fd1b1b2d" path="/var/lib/kubelet/pods/9d483132-836d-458c-b3b6-e009fd1b1b2d/volumes"
Oct 30 01:12:25.998844 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:25.998810    4425 generic.go:356] "Generic (PLEG): container finished" podID="115c494a-00fd-4c37-a258-e9675dd36c02" containerID="2fefc02d9d273169e59171067592ea881e7a7962290295507852fad27044062d" exitCode=0
Oct 30 01:12:25.999296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:25.998908    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p" event={"ID":"115c494a-00fd-4c37-a258-e9675dd36c02","Type":"ContainerDied","Data":"2fefc02d9d273169e59171067592ea881e7a7962290295507852fad27044062d"}
Oct 30 01:12:27.180631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:27.180603    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"
Oct 30 01:12:27.292097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:27.292048    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115c494a-00fd-4c37-a258-e9675dd36c02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115c494a-00fd-4c37-a258-e9675dd36c02-maas-api-openshift-ca\") pod \"115c494a-00fd-4c37-a258-e9675dd36c02\" (UID: \"115c494a-00fd-4c37-a258-e9675dd36c02\") "
Oct 30 01:12:27.292319 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:27.292230    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/115c494a-00fd-4c37-a258-e9675dd36c02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115c494a-00fd-4c37-a258-e9675dd36c02-maas-api-xks-ca\") pod \"115c494a-00fd-4c37-a258-e9675dd36c02\" (UID: \"115c494a-00fd-4c37-a258-e9675dd36c02\") "
Oct 30 01:12:27.292319 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:27.292298    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115c494a-00fd-4c37-a258-e9675dd36c02-kube-api-access-h7l5n\" (UniqueName: \"kubernetes.io/projected/115c494a-00fd-4c37-a258-e9675dd36c02-kube-api-access-h7l5n\") pod \"115c494a-00fd-4c37-a258-e9675dd36c02\" (UID: \"115c494a-00fd-4c37-a258-e9675dd36c02\") "
Oct 30 01:12:27.292592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:27.292567    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115c494a-00fd-4c37-a258-e9675dd36c02-maas-api-openshift-ca" pod "115c494a-00fd-4c37-a258-e9675dd36c02" (UID: "115c494a-00fd-4c37-a258-e9675dd36c02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:27.292863 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:27.292846    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/115c494a-00fd-4c37-a258-e9675dd36c02-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:27.294601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:27.294563    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115c494a-00fd-4c37-a258-e9675dd36c02-kube-api-access-h7l5n" pod "115c494a-00fd-4c37-a258-e9675dd36c02" (UID: "115c494a-00fd-4c37-a258-e9675dd36c02"). InnerVolumeSpecName "kube-api-access-h7l5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:27.294794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:27.294655    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/115c494a-00fd-4c37-a258-e9675dd36c02-maas-api-xks-ca" pod "115c494a-00fd-4c37-a258-e9675dd36c02" (UID: "115c494a-00fd-4c37-a258-e9675dd36c02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:27.394518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:27.394478    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/115c494a-00fd-4c37-a258-e9675dd36c02-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:27.394518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:27.394510    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7l5n\" (UniqueName: \"kubernetes.io/projected/115c494a-00fd-4c37-a258-e9675dd36c02-kube-api-access-h7l5n\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:27.993077 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:27.993028    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"]
Oct 30 01:12:27.993784 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:27.993761    4425 cpu_manager.go:414] "containerMap: removing container" podUID="115c494a-00fd-4c37-a258-e9675dd36c02" containerName="revoke-keys"
Oct 30 01:12:27.993784 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:27.993784    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115c494a-00fd-4c37-a258-e9675dd36c02" containerName="revoke-keys"
Oct 30 01:12:27.993991 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:27.993920    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="115c494a-00fd-4c37-a258-e9675dd36c02" containerName="revoke-keys"
Oct 30 01:12:27.997653 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:27.997625    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"
Oct 30 01:12:28.016505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:28.016453    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p" event={"ID":"115c494a-00fd-4c37-a258-e9675dd36c02","Type":"ContainerDied","Data":"a0494845e6ae1a51d7cd202fed5166646649beb56608820667cc3dd00e9a15d0"}
Oct 30 01:12:28.016766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:28.016750    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0494845e6ae1a51d7cd202fed5166646649beb56608820667cc3dd00e9a15d0"
Oct 30 01:12:28.016942 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:28.016918    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"
Oct 30 01:12:28.019262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:28.019233    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"]
Oct 30 01:12:28.106196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:28.106117    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqgkq\" (UniqueName: \"kubernetes.io/projected/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-kube-api-access-fqgkq\") pod \"maas-api-revoke-sub-e6a9a4c7-68nbp\" (UID: \"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"
Oct 30 01:12:28.106414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:28.106305    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6a9a4c7-68nbp\" (UID: \"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"
Oct 30 01:12:28.106414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:28.106334    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6a9a4c7-68nbp\" (UID: \"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"
Oct 30 01:12:28.207599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:28.207558    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqgkq\" (UniqueName: \"kubernetes.io/projected/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-kube-api-access-fqgkq\") pod \"maas-api-revoke-sub-e6a9a4c7-68nbp\" (UID: \"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"
Oct 30 01:12:28.208038 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:28.207639    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6a9a4c7-68nbp\" (UID: \"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"
Oct 30 01:12:28.208038 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:28.207666    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6a9a4c7-68nbp\" (UID: \"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"
Oct 30 01:12:28.208465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:28.208441    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6a9a4c7-68nbp\" (UID: \"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"
Oct 30 01:12:28.210325 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:28.210305    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6a9a4c7-68nbp\" (UID: \"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"
Oct 30 01:12:28.229812 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:28.229565    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqgkq\" (UniqueName: \"kubernetes.io/projected/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-kube-api-access-fqgkq\") pod \"maas-api-revoke-sub-e6a9a4c7-68nbp\" (UID: \"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"
Oct 30 01:12:28.315044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:28.315009    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"
Oct 30 01:12:28.626824 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:28.626794    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"]
Oct 30 01:12:28.629239 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:12:28.629207    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb662dd3_c1cb_4cd8_8d8e_e6cd2428cc21.slice/crio-2149af8208d865c1e9c8d55d2240c6a4f19494b52a9f80cf0d6c4a970ffa53f6 WatchSource:0}: Error finding container 2149af8208d865c1e9c8d55d2240c6a4f19494b52a9f80cf0d6c4a970ffa53f6: Status 404 returned error can't find the container with id 2149af8208d865c1e9c8d55d2240c6a4f19494b52a9f80cf0d6c4a970ffa53f6
Oct 30 01:12:29.025094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:29.025045    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp" event={"ID":"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21","Type":"ContainerStarted","Data":"2149af8208d865c1e9c8d55d2240c6a4f19494b52a9f80cf0d6c4a970ffa53f6"}
Oct 30 01:12:29.589915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:29.589873    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"]
Oct 30 01:12:29.633805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:29.633755    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"]
Oct 30 01:12:30.035890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:30.035844    4425 generic.go:356] "Generic (PLEG): container finished" podID="db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21" containerID="510dec8e1beeb9bd9f6789530324863381d3eb6b3c6108267895af9fca2ec44a" exitCode=0
Oct 30 01:12:30.036117 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:30.035930    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp" event={"ID":"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21","Type":"ContainerDied","Data":"510dec8e1beeb9bd9f6789530324863381d3eb6b3c6108267895af9fca2ec44a"}
Oct 30 01:12:31.192864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:31.192783    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"]
Oct 30 01:12:31.209889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:31.209836    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5f8cbc-5r82h"]
Oct 30 01:12:31.226964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:31.226927    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"
Oct 30 01:12:31.276711 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:31.276674    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-maas-api-xks-ca\") pod \"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21\" (UID: \"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21\") "
Oct 30 01:12:31.276907 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:31.276867    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-kube-api-access-fqgkq\" (UniqueName: \"kubernetes.io/projected/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-kube-api-access-fqgkq\") pod \"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21\" (UID: \"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21\") "
Oct 30 01:12:31.277002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:31.276919    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-maas-api-openshift-ca\") pod \"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21\" (UID: \"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21\") "
Oct 30 01:12:31.279633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:31.277908    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-maas-api-openshift-ca" pod "db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21" (UID: "db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:31.279633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:31.279327    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-maas-api-xks-ca" pod "db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21" (UID: "db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:31.280012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:31.279980    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-kube-api-access-fqgkq" pod "db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21" (UID: "db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21"). InnerVolumeSpecName "kube-api-access-fqgkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:31.378570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:31.378458    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:31.378570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:31.378510    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:31.378570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:31.378520    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqgkq\" (UniqueName: \"kubernetes.io/projected/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21-kube-api-access-fqgkq\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:31.423637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:31.423603    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d" path="/var/lib/kubelet/pods/0e8ce56a-093a-41a8-a0f8-65a0bf9dfe3d/volumes"
Oct 30 01:12:32.063821 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:32.063792    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"
Oct 30 01:12:32.064036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:32.063823    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp" event={"ID":"db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21","Type":"ContainerDied","Data":"2149af8208d865c1e9c8d55d2240c6a4f19494b52a9f80cf0d6c4a970ffa53f6"}
Oct 30 01:12:32.064036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:32.063889    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2149af8208d865c1e9c8d55d2240c6a4f19494b52a9f80cf0d6c4a970ffa53f6"
Oct 30 01:12:32.739125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:32.739079    4425 scope.go:122] "RemoveContainer" containerID="de6ffc8d49d69f4343c624eda31fc68e3d4e3c8f8810cd98d17a75feeeb4e5de"
Oct 30 01:12:34.300016 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.299970    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"]
Oct 30 01:12:34.300808 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.300783    4425 cpu_manager.go:414] "containerMap: removing container" podUID="db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21" containerName="revoke-keys"
Oct 30 01:12:34.300929 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.300809    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21" containerName="revoke-keys"
Oct 30 01:12:34.301017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.301003    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21" containerName="revoke-keys"
Oct 30 01:12:34.308317 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.308272    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"
Oct 30 01:12:34.322913 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.322880    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:12:34.326857 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.326824    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"]
Oct 30 01:12:34.332561 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.332534    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:12:34.332561 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.332552    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:12:34.332785 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.332594    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:12:34.419526 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.419476    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed932198-ee76-4ff5-a777-34e0363f290f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24e8b543-84r2f\" (UID: \"ed932198-ee76-4ff5-a777-34e0363f290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"
Oct 30 01:12:34.419709 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.419620    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed932198-ee76-4ff5-a777-34e0363f290f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24e8b543-84r2f\" (UID: \"ed932198-ee76-4ff5-a777-34e0363f290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"
Oct 30 01:12:34.419709 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.419659    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n95lw\" (UniqueName: \"kubernetes.io/projected/ed932198-ee76-4ff5-a777-34e0363f290f-kube-api-access-n95lw\") pod \"maas-api-revoke-sub-24e8b543-84r2f\" (UID: \"ed932198-ee76-4ff5-a777-34e0363f290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"
Oct 30 01:12:34.520395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.520357    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed932198-ee76-4ff5-a777-34e0363f290f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24e8b543-84r2f\" (UID: \"ed932198-ee76-4ff5-a777-34e0363f290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"
Oct 30 01:12:34.520578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.520406    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n95lw\" (UniqueName: \"kubernetes.io/projected/ed932198-ee76-4ff5-a777-34e0363f290f-kube-api-access-n95lw\") pod \"maas-api-revoke-sub-24e8b543-84r2f\" (UID: \"ed932198-ee76-4ff5-a777-34e0363f290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"
Oct 30 01:12:34.520578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.520526    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed932198-ee76-4ff5-a777-34e0363f290f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24e8b543-84r2f\" (UID: \"ed932198-ee76-4ff5-a777-34e0363f290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"
Oct 30 01:12:34.521203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.521168    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed932198-ee76-4ff5-a777-34e0363f290f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24e8b543-84r2f\" (UID: \"ed932198-ee76-4ff5-a777-34e0363f290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"
Oct 30 01:12:34.523027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.523001    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed932198-ee76-4ff5-a777-34e0363f290f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24e8b543-84r2f\" (UID: \"ed932198-ee76-4ff5-a777-34e0363f290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"
Oct 30 01:12:34.542290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.542249    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n95lw\" (UniqueName: \"kubernetes.io/projected/ed932198-ee76-4ff5-a777-34e0363f290f-kube-api-access-n95lw\") pod \"maas-api-revoke-sub-24e8b543-84r2f\" (UID: \"ed932198-ee76-4ff5-a777-34e0363f290f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"
Oct 30 01:12:34.591100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.590974    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v" podUID="f16cc640-5e15-45f9-87ba-942cf8a21325" containerName="payload-processing" containerID="cri-o://06b1734144dd952c8f6399f675b9c589f84ceaef0d6a1df1460b4f185b1efc68" gracePeriod=25
Oct 30 01:12:34.625544 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.625507    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"
Oct 30 01:12:34.634730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.634650    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z" podUID="9eee6597-2b6b-4760-9144-4901fc0c342b" containerName="payload-pre-processing" containerID="cri-o://6b3d3d9e1570205ae2873503bd0449139d6751fdb31f79d859085b164844f4f9" gracePeriod=25
Oct 30 01:12:34.913888 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.913854    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"
Oct 30 01:12:34.939925 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.939889    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"]
Oct 30 01:12:34.941729 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:12:34.941699    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded932198_ee76_4ff5_a777_34e0363f290f.slice/crio-3b269339c8c1787cd16ff8d9cf07ecd8151d1881181ff51424e53dc6b0038a77 WatchSource:0}: Error finding container 3b269339c8c1787cd16ff8d9cf07ecd8151d1881181ff51424e53dc6b0038a77: Status 404 returned error can't find the container with id 3b269339c8c1787cd16ff8d9cf07ecd8151d1881181ff51424e53dc6b0038a77
Oct 30 01:12:34.956566 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:34.956534    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"
Oct 30 01:12:35.027058 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.027006    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eee6597-2b6b-4760-9144-4901fc0c342b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eee6597-2b6b-4760-9144-4901fc0c342b-plugins-config-volume\") pod \"9eee6597-2b6b-4760-9144-4901fc0c342b\" (UID: \"9eee6597-2b6b-4760-9144-4901fc0c342b\") "
Oct 30 01:12:35.027310 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.027122    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16cc640-5e15-45f9-87ba-942cf8a21325-kube-api-access-w9n22\" (UniqueName: \"kubernetes.io/projected/f16cc640-5e15-45f9-87ba-942cf8a21325-kube-api-access-w9n22\") pod \"f16cc640-5e15-45f9-87ba-942cf8a21325\" (UID: \"f16cc640-5e15-45f9-87ba-942cf8a21325\") "
Oct 30 01:12:35.027310 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.027254    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eee6597-2b6b-4760-9144-4901fc0c342b-kube-api-access-cspqt\" (UniqueName: \"kubernetes.io/projected/9eee6597-2b6b-4760-9144-4901fc0c342b-kube-api-access-cspqt\") pod \"9eee6597-2b6b-4760-9144-4901fc0c342b\" (UID: \"9eee6597-2b6b-4760-9144-4901fc0c342b\") "
Oct 30 01:12:35.027479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.027383    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16cc640-5e15-45f9-87ba-942cf8a21325-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f16cc640-5e15-45f9-87ba-942cf8a21325-plugins-config-volume\") pod \"f16cc640-5e15-45f9-87ba-942cf8a21325\" (UID: \"f16cc640-5e15-45f9-87ba-942cf8a21325\") "
Oct 30 01:12:35.027539 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.027496    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eee6597-2b6b-4760-9144-4901fc0c342b-plugins-config-volume" pod "9eee6597-2b6b-4760-9144-4901fc0c342b" (UID: "9eee6597-2b6b-4760-9144-4901fc0c342b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:35.027920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.027888    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eee6597-2b6b-4760-9144-4901fc0c342b-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:35.028052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.027965    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16cc640-5e15-45f9-87ba-942cf8a21325-plugins-config-volume" pod "f16cc640-5e15-45f9-87ba-942cf8a21325" (UID: "f16cc640-5e15-45f9-87ba-942cf8a21325"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:35.029598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.029557    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16cc640-5e15-45f9-87ba-942cf8a21325-kube-api-access-w9n22" pod "f16cc640-5e15-45f9-87ba-942cf8a21325" (UID: "f16cc640-5e15-45f9-87ba-942cf8a21325"). InnerVolumeSpecName "kube-api-access-w9n22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:35.030325 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.030293    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eee6597-2b6b-4760-9144-4901fc0c342b-kube-api-access-cspqt" pod "9eee6597-2b6b-4760-9144-4901fc0c342b" (UID: "9eee6597-2b6b-4760-9144-4901fc0c342b"). InnerVolumeSpecName "kube-api-access-cspqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:35.087558 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.087514    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f" event={"ID":"ed932198-ee76-4ff5-a777-34e0363f290f","Type":"ContainerStarted","Data":"3b269339c8c1787cd16ff8d9cf07ecd8151d1881181ff51424e53dc6b0038a77"}
Oct 30 01:12:35.089075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.089034    4425 generic.go:356] "Generic (PLEG): container finished" podID="f16cc640-5e15-45f9-87ba-942cf8a21325" containerID="06b1734144dd952c8f6399f675b9c589f84ceaef0d6a1df1460b4f185b1efc68" exitCode=0
Oct 30 01:12:35.089274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.089093    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"
Oct 30 01:12:35.089274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.089116    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v" event={"ID":"f16cc640-5e15-45f9-87ba-942cf8a21325","Type":"ContainerDied","Data":"06b1734144dd952c8f6399f675b9c589f84ceaef0d6a1df1460b4f185b1efc68"}
Oct 30 01:12:35.089274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.089204    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v" event={"ID":"f16cc640-5e15-45f9-87ba-942cf8a21325","Type":"ContainerDied","Data":"014c1bd4f0de8a60da38b161254b0a7de46c41d20e19d0ea3ac0d790395fcbc7"}
Oct 30 01:12:35.089274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.089252    4425 scope.go:122] "RemoveContainer" containerID="06b1734144dd952c8f6399f675b9c589f84ceaef0d6a1df1460b4f185b1efc68"
Oct 30 01:12:35.090980 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.090943    4425 generic.go:356] "Generic (PLEG): container finished" podID="9eee6597-2b6b-4760-9144-4901fc0c342b" containerID="6b3d3d9e1570205ae2873503bd0449139d6751fdb31f79d859085b164844f4f9" exitCode=0
Oct 30 01:12:35.091097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.091004    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"
Oct 30 01:12:35.091097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.091018    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z" event={"ID":"9eee6597-2b6b-4760-9144-4901fc0c342b","Type":"ContainerDied","Data":"6b3d3d9e1570205ae2873503bd0449139d6751fdb31f79d859085b164844f4f9"}
Oct 30 01:12:35.091097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.091088    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z" event={"ID":"9eee6597-2b6b-4760-9144-4901fc0c342b","Type":"ContainerDied","Data":"177ebe244f4f0f0f5c5429335f1a81ce7208038279e35f877ef743425a34f0aa"}
Oct 30 01:12:35.101339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.101268    4425 scope.go:122] "RemoveContainer" containerID="06b1734144dd952c8f6399f675b9c589f84ceaef0d6a1df1460b4f185b1efc68"
Oct 30 01:12:35.102170 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:12:35.102133    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b1734144dd952c8f6399f675b9c589f84ceaef0d6a1df1460b4f185b1efc68\": container with ID starting with 06b1734144dd952c8f6399f675b9c589f84ceaef0d6a1df1460b4f185b1efc68 not found: ID does not exist" containerID="06b1734144dd952c8f6399f675b9c589f84ceaef0d6a1df1460b4f185b1efc68"
Oct 30 01:12:35.102300 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.102210    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b1734144dd952c8f6399f675b9c589f84ceaef0d6a1df1460b4f185b1efc68"} err="failed to get container status \"06b1734144dd952c8f6399f675b9c589f84ceaef0d6a1df1460b4f185b1efc68\": rpc error: code = NotFound desc = could not find container \"06b1734144dd952c8f6399f675b9c589f84ceaef0d6a1df1460b4f185b1efc68\": container with ID starting with 06b1734144dd952c8f6399f675b9c589f84ceaef0d6a1df1460b4f185b1efc68 not found: ID does not exist"
Oct 30 01:12:35.102300 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.102234    4425 scope.go:122] "RemoveContainer" containerID="6b3d3d9e1570205ae2873503bd0449139d6751fdb31f79d859085b164844f4f9"
Oct 30 01:12:35.114697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.114666    4425 scope.go:122] "RemoveContainer" containerID="6b3d3d9e1570205ae2873503bd0449139d6751fdb31f79d859085b164844f4f9"
Oct 30 01:12:35.115117 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:12:35.115088    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3d3d9e1570205ae2873503bd0449139d6751fdb31f79d859085b164844f4f9\": container with ID starting with 6b3d3d9e1570205ae2873503bd0449139d6751fdb31f79d859085b164844f4f9 not found: ID does not exist" containerID="6b3d3d9e1570205ae2873503bd0449139d6751fdb31f79d859085b164844f4f9"
Oct 30 01:12:35.115243 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.115129    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3d3d9e1570205ae2873503bd0449139d6751fdb31f79d859085b164844f4f9"} err="failed to get container status \"6b3d3d9e1570205ae2873503bd0449139d6751fdb31f79d859085b164844f4f9\": rpc error: code = NotFound desc = could not find container \"6b3d3d9e1570205ae2873503bd0449139d6751fdb31f79d859085b164844f4f9\": container with ID starting with 6b3d3d9e1570205ae2873503bd0449139d6751fdb31f79d859085b164844f4f9 not found: ID does not exist"
Oct 30 01:12:35.129014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.128929    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f16cc640-5e15-45f9-87ba-942cf8a21325-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:35.129014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.128962    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9n22\" (UniqueName: \"kubernetes.io/projected/f16cc640-5e15-45f9-87ba-942cf8a21325-kube-api-access-w9n22\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:35.129014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.128973    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cspqt\" (UniqueName: \"kubernetes.io/projected/9eee6597-2b6b-4760-9144-4901fc0c342b-kube-api-access-cspqt\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:35.135541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.135500    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"]
Oct 30 01:12:35.142349 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.142310    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5f8cbc-d876bc7bb-gll6v"]
Oct 30 01:12:35.181501 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.181469    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"]
Oct 30 01:12:35.185017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.184981    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5f8cbc-777fdcdbf5-bkw9z"]
Oct 30 01:12:35.422946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.422862    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eee6597-2b6b-4760-9144-4901fc0c342b" path="/var/lib/kubelet/pods/9eee6597-2b6b-4760-9144-4901fc0c342b/volumes"
Oct 30 01:12:35.423364 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:35.423338    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f16cc640-5e15-45f9-87ba-942cf8a21325" path="/var/lib/kubelet/pods/f16cc640-5e15-45f9-87ba-942cf8a21325/volumes"
Oct 30 01:12:36.099243 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:36.099201    4425 generic.go:356] "Generic (PLEG): container finished" podID="ed932198-ee76-4ff5-a777-34e0363f290f" containerID="c43b09aa2dcf6f5b50167b7812b9c8478f8e34653ff4c3e73056797b93f6957c" exitCode=0
Oct 30 01:12:36.099447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:36.099288    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f" event={"ID":"ed932198-ee76-4ff5-a777-34e0363f290f","Type":"ContainerDied","Data":"c43b09aa2dcf6f5b50167b7812b9c8478f8e34653ff4c3e73056797b93f6957c"}
Oct 30 01:12:37.280402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:37.280366    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"
Oct 30 01:12:37.354181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:37.354114    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed932198-ee76-4ff5-a777-34e0363f290f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed932198-ee76-4ff5-a777-34e0363f290f-maas-api-openshift-ca\") pod \"ed932198-ee76-4ff5-a777-34e0363f290f\" (UID: \"ed932198-ee76-4ff5-a777-34e0363f290f\") "
Oct 30 01:12:37.354397 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:37.354332    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed932198-ee76-4ff5-a777-34e0363f290f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed932198-ee76-4ff5-a777-34e0363f290f-maas-api-xks-ca\") pod \"ed932198-ee76-4ff5-a777-34e0363f290f\" (UID: \"ed932198-ee76-4ff5-a777-34e0363f290f\") "
Oct 30 01:12:37.354397 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:37.354360    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed932198-ee76-4ff5-a777-34e0363f290f-kube-api-access-n95lw\" (UniqueName: \"kubernetes.io/projected/ed932198-ee76-4ff5-a777-34e0363f290f-kube-api-access-n95lw\") pod \"ed932198-ee76-4ff5-a777-34e0363f290f\" (UID: \"ed932198-ee76-4ff5-a777-34e0363f290f\") "
Oct 30 01:12:37.354556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:37.354524    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed932198-ee76-4ff5-a777-34e0363f290f-maas-api-openshift-ca" pod "ed932198-ee76-4ff5-a777-34e0363f290f" (UID: "ed932198-ee76-4ff5-a777-34e0363f290f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:37.354874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:37.354852    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed932198-ee76-4ff5-a777-34e0363f290f-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:37.356772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:37.356740    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed932198-ee76-4ff5-a777-34e0363f290f-kube-api-access-n95lw" pod "ed932198-ee76-4ff5-a777-34e0363f290f" (UID: "ed932198-ee76-4ff5-a777-34e0363f290f"). InnerVolumeSpecName "kube-api-access-n95lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:37.356889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:37.356744    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed932198-ee76-4ff5-a777-34e0363f290f-maas-api-xks-ca" pod "ed932198-ee76-4ff5-a777-34e0363f290f" (UID: "ed932198-ee76-4ff5-a777-34e0363f290f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:37.455849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:37.455803    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed932198-ee76-4ff5-a777-34e0363f290f-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:37.455849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:37.455842    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n95lw\" (UniqueName: \"kubernetes.io/projected/ed932198-ee76-4ff5-a777-34e0363f290f-kube-api-access-n95lw\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:37.557368 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:12:37.557323    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:12:38.123132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:38.123100    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"
Oct 30 01:12:38.123132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:38.123127    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f" event={"ID":"ed932198-ee76-4ff5-a777-34e0363f290f","Type":"ContainerDied","Data":"3b269339c8c1787cd16ff8d9cf07ecd8151d1881181ff51424e53dc6b0038a77"}
Oct 30 01:12:38.123388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:38.123190    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b269339c8c1787cd16ff8d9cf07ecd8151d1881181ff51424e53dc6b0038a77"
Oct 30 01:12:39.383578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.383531    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"]
Oct 30 01:12:39.384107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.384095    4425 cpu_manager.go:414] "containerMap: removing container" podUID="f16cc640-5e15-45f9-87ba-942cf8a21325" containerName="payload-processing"
Oct 30 01:12:39.384190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.384112    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16cc640-5e15-45f9-87ba-942cf8a21325" containerName="payload-processing"
Oct 30 01:12:39.384190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.384141    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9eee6597-2b6b-4760-9144-4901fc0c342b" containerName="payload-pre-processing"
Oct 30 01:12:39.384190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.384146    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eee6597-2b6b-4760-9144-4901fc0c342b" containerName="payload-pre-processing"
Oct 30 01:12:39.384190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.384179    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ed932198-ee76-4ff5-a777-34e0363f290f" containerName="revoke-keys"
Oct 30 01:12:39.384190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.384184    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed932198-ee76-4ff5-a777-34e0363f290f" containerName="revoke-keys"
Oct 30 01:12:39.384372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.384279    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eee6597-2b6b-4760-9144-4901fc0c342b" containerName="payload-pre-processing"
Oct 30 01:12:39.384372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.384291    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed932198-ee76-4ff5-a777-34e0363f290f" containerName="revoke-keys"
Oct 30 01:12:39.384372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.384302    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16cc640-5e15-45f9-87ba-942cf8a21325" containerName="payload-processing"
Oct 30 01:12:39.388110 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.388084    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.428281 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.428243    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"]
Oct 30 01:12:39.428466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.428322    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"]
Oct 30 01:12:39.433738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.433705    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.454923 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.454884    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"]
Oct 30 01:12:39.482679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.482633    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.482999 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.482966    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.483141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.483013    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.483141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.483109    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.483279 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.483234    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg949\" (UniqueName: \"kubernetes.io/projected/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-kube-api-access-sg949\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.483323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.483275    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.483323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.483316    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.483398 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.483343    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qff77\" (UniqueName: \"kubernetes.io/projected/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-kube-api-access-qff77\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.483398 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.483370    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.483474 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.483400    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.483474 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.483451    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.483548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.483487    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.483548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.483522    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.483623 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.483547    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.483623 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.483599    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.483708 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.483688    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.483748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.483739    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.483826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.483810    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.584813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.584770    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.585075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.584828    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.585075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.584882    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.585075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.584944    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.585075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585005    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.585075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585031    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.585075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585068    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.585447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585116    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg949\" (UniqueName: \"kubernetes.io/projected/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-kube-api-access-sg949\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.585447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585189    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.585447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585261    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.585447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585299    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qff77\" (UniqueName: \"kubernetes.io/projected/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-kube-api-access-qff77\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.585447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585334    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.585447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585367    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.585447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585402    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.585447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585405    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.585447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585438    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.586132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585467    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.586132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585493    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.586132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585498    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.586132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585475    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.586132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585559    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.586132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585612    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.586132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585749    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-workload-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.586132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585823    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.586132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.585810    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-workload-certs\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.586710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.586147    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-data\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.586710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.586178    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istiod-ca-cert\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.586710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.586282    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-credential-socket\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.587976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.587945    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.588142 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.588118    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-envoy\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.588286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.588183    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.588414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.588393    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-podinfo\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.605295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.605254    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"]
Oct 30 01:12:39.611128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.611092    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"
Oct 30 01:12:39.612372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.612341    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.612518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.612352    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-token\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.612518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.612408    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg949\" (UniqueName: \"kubernetes.io/projected/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-kube-api-access-sg949\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.612518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.612504    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qff77\" (UniqueName: \"kubernetes.io/projected/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-kube-api-access-qff77\") pod \"e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") " pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.625672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.625635    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:12:39.625870 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.625808    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:12:39.626014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.625987    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:12:39.626477 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.626454    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:12:39.629761 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.629734    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"]
Oct 30 01:12:39.687309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.687204    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30fd198-76d3-4cab-b147-818fb42be97d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d\" (UID: \"c30fd198-76d3-4cab-b147-818fb42be97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"
Oct 30 01:12:39.687309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.687254    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swxt5\" (UniqueName: \"kubernetes.io/projected/c30fd198-76d3-4cab-b147-818fb42be97d-kube-api-access-swxt5\") pod \"maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d\" (UID: \"c30fd198-76d3-4cab-b147-818fb42be97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"
Oct 30 01:12:39.687538 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.687435    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30fd198-76d3-4cab-b147-818fb42be97d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d\" (UID: \"c30fd198-76d3-4cab-b147-818fb42be97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"
Oct 30 01:12:39.703443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.703404    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:39.757509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.757472    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:39.797201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.789076    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30fd198-76d3-4cab-b147-818fb42be97d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d\" (UID: \"c30fd198-76d3-4cab-b147-818fb42be97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"
Oct 30 01:12:39.797201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.789131    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swxt5\" (UniqueName: \"kubernetes.io/projected/c30fd198-76d3-4cab-b147-818fb42be97d-kube-api-access-swxt5\") pod \"maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d\" (UID: \"c30fd198-76d3-4cab-b147-818fb42be97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"
Oct 30 01:12:39.797201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.789210    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30fd198-76d3-4cab-b147-818fb42be97d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d\" (UID: \"c30fd198-76d3-4cab-b147-818fb42be97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"
Oct 30 01:12:39.797201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.789858    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30fd198-76d3-4cab-b147-818fb42be97d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d\" (UID: \"c30fd198-76d3-4cab-b147-818fb42be97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"
Oct 30 01:12:39.797201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.792400    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30fd198-76d3-4cab-b147-818fb42be97d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d\" (UID: \"c30fd198-76d3-4cab-b147-818fb42be97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"
Oct 30 01:12:39.821908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.821867    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swxt5\" (UniqueName: \"kubernetes.io/projected/c30fd198-76d3-4cab-b147-818fb42be97d-kube-api-access-swxt5\") pod \"maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d\" (UID: \"c30fd198-76d3-4cab-b147-818fb42be97d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"
Oct 30 01:12:39.941481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:39.941379    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"
Oct 30 01:12:40.030668 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:40.030622    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"]
Oct 30 01:12:40.037093 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:40.037040    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:12:40.037235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:40.037144    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:12:40.037301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:40.037271    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:12:40.098897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:40.098864    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"]
Oct 30 01:12:40.109110 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:12:40.108412    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1617c4a9_d9b2_41b7_9907_c1d2d8878eb9.slice/crio-c2df390cd36d40c585e02e45a80637afa8c96bece1cf533ab18796bf07db074d WatchSource:0}: Error finding container c2df390cd36d40c585e02e45a80637afa8c96bece1cf533ab18796bf07db074d: Status 404 returned error can't find the container with id c2df390cd36d40c585e02e45a80637afa8c96bece1cf533ab18796bf07db074d
Oct 30 01:12:40.112090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:40.112043    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:12:40.112213 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:40.112137    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:12:40.112213 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:40.112196    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:12:40.158184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:40.158135    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj" event={"ID":"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e","Type":"ContainerStarted","Data":"b41f113eca674e7adb277648bd17a66d1dbbf801c19b0c42d07f7c1e4c220e7f"}
Oct 30 01:12:40.159836 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:40.159807    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8" event={"ID":"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9","Type":"ContainerStarted","Data":"c2df390cd36d40c585e02e45a80637afa8c96bece1cf533ab18796bf07db074d"}
Oct 30 01:12:40.256477 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:40.256442    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"]
Oct 30 01:12:40.258465 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:12:40.258433    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30fd198_76d3_4cab_b147_818fb42be97d.slice/crio-6d5b26a00b0d413a910c24310a118c0905d99ed0b66c64e61e65e6b7b66af1b5 WatchSource:0}: Error finding container 6d5b26a00b0d413a910c24310a118c0905d99ed0b66c64e61e65e6b7b66af1b5: Status 404 returned error can't find the container with id 6d5b26a00b0d413a910c24310a118c0905d99ed0b66c64e61e65e6b7b66af1b5
Oct 30 01:12:41.212291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:41.212243    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d" event={"ID":"c30fd198-76d3-4cab-b147-818fb42be97d","Type":"ContainerStarted","Data":"275f4388e89209e027d8d991653d212a329811e7571a1f57352afe9674b4be8d"}
Oct 30 01:12:41.212840 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:41.212332    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d" event={"ID":"c30fd198-76d3-4cab-b147-818fb42be97d","Type":"ContainerStarted","Data":"6d5b26a00b0d413a910c24310a118c0905d99ed0b66c64e61e65e6b7b66af1b5"}
Oct 30 01:12:41.214501 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:41.214467    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj" event={"ID":"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e","Type":"ContainerStarted","Data":"a0d57118a0ed30f19382259ddd53b57a17d750e2800b387ca728c49775a9c388"}
Oct 30 01:12:41.216597 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:41.216565    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8" event={"ID":"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9","Type":"ContainerStarted","Data":"196947cd3b579255d01288142b6f45ae83f374d29291d6470066b5b575c89741"}
Oct 30 01:12:41.259187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:41.256409    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d" podStartSLOduration=2.256384874 podStartE2EDuration="2.256384874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:41.249119356 +0000 UTC m=+1869.998774665" watchObservedRunningTime="2026-10-30 01:12:41.256384874 +0000 UTC m=+1870.006040184"
Oct 30 01:12:41.311663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:41.311589    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj" podStartSLOduration=2.311568017 podStartE2EDuration="2.311568017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:41.299694153 +0000 UTC m=+1870.049349473" watchObservedRunningTime="2026-10-30 01:12:41.311568017 +0000 UTC m=+1870.061223327"
Oct 30 01:12:41.342733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:41.342671    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8" podStartSLOduration=2.342651848 podStartE2EDuration="2.342651848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:41.340494064 +0000 UTC m=+1870.090149372" watchObservedRunningTime="2026-10-30 01:12:41.342651848 +0000 UTC m=+1870.092307157"
Oct 30 01:12:41.704336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:41.704294    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:41.705944 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:41.705918    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 01:12:41.706032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:41.705971    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj" podUID="c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 01:12:41.758266 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:41.758220    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:41.766180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:41.762485    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:12:41.766180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:41.762558    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8" podUID="1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 01:12:42.236368 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.236324    4425 generic.go:356] "Generic (PLEG): container finished" podID="c30fd198-76d3-4cab-b147-818fb42be97d" containerID="275f4388e89209e027d8d991653d212a329811e7571a1f57352afe9674b4be8d" exitCode=0
Oct 30 01:12:42.237277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.237173    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d" event={"ID":"c30fd198-76d3-4cab-b147-818fb42be97d","Type":"ContainerDied","Data":"275f4388e89209e027d8d991653d212a329811e7571a1f57352afe9674b4be8d"}
Oct 30 01:12:42.440312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.440267    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"]
Oct 30 01:12:42.444716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.444679    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"
Oct 30 01:12:42.512339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.512268    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"]
Oct 30 01:12:42.537297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.537260    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebbb688e-dhj2c\" (UID: \"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"
Oct 30 01:12:42.537490 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.537311    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebbb688e-dhj2c\" (UID: \"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"
Oct 30 01:12:42.537540 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.537488    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9knh\" (UniqueName: \"kubernetes.io/projected/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-kube-api-access-q9knh\") pod \"maas-api-revoke-sub-ebbb688e-dhj2c\" (UID: \"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"
Oct 30 01:12:42.638557 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.638512    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebbb688e-dhj2c\" (UID: \"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"
Oct 30 01:12:42.638750 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.638568    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebbb688e-dhj2c\" (UID: \"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"
Oct 30 01:12:42.638750 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.638686    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9knh\" (UniqueName: \"kubernetes.io/projected/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-kube-api-access-q9knh\") pod \"maas-api-revoke-sub-ebbb688e-dhj2c\" (UID: \"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"
Oct 30 01:12:42.639306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.639282    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebbb688e-dhj2c\" (UID: \"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"
Oct 30 01:12:42.641427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.641400    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebbb688e-dhj2c\" (UID: \"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"
Oct 30 01:12:42.704179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.704079    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 01:12:42.704366 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.704179    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj" podUID="c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 01:12:42.704774 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.704741    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9knh\" (UniqueName: \"kubernetes.io/projected/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-kube-api-access-q9knh\") pod \"maas-api-revoke-sub-ebbb688e-dhj2c\" (UID: \"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"
Oct 30 01:12:42.765256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.762356    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:12:42.765256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.762466    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8" podUID="1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 01:12:42.812770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:42.812731    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"
Oct 30 01:12:43.189239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:43.185314    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"]
Oct 30 01:12:43.205732 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:12:43.205699    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48dd4d4_9bcb_443c_87f0_1f3b68d4849f.slice/crio-a6a66e1e1f3b257b34ef29546920cb452444450af9002dc3da3d233b255ae7f3 WatchSource:0}: Error finding container a6a66e1e1f3b257b34ef29546920cb452444450af9002dc3da3d233b255ae7f3: Status 404 returned error can't find the container with id a6a66e1e1f3b257b34ef29546920cb452444450af9002dc3da3d233b255ae7f3
Oct 30 01:12:43.272039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:43.253864    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c" event={"ID":"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f","Type":"ContainerStarted","Data":"a6a66e1e1f3b257b34ef29546920cb452444450af9002dc3da3d233b255ae7f3"}
Oct 30 01:12:43.515786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:43.515755    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"
Oct 30 01:12:43.613804 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:43.613768    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c30fd198-76d3-4cab-b147-818fb42be97d-kube-api-access-swxt5\" (UniqueName: \"kubernetes.io/projected/c30fd198-76d3-4cab-b147-818fb42be97d-kube-api-access-swxt5\") pod \"c30fd198-76d3-4cab-b147-818fb42be97d\" (UID: \"c30fd198-76d3-4cab-b147-818fb42be97d\") "
Oct 30 01:12:43.614032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:43.613948    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c30fd198-76d3-4cab-b147-818fb42be97d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30fd198-76d3-4cab-b147-818fb42be97d-maas-api-xks-ca\") pod \"c30fd198-76d3-4cab-b147-818fb42be97d\" (UID: \"c30fd198-76d3-4cab-b147-818fb42be97d\") "
Oct 30 01:12:43.614095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:43.614062    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c30fd198-76d3-4cab-b147-818fb42be97d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30fd198-76d3-4cab-b147-818fb42be97d-maas-api-openshift-ca\") pod \"c30fd198-76d3-4cab-b147-818fb42be97d\" (UID: \"c30fd198-76d3-4cab-b147-818fb42be97d\") "
Oct 30 01:12:43.614697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:43.614655    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c30fd198-76d3-4cab-b147-818fb42be97d-maas-api-openshift-ca" pod "c30fd198-76d3-4cab-b147-818fb42be97d" (UID: "c30fd198-76d3-4cab-b147-818fb42be97d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:43.616644 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:43.616608    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c30fd198-76d3-4cab-b147-818fb42be97d-maas-api-xks-ca" pod "c30fd198-76d3-4cab-b147-818fb42be97d" (UID: "c30fd198-76d3-4cab-b147-818fb42be97d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:43.616644 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:43.616637    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30fd198-76d3-4cab-b147-818fb42be97d-kube-api-access-swxt5" pod "c30fd198-76d3-4cab-b147-818fb42be97d" (UID: "c30fd198-76d3-4cab-b147-818fb42be97d"). InnerVolumeSpecName "kube-api-access-swxt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:43.705220 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:43.705136    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 01:12:43.705390 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:43.705252    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj" podUID="c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 01:12:43.716232 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:43.716188    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swxt5\" (UniqueName: \"kubernetes.io/projected/c30fd198-76d3-4cab-b147-818fb42be97d-kube-api-access-swxt5\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:43.716232 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:43.716229    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30fd198-76d3-4cab-b147-818fb42be97d-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:43.716460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:43.716244    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30fd198-76d3-4cab-b147-818fb42be97d-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:43.785182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:43.769016    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:12:43.785182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:43.769096    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8" podUID="1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 01:12:44.303207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:44.300387    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d" event={"ID":"c30fd198-76d3-4cab-b147-818fb42be97d","Type":"ContainerDied","Data":"6d5b26a00b0d413a910c24310a118c0905d99ed0b66c64e61e65e6b7b66af1b5"}
Oct 30 01:12:44.303207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:44.300500    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d5b26a00b0d413a910c24310a118c0905d99ed0b66c64e61e65e6b7b66af1b5"
Oct 30 01:12:44.303207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:44.300576    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"
Oct 30 01:12:44.307940 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:44.307898    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c" event={"ID":"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f","Type":"ContainerStarted","Data":"c73443d582cfb251622783aca1bf815505dfd0f12ce2b503a9d78b852432da22"}
Oct 30 01:12:44.347210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:44.347105    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c" podStartSLOduration=2.347088045 podStartE2EDuration="2.347088045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:44.343277991 +0000 UTC m=+1873.092933300" watchObservedRunningTime="2026-10-30 01:12:44.347088045 +0000 UTC m=+1873.096743374"
Oct 30 01:12:44.706183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:44.703979    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 01:12:44.706183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:44.704054    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj" podUID="c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 01:12:44.778183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:44.772470    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:12:44.785332 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:44.785279    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8" podUID="1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 01:12:45.319379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:45.319092    4425 generic.go:356] "Generic (PLEG): container finished" podID="b48dd4d4-9bcb-443c-87f0-1f3b68d4849f" containerID="c73443d582cfb251622783aca1bf815505dfd0f12ce2b503a9d78b852432da22" exitCode=0
Oct 30 01:12:45.319379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:45.319185    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c" event={"ID":"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f","Type":"ContainerDied","Data":"c73443d582cfb251622783aca1bf815505dfd0f12ce2b503a9d78b852432da22"}
Oct 30 01:12:45.705243 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:45.704822    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 01:12:45.705243 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:45.704902    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj" podUID="c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 01:12:45.765183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:45.762031    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:12:45.765183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:45.762096    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8" podUID="1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 01:12:46.645966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:46.645917    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"
Oct 30 01:12:46.708218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:46.705622    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 01:12:46.708218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:46.705688    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj" podUID="c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 01:12:46.708988 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:46.708959    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-maas-api-openshift-ca\") pod \"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f\" (UID: \"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f\") "
Oct 30 01:12:46.709136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:46.709119    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-maas-api-xks-ca\") pod \"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f\" (UID: \"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f\") "
Oct 30 01:12:46.709202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:46.709170    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-kube-api-access-q9knh\" (UniqueName: \"kubernetes.io/projected/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-kube-api-access-q9knh\") pod \"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f\" (UID: \"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f\") "
Oct 30 01:12:46.709841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:46.709813    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-maas-api-openshift-ca" pod "b48dd4d4-9bcb-443c-87f0-1f3b68d4849f" (UID: "b48dd4d4-9bcb-443c-87f0-1f3b68d4849f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:46.712409 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:46.712370    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-kube-api-access-q9knh" pod "b48dd4d4-9bcb-443c-87f0-1f3b68d4849f" (UID: "b48dd4d4-9bcb-443c-87f0-1f3b68d4849f"). InnerVolumeSpecName "kube-api-access-q9knh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:46.712575 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:46.712528    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-maas-api-xks-ca" pod "b48dd4d4-9bcb-443c-87f0-1f3b68d4849f" (UID: "b48dd4d4-9bcb-443c-87f0-1f3b68d4849f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:46.762761 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:46.762098    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 01:12:46.762761 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:46.762211    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8" podUID="1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 01:12:46.774290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:46.774233    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:12:46.812794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:46.812294    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:46.812794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:46.812335    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9knh\" (UniqueName: \"kubernetes.io/projected/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-kube-api-access-q9knh\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:46.812794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:46.812350    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:47.337351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:47.337310    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"
Oct 30 01:12:47.337351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:47.337340    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c" event={"ID":"b48dd4d4-9bcb-443c-87f0-1f3b68d4849f","Type":"ContainerDied","Data":"a6a66e1e1f3b257b34ef29546920cb452444450af9002dc3da3d233b255ae7f3"}
Oct 30 01:12:47.337660 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:47.337396    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6a66e1e1f3b257b34ef29546920cb452444450af9002dc3da3d233b255ae7f3"
Oct 30 01:12:48.727181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:48.727079    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:12:48.727181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:48.727148    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj" podUID="c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:12:48.781183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:48.777280    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:12:48.781183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:48.777357    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8" podUID="1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:12:49.190184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:49.173773    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"]
Oct 30 01:12:49.190184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:49.174680    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt" podUID="336deb2f-912c-4e44-bbf2-356952a1cebf" containerName="payload-processing" containerID="cri-o://7fd88ab3f029e1ea805446ed6ea4c913cae00f6513a6cdb39af6066a0e476137" gracePeriod=30
Oct 30 01:12:49.481558 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:49.456699    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"]
Oct 30 01:12:49.481558 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:49.457075    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk" podUID="4311a891-a923-436f-8f5c-086dbafcaf8c" containerName="payload-pre-processing" containerID="cri-o://24aebdb0b5390ef11efb755f3c8f376838176a57bb198e06fa494af6d5a51f2c" gracePeriod=30
Oct 30 01:12:49.783184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:49.774294    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:12:49.783184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:49.774437    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj" podUID="c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:12:50.029181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.026951    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:12:50.029181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.027060    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8" podUID="1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:12:50.192637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.192598    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw" podUID="4d3d278e-ecd0-46ee-a203-33003899a369" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 30 01:12:50.260185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.252348    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z" podUID="b4ed47ff-0328-4645-be5f-6a6d51a94fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:12:50.260185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.253557    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s" podUID="b8b27e40-2516-4d18-8c84-5e0925b3f4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:12:50.323245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.322266    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2" podUID="13f688db-25e0-4a88-9d1e-c3fd78b7f659" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:12:50.323245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.322729    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc" podUID="c8d1f900-b8c4-43dc-814e-9491e560ea3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:12:50.344290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.344249    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"]
Oct 30 01:12:50.370185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.363794    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f05249-74nnw"]
Oct 30 01:12:50.472108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.471289    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" podUID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:12:50.545187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.542263    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"]
Oct 30 01:12:50.572753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.572709    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"]
Oct 30 01:12:50.709245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.655977    4425 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s" podUID="b8b27e40-2516-4d18-8c84-5e0925b3f4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:12:50.745183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.709917    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:12:50.828212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.826004    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:12:50.828212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.826076    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj" podUID="c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:12:50.869910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.869307    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w" podUID="66a19982-1f16-4a03-a629-bf9b7d6b50d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:12:51.004353 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:50.995571    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:51.004353 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.001637    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:51.044364 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.040911    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:12:51.197183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.196119    4425 generic.go:356] "Generic (PLEG): container finished" podID="336deb2f-912c-4e44-bbf2-356952a1cebf" containerID="7fd88ab3f029e1ea805446ed6ea4c913cae00f6513a6cdb39af6066a0e476137" exitCode=0
Oct 30 01:12:51.197183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.196439    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt" event={"ID":"336deb2f-912c-4e44-bbf2-356952a1cebf","Type":"ContainerDied","Data":"7fd88ab3f029e1ea805446ed6ea4c913cae00f6513a6cdb39af6066a0e476137"}
Oct 30 01:12:51.218431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.200233    4425 generic.go:356] "Generic (PLEG): container finished" podID="4311a891-a923-436f-8f5c-086dbafcaf8c" containerID="24aebdb0b5390ef11efb755f3c8f376838176a57bb198e06fa494af6d5a51f2c" exitCode=0
Oct 30 01:12:51.218431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.203052    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk" event={"ID":"4311a891-a923-436f-8f5c-086dbafcaf8c","Type":"ContainerDied","Data":"24aebdb0b5390ef11efb755f3c8f376838176a57bb198e06fa494af6d5a51f2c"}
Oct 30 01:12:51.249196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.247222    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:51.249196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.248558    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:51.252201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.251891    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:12:51.269212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.266884    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s" podUID="b8b27e40-2516-4d18-8c84-5e0925b3f4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:12:51.269212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.267274    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z" podUID="b4ed47ff-0328-4645-be5f-6a6d51a94fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:12:51.269212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.267830    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 01:12:51.269212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.267926    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 01:12:51.334137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.334107    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 01:12:51.341023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.340713    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 01:12:51.367907 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.367868    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"
Oct 30 01:12:51.372481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.372416    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"
Oct 30 01:12:51.428297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.428257    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db5e01ab-e8e6-4c06-a3cd-f3d112b3900a" path="/var/lib/kubelet/pods/db5e01ab-e8e6-4c06-a3cd-f3d112b3900a/volumes"
Oct 30 01:12:51.497281 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.497187    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336deb2f-912c-4e44-bbf2-356952a1cebf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336deb2f-912c-4e44-bbf2-356952a1cebf-plugins-config-volume\") pod \"336deb2f-912c-4e44-bbf2-356952a1cebf\" (UID: \"336deb2f-912c-4e44-bbf2-356952a1cebf\") "
Oct 30 01:12:51.497438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.497365    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4311a891-a923-436f-8f5c-086dbafcaf8c-kube-api-access-x6zcc\" (UniqueName: \"kubernetes.io/projected/4311a891-a923-436f-8f5c-086dbafcaf8c-kube-api-access-x6zcc\") pod \"4311a891-a923-436f-8f5c-086dbafcaf8c\" (UID: \"4311a891-a923-436f-8f5c-086dbafcaf8c\") "
Oct 30 01:12:51.497438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.497401    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4311a891-a923-436f-8f5c-086dbafcaf8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4311a891-a923-436f-8f5c-086dbafcaf8c-plugins-config-volume\") pod \"4311a891-a923-436f-8f5c-086dbafcaf8c\" (UID: \"4311a891-a923-436f-8f5c-086dbafcaf8c\") "
Oct 30 01:12:51.497569 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.497446    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336deb2f-912c-4e44-bbf2-356952a1cebf-kube-api-access-w4f27\" (UniqueName: \"kubernetes.io/projected/336deb2f-912c-4e44-bbf2-356952a1cebf-kube-api-access-w4f27\") pod \"336deb2f-912c-4e44-bbf2-356952a1cebf\" (UID: \"336deb2f-912c-4e44-bbf2-356952a1cebf\") "
Oct 30 01:12:51.497883 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.497842    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336deb2f-912c-4e44-bbf2-356952a1cebf-plugins-config-volume" pod "336deb2f-912c-4e44-bbf2-356952a1cebf" (UID: "336deb2f-912c-4e44-bbf2-356952a1cebf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:51.497883 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.497875    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4311a891-a923-436f-8f5c-086dbafcaf8c-plugins-config-volume" pod "4311a891-a923-436f-8f5c-086dbafcaf8c" (UID: "4311a891-a923-436f-8f5c-086dbafcaf8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:51.500032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.500001    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4311a891-a923-436f-8f5c-086dbafcaf8c-kube-api-access-x6zcc" pod "4311a891-a923-436f-8f5c-086dbafcaf8c" (UID: "4311a891-a923-436f-8f5c-086dbafcaf8c"). InnerVolumeSpecName "kube-api-access-x6zcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:51.500176 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.500148    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336deb2f-912c-4e44-bbf2-356952a1cebf-kube-api-access-w4f27" pod "336deb2f-912c-4e44-bbf2-356952a1cebf" (UID: "336deb2f-912c-4e44-bbf2-356952a1cebf"). InnerVolumeSpecName "kube-api-access-w4f27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:51.599810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.599771    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6zcc\" (UniqueName: \"kubernetes.io/projected/4311a891-a923-436f-8f5c-086dbafcaf8c-kube-api-access-x6zcc\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:51.600009 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.599988    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4311a891-a923-436f-8f5c-086dbafcaf8c-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:51.600106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.600012    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4f27\" (UniqueName: \"kubernetes.io/projected/336deb2f-912c-4e44-bbf2-356952a1cebf-kube-api-access-w4f27\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:51.600106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:51.600025    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336deb2f-912c-4e44-bbf2-356952a1cebf-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:52.119799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.119751    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"]
Oct 30 01:12:52.120424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.120407    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c30fd198-76d3-4cab-b147-818fb42be97d" containerName="revoke-keys"
Oct 30 01:12:52.120490 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.120425    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c30fd198-76d3-4cab-b147-818fb42be97d" containerName="revoke-keys"
Oct 30 01:12:52.120490 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.120443    4425 cpu_manager.go:414] "containerMap: removing container" podUID="4311a891-a923-436f-8f5c-086dbafcaf8c" containerName="payload-pre-processing"
Oct 30 01:12:52.120490 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.120449    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4311a891-a923-436f-8f5c-086dbafcaf8c" containerName="payload-pre-processing"
Oct 30 01:12:52.120490 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.120474    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b48dd4d4-9bcb-443c-87f0-1f3b68d4849f" containerName="revoke-keys"
Oct 30 01:12:52.120490 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.120482    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48dd4d4-9bcb-443c-87f0-1f3b68d4849f" containerName="revoke-keys"
Oct 30 01:12:52.120645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.120509    4425 cpu_manager.go:414] "containerMap: removing container" podUID="336deb2f-912c-4e44-bbf2-356952a1cebf" containerName="payload-processing"
Oct 30 01:12:52.120645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.120513    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336deb2f-912c-4e44-bbf2-356952a1cebf" containerName="payload-processing"
Oct 30 01:12:52.120645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.120602    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c30fd198-76d3-4cab-b147-818fb42be97d" containerName="revoke-keys"
Oct 30 01:12:52.120645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.120614    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4311a891-a923-436f-8f5c-086dbafcaf8c" containerName="payload-pre-processing"
Oct 30 01:12:52.120645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.120622    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48dd4d4-9bcb-443c-87f0-1f3b68d4849f" containerName="revoke-keys"
Oct 30 01:12:52.120645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.120637    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="336deb2f-912c-4e44-bbf2-356952a1cebf" containerName="payload-processing"
Oct 30 01:12:52.125502 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.125477    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"
Oct 30 01:12:52.144497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.144463    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:12:52.144699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.144593    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:12:52.146226 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.146151    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:12:52.151330 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.151294    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"]
Oct 30 01:12:52.155067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.155022    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:12:52.207852 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.207808    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf\" (UID: \"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"
Oct 30 01:12:52.208025 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.207891    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb78q\" (UniqueName: \"kubernetes.io/projected/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-kube-api-access-hb78q\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf\" (UID: \"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"
Oct 30 01:12:52.208025 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.207960    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf\" (UID: \"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"
Oct 30 01:12:52.210146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.210117    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"
Oct 30 01:12:52.210146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.210134    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt" event={"ID":"336deb2f-912c-4e44-bbf2-356952a1cebf","Type":"ContainerDied","Data":"3719fb4f09ee284fada7823948f359f783885c1488020f0d6d40143c18715422"}
Oct 30 01:12:52.210363 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.210220    4425 scope.go:122] "RemoveContainer" containerID="7fd88ab3f029e1ea805446ed6ea4c913cae00f6513a6cdb39af6066a0e476137"
Oct 30 01:12:52.212258 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.212225    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk" event={"ID":"4311a891-a923-436f-8f5c-086dbafcaf8c","Type":"ContainerDied","Data":"d95206484846bd351920aa5ec24d865b5003e75256d1a79a2e0802535ffa81d9"}
Oct 30 01:12:52.212397 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.212278    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"
Oct 30 01:12:52.228963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.228938    4425 scope.go:122] "RemoveContainer" containerID="24aebdb0b5390ef11efb755f3c8f376838176a57bb198e06fa494af6d5a51f2c"
Oct 30 01:12:52.271534 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.271502    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"]
Oct 30 01:12:52.276346 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.276312    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f05249-7784bd7958-9bswt"]
Oct 30 01:12:52.305082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.305038    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"]
Oct 30 01:12:52.309184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.309131    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf\" (UID: \"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"
Oct 30 01:12:52.309432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.309324    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb78q\" (UniqueName: \"kubernetes.io/projected/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-kube-api-access-hb78q\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf\" (UID: \"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"
Oct 30 01:12:52.309432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.309392    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf\" (UID: \"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"
Oct 30 01:12:52.309825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.309794    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f05249-6dc96b868c-l4gfk"]
Oct 30 01:12:52.310138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.310114    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf\" (UID: \"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"
Oct 30 01:12:52.312179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.312139    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf\" (UID: \"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"
Oct 30 01:12:52.330510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.330477    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb78q\" (UniqueName: \"kubernetes.io/projected/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-kube-api-access-hb78q\") pod \"maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf\" (UID: \"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"
Oct 30 01:12:52.454299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.454144    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"
Oct 30 01:12:52.563844 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.563694    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"]
Oct 30 01:12:52.569127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.569100    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"
Oct 30 01:12:52.589149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.589102    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"]
Oct 30 01:12:52.613509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.613473    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77aca47d-c426-4970-a1f3-44a22295b2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10fc7c87-q5vxc\" (UID: \"77aca47d-c426-4970-a1f3-44a22295b2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"
Oct 30 01:12:52.613713 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.613542    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hst95\" (UniqueName: \"kubernetes.io/projected/77aca47d-c426-4970-a1f3-44a22295b2dc-kube-api-access-hst95\") pod \"maas-api-revoke-sub-10fc7c87-q5vxc\" (UID: \"77aca47d-c426-4970-a1f3-44a22295b2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"
Oct 30 01:12:52.613713 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.613670    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77aca47d-c426-4970-a1f3-44a22295b2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10fc7c87-q5vxc\" (UID: \"77aca47d-c426-4970-a1f3-44a22295b2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"
Oct 30 01:12:52.715076 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.714987    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77aca47d-c426-4970-a1f3-44a22295b2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10fc7c87-q5vxc\" (UID: \"77aca47d-c426-4970-a1f3-44a22295b2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"
Oct 30 01:12:52.715261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.715085    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hst95\" (UniqueName: \"kubernetes.io/projected/77aca47d-c426-4970-a1f3-44a22295b2dc-kube-api-access-hst95\") pod \"maas-api-revoke-sub-10fc7c87-q5vxc\" (UID: \"77aca47d-c426-4970-a1f3-44a22295b2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"
Oct 30 01:12:52.715261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.715148    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77aca47d-c426-4970-a1f3-44a22295b2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10fc7c87-q5vxc\" (UID: \"77aca47d-c426-4970-a1f3-44a22295b2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"
Oct 30 01:12:52.715911 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.715885    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77aca47d-c426-4970-a1f3-44a22295b2dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10fc7c87-q5vxc\" (UID: \"77aca47d-c426-4970-a1f3-44a22295b2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"
Oct 30 01:12:52.717833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.717794    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77aca47d-c426-4970-a1f3-44a22295b2dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10fc7c87-q5vxc\" (UID: \"77aca47d-c426-4970-a1f3-44a22295b2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"
Oct 30 01:12:52.738778 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.738739    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hst95\" (UniqueName: \"kubernetes.io/projected/77aca47d-c426-4970-a1f3-44a22295b2dc-kube-api-access-hst95\") pod \"maas-api-revoke-sub-10fc7c87-q5vxc\" (UID: \"77aca47d-c426-4970-a1f3-44a22295b2dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"
Oct 30 01:12:52.763749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.763712    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"]
Oct 30 01:12:52.765314 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:12:52.765279    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be4a7e8_85ef_4ee1_8dd9_4fa4c73434b4.slice/crio-48efe556a3d9da453014ee6dbedff6da22e352240fa027f73c68a2dd892fe91b WatchSource:0}: Error finding container 48efe556a3d9da453014ee6dbedff6da22e352240fa027f73c68a2dd892fe91b: Status 404 returned error can't find the container with id 48efe556a3d9da453014ee6dbedff6da22e352240fa027f73c68a2dd892fe91b
Oct 30 01:12:52.885487 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:52.885437    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"
Oct 30 01:12:53.193474 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:53.193447    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"]
Oct 30 01:12:53.197086 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:12:53.197039    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77aca47d_c426_4970_a1f3_44a22295b2dc.slice/crio-2a05ef4c7541dbcf7e55ce631857079c8e353cb577aab1e895e5d6155451afbd WatchSource:0}: Error finding container 2a05ef4c7541dbcf7e55ce631857079c8e353cb577aab1e895e5d6155451afbd: Status 404 returned error can't find the container with id 2a05ef4c7541dbcf7e55ce631857079c8e353cb577aab1e895e5d6155451afbd
Oct 30 01:12:53.223006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:53.222965    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc" event={"ID":"77aca47d-c426-4970-a1f3-44a22295b2dc","Type":"ContainerStarted","Data":"2a05ef4c7541dbcf7e55ce631857079c8e353cb577aab1e895e5d6155451afbd"}
Oct 30 01:12:53.228354 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:53.228322    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf" event={"ID":"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4","Type":"ContainerStarted","Data":"55dcbb5545602710e226600f64c16c98ab730b524c60641e5a408fd6db858438"}
Oct 30 01:12:53.228522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:53.228382    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf" event={"ID":"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4","Type":"ContainerStarted","Data":"48efe556a3d9da453014ee6dbedff6da22e352240fa027f73c68a2dd892fe91b"}
Oct 30 01:12:53.274751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:53.274682    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf" podStartSLOduration=1.2746614840000001 podStartE2EDuration="1.274661484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:53.271572666 +0000 UTC m=+1882.021227975" watchObservedRunningTime="2026-10-30 01:12:53.274661484 +0000 UTC m=+1882.024316793"
Oct 30 01:12:53.425751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:53.425654    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336deb2f-912c-4e44-bbf2-356952a1cebf" path="/var/lib/kubelet/pods/336deb2f-912c-4e44-bbf2-356952a1cebf/volumes"
Oct 30 01:12:53.426191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:53.426129    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4311a891-a923-436f-8f5c-086dbafcaf8c" path="/var/lib/kubelet/pods/4311a891-a923-436f-8f5c-086dbafcaf8c/volumes"
Oct 30 01:12:54.240783 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:54.240738    4425 generic.go:356] "Generic (PLEG): container finished" podID="77aca47d-c426-4970-a1f3-44a22295b2dc" containerID="5a457810aea6d92410308ec71e7a8a87533a1dd6a4c0bf12e439a177fb4691c6" exitCode=0
Oct 30 01:12:54.241367 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:54.240840    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc" event={"ID":"77aca47d-c426-4970-a1f3-44a22295b2dc","Type":"ContainerDied","Data":"5a457810aea6d92410308ec71e7a8a87533a1dd6a4c0bf12e439a177fb4691c6"}
Oct 30 01:12:54.243382 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:54.243345    4425 generic.go:356] "Generic (PLEG): container finished" podID="8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4" containerID="55dcbb5545602710e226600f64c16c98ab730b524c60641e5a408fd6db858438" exitCode=0
Oct 30 01:12:54.243580 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:54.243423    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf" event={"ID":"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4","Type":"ContainerDied","Data":"55dcbb5545602710e226600f64c16c98ab730b524c60641e5a408fd6db858438"}
Oct 30 01:12:55.424472 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.424436    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"
Oct 30 01:12:55.452534 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.452482    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-maas-api-openshift-ca\") pod \"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4\" (UID: \"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4\") "
Oct 30 01:12:55.452746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.452588    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-kube-api-access-hb78q\" (UniqueName: \"kubernetes.io/projected/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-kube-api-access-hb78q\") pod \"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4\" (UID: \"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4\") "
Oct 30 01:12:55.452816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.452770    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-maas-api-xks-ca\") pod \"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4\" (UID: \"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4\") "
Oct 30 01:12:55.452899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.452870    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-maas-api-openshift-ca" pod "8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4" (UID: "8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:55.453285 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.453265    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:55.454986 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.454948    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-kube-api-access-hb78q" pod "8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4" (UID: "8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4"). InnerVolumeSpecName "kube-api-access-hb78q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:55.454986 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.454965    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-maas-api-xks-ca" pod "8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4" (UID: "8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:55.460042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.460010    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"
Oct 30 01:12:55.542837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.542765    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh" podUID="c820790b-21c4-44d7-b2e8-7a7d1c6af82d" containerName="payload-processing" containerID="cri-o://64b380544adaa80c632cf47b6661ea731aadf4d23d4d7d6b5bc091dbc6b16f4d" gracePeriod=25
Oct 30 01:12:55.554757 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.554674    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77aca47d-c426-4970-a1f3-44a22295b2dc-kube-api-access-hst95\" (UniqueName: \"kubernetes.io/projected/77aca47d-c426-4970-a1f3-44a22295b2dc-kube-api-access-hst95\") pod \"77aca47d-c426-4970-a1f3-44a22295b2dc\" (UID: \"77aca47d-c426-4970-a1f3-44a22295b2dc\") "
Oct 30 01:12:55.554757 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.554729    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77aca47d-c426-4970-a1f3-44a22295b2dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77aca47d-c426-4970-a1f3-44a22295b2dc-maas-api-openshift-ca\") pod \"77aca47d-c426-4970-a1f3-44a22295b2dc\" (UID: \"77aca47d-c426-4970-a1f3-44a22295b2dc\") "
Oct 30 01:12:55.554908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.554884    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77aca47d-c426-4970-a1f3-44a22295b2dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77aca47d-c426-4970-a1f3-44a22295b2dc-maas-api-xks-ca\") pod \"77aca47d-c426-4970-a1f3-44a22295b2dc\" (UID: \"77aca47d-c426-4970-a1f3-44a22295b2dc\") "
Oct 30 01:12:55.555183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.555126    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77aca47d-c426-4970-a1f3-44a22295b2dc-maas-api-openshift-ca" pod "77aca47d-c426-4970-a1f3-44a22295b2dc" (UID: "77aca47d-c426-4970-a1f3-44a22295b2dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:55.555523 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.555499    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb78q\" (UniqueName: \"kubernetes.io/projected/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-kube-api-access-hb78q\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:55.555523 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.555522    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:55.555665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.555531    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77aca47d-c426-4970-a1f3-44a22295b2dc-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:55.557290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.557264    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77aca47d-c426-4970-a1f3-44a22295b2dc-maas-api-xks-ca" pod "77aca47d-c426-4970-a1f3-44a22295b2dc" (UID: "77aca47d-c426-4970-a1f3-44a22295b2dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:55.557358 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.557287    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77aca47d-c426-4970-a1f3-44a22295b2dc-kube-api-access-hst95" pod "77aca47d-c426-4970-a1f3-44a22295b2dc" (UID: "77aca47d-c426-4970-a1f3-44a22295b2dc"). InnerVolumeSpecName "kube-api-access-hst95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:55.574205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.574129    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d" podUID="888c42f2-9961-49e5-a8e8-30af506bfe25" containerName="payload-pre-processing" containerID="cri-o://12f6ce8987215b3f25a6d43284c7f715630a1de690c99abde477f85ecd7d1699" gracePeriod=25
Oct 30 01:12:55.657544 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.657504    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77aca47d-c426-4970-a1f3-44a22295b2dc-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:55.657544 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.657546    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hst95\" (UniqueName: \"kubernetes.io/projected/77aca47d-c426-4970-a1f3-44a22295b2dc-kube-api-access-hst95\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:55.841908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.841855    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"
Oct 30 01:12:55.900333 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.900297    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"
Oct 30 01:12:55.962611 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.962572    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c820790b-21c4-44d7-b2e8-7a7d1c6af82d-kube-api-access-wzpkv\" (UniqueName: \"kubernetes.io/projected/c820790b-21c4-44d7-b2e8-7a7d1c6af82d-kube-api-access-wzpkv\") pod \"c820790b-21c4-44d7-b2e8-7a7d1c6af82d\" (UID: \"c820790b-21c4-44d7-b2e8-7a7d1c6af82d\") "
Oct 30 01:12:55.962833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.962783    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/888c42f2-9961-49e5-a8e8-30af506bfe25-kube-api-access-v6ptf\" (UniqueName: \"kubernetes.io/projected/888c42f2-9961-49e5-a8e8-30af506bfe25-kube-api-access-v6ptf\") pod \"888c42f2-9961-49e5-a8e8-30af506bfe25\" (UID: \"888c42f2-9961-49e5-a8e8-30af506bfe25\") "
Oct 30 01:12:55.962896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.962842    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c820790b-21c4-44d7-b2e8-7a7d1c6af82d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c820790b-21c4-44d7-b2e8-7a7d1c6af82d-plugins-config-volume\") pod \"c820790b-21c4-44d7-b2e8-7a7d1c6af82d\" (UID: \"c820790b-21c4-44d7-b2e8-7a7d1c6af82d\") "
Oct 30 01:12:55.962896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.962862    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/888c42f2-9961-49e5-a8e8-30af506bfe25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888c42f2-9961-49e5-a8e8-30af506bfe25-plugins-config-volume\") pod \"888c42f2-9961-49e5-a8e8-30af506bfe25\" (UID: \"888c42f2-9961-49e5-a8e8-30af506bfe25\") "
Oct 30 01:12:55.963419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.963385    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/888c42f2-9961-49e5-a8e8-30af506bfe25-plugins-config-volume" pod "888c42f2-9961-49e5-a8e8-30af506bfe25" (UID: "888c42f2-9961-49e5-a8e8-30af506bfe25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:55.963698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.963666    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c820790b-21c4-44d7-b2e8-7a7d1c6af82d-plugins-config-volume" pod "c820790b-21c4-44d7-b2e8-7a7d1c6af82d" (UID: "c820790b-21c4-44d7-b2e8-7a7d1c6af82d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:12:55.965759 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.965723    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888c42f2-9961-49e5-a8e8-30af506bfe25-kube-api-access-v6ptf" pod "888c42f2-9961-49e5-a8e8-30af506bfe25" (UID: "888c42f2-9961-49e5-a8e8-30af506bfe25"). InnerVolumeSpecName "kube-api-access-v6ptf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:55.965872 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:55.965843    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c820790b-21c4-44d7-b2e8-7a7d1c6af82d-kube-api-access-wzpkv" pod "c820790b-21c4-44d7-b2e8-7a7d1c6af82d" (UID: "c820790b-21c4-44d7-b2e8-7a7d1c6af82d"). InnerVolumeSpecName "kube-api-access-wzpkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:56.064735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.064692    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzpkv\" (UniqueName: \"kubernetes.io/projected/c820790b-21c4-44d7-b2e8-7a7d1c6af82d-kube-api-access-wzpkv\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:56.064735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.064733    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6ptf\" (UniqueName: \"kubernetes.io/projected/888c42f2-9961-49e5-a8e8-30af506bfe25-kube-api-access-v6ptf\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:56.065000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.064750    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c820790b-21c4-44d7-b2e8-7a7d1c6af82d-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:56.065000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.064766    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/888c42f2-9961-49e5-a8e8-30af506bfe25-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:12:56.262848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.262804    4425 generic.go:356] "Generic (PLEG): container finished" podID="888c42f2-9961-49e5-a8e8-30af506bfe25" containerID="12f6ce8987215b3f25a6d43284c7f715630a1de690c99abde477f85ecd7d1699" exitCode=0
Oct 30 01:12:56.263052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.262959    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d" event={"ID":"888c42f2-9961-49e5-a8e8-30af506bfe25","Type":"ContainerDied","Data":"12f6ce8987215b3f25a6d43284c7f715630a1de690c99abde477f85ecd7d1699"}
Oct 30 01:12:56.263052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.263015    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d" event={"ID":"888c42f2-9961-49e5-a8e8-30af506bfe25","Type":"ContainerDied","Data":"6defe6d49cfb80a88b059b283ee84d1b6d0b142e69fb18f1b553cafb2627bfa4"}
Oct 30 01:12:56.263052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.263046    4425 scope.go:122] "RemoveContainer" containerID="12f6ce8987215b3f25a6d43284c7f715630a1de690c99abde477f85ecd7d1699"
Oct 30 01:12:56.263289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.263219    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"
Oct 30 01:12:56.275478 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.275446    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc" event={"ID":"77aca47d-c426-4970-a1f3-44a22295b2dc","Type":"ContainerDied","Data":"2a05ef4c7541dbcf7e55ce631857079c8e353cb577aab1e895e5d6155451afbd"}
Oct 30 01:12:56.275652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.275501    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a05ef4c7541dbcf7e55ce631857079c8e353cb577aab1e895e5d6155451afbd"
Oct 30 01:12:56.275652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.275573    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"
Oct 30 01:12:56.280097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.280061    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"
Oct 30 01:12:56.288351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.288273    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf" event={"ID":"8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4","Type":"ContainerDied","Data":"48efe556a3d9da453014ee6dbedff6da22e352240fa027f73c68a2dd892fe91b"}
Oct 30 01:12:56.288488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.288424    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48efe556a3d9da453014ee6dbedff6da22e352240fa027f73c68a2dd892fe91b"
Oct 30 01:12:56.288610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.288593    4425 scope.go:122] "RemoveContainer" containerID="12f6ce8987215b3f25a6d43284c7f715630a1de690c99abde477f85ecd7d1699"
Oct 30 01:12:56.292476 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:12:56.292386    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f6ce8987215b3f25a6d43284c7f715630a1de690c99abde477f85ecd7d1699\": container with ID starting with 12f6ce8987215b3f25a6d43284c7f715630a1de690c99abde477f85ecd7d1699 not found: ID does not exist" containerID="12f6ce8987215b3f25a6d43284c7f715630a1de690c99abde477f85ecd7d1699"
Oct 30 01:12:56.292687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.292658    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f6ce8987215b3f25a6d43284c7f715630a1de690c99abde477f85ecd7d1699"} err="failed to get container status \"12f6ce8987215b3f25a6d43284c7f715630a1de690c99abde477f85ecd7d1699\": rpc error: code = NotFound desc = could not find container \"12f6ce8987215b3f25a6d43284c7f715630a1de690c99abde477f85ecd7d1699\": container with ID starting with 12f6ce8987215b3f25a6d43284c7f715630a1de690c99abde477f85ecd7d1699 not found: ID does not exist"
Oct 30 01:12:56.298275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.298232    4425 generic.go:356] "Generic (PLEG): container finished" podID="c820790b-21c4-44d7-b2e8-7a7d1c6af82d" containerID="64b380544adaa80c632cf47b6661ea731aadf4d23d4d7d6b5bc091dbc6b16f4d" exitCode=0
Oct 30 01:12:56.298477 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.298366    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh" event={"ID":"c820790b-21c4-44d7-b2e8-7a7d1c6af82d","Type":"ContainerDied","Data":"64b380544adaa80c632cf47b6661ea731aadf4d23d4d7d6b5bc091dbc6b16f4d"}
Oct 30 01:12:56.298477 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.298424    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh" event={"ID":"c820790b-21c4-44d7-b2e8-7a7d1c6af82d","Type":"ContainerDied","Data":"436ed0fa3aa6458dbcf6540b5ae59d8e9f1c5571f0b98205e8c06b05db36f28a"}
Oct 30 01:12:56.298477 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.298467    4425 scope.go:122] "RemoveContainer" containerID="64b380544adaa80c632cf47b6661ea731aadf4d23d4d7d6b5bc091dbc6b16f4d"
Oct 30 01:12:56.298678 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.298660    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"
Oct 30 01:12:56.315116 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.315091    4425 scope.go:122] "RemoveContainer" containerID="64b380544adaa80c632cf47b6661ea731aadf4d23d4d7d6b5bc091dbc6b16f4d"
Oct 30 01:12:56.315551 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:12:56.315528    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b380544adaa80c632cf47b6661ea731aadf4d23d4d7d6b5bc091dbc6b16f4d\": container with ID starting with 64b380544adaa80c632cf47b6661ea731aadf4d23d4d7d6b5bc091dbc6b16f4d not found: ID does not exist" containerID="64b380544adaa80c632cf47b6661ea731aadf4d23d4d7d6b5bc091dbc6b16f4d"
Oct 30 01:12:56.315639 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.315562    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b380544adaa80c632cf47b6661ea731aadf4d23d4d7d6b5bc091dbc6b16f4d"} err="failed to get container status \"64b380544adaa80c632cf47b6661ea731aadf4d23d4d7d6b5bc091dbc6b16f4d\": rpc error: code = NotFound desc = could not find container \"64b380544adaa80c632cf47b6661ea731aadf4d23d4d7d6b5bc091dbc6b16f4d\": container with ID starting with 64b380544adaa80c632cf47b6661ea731aadf4d23d4d7d6b5bc091dbc6b16f4d not found: ID does not exist"
Oct 30 01:12:56.326710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.326649    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"]
Oct 30 01:12:56.332709 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.332668    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1d6d47-5b7995c6c9-lr67d"]
Oct 30 01:12:56.360444 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.360392    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"]
Oct 30 01:12:56.363654 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.363625    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1d6d47-6d9786d5d-bh8xh"]
Oct 30 01:12:56.734618 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.734527    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"]
Oct 30 01:12:56.737935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:56.737898    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1d6d47-dqm6d"]
Oct 30 01:12:57.423625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:57.423592    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="888c42f2-9961-49e5-a8e8-30af506bfe25" path="/var/lib/kubelet/pods/888c42f2-9961-49e5-a8e8-30af506bfe25/volumes"
Oct 30 01:12:57.424104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:57.424090    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c30fd198-76d3-4cab-b147-818fb42be97d" path="/var/lib/kubelet/pods/c30fd198-76d3-4cab-b147-818fb42be97d/volumes"
Oct 30 01:12:57.424503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:12:57.424489    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c820790b-21c4-44d7-b2e8-7a7d1c6af82d" path="/var/lib/kubelet/pods/c820790b-21c4-44d7-b2e8-7a7d1c6af82d/volumes"
Oct 30 01:13:00.039424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.039380    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"]
Oct 30 01:13:00.039974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.039958    4425 cpu_manager.go:414] "containerMap: removing container" podUID="77aca47d-c426-4970-a1f3-44a22295b2dc" containerName="revoke-keys"
Oct 30 01:13:00.040018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.039975    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77aca47d-c426-4970-a1f3-44a22295b2dc" containerName="revoke-keys"
Oct 30 01:13:00.040018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.039998    4425 cpu_manager.go:414] "containerMap: removing container" podUID="888c42f2-9961-49e5-a8e8-30af506bfe25" containerName="payload-pre-processing"
Oct 30 01:13:00.040018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.040003    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="888c42f2-9961-49e5-a8e8-30af506bfe25" containerName="payload-pre-processing"
Oct 30 01:13:00.040113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.040034    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4" containerName="revoke-keys"
Oct 30 01:13:00.040113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.040039    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4" containerName="revoke-keys"
Oct 30 01:13:00.040113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.040050    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c820790b-21c4-44d7-b2e8-7a7d1c6af82d" containerName="payload-processing"
Oct 30 01:13:00.040113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.040055    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c820790b-21c4-44d7-b2e8-7a7d1c6af82d" containerName="payload-processing"
Oct 30 01:13:00.040274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.040133    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4" containerName="revoke-keys"
Oct 30 01:13:00.040274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.040143    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c820790b-21c4-44d7-b2e8-7a7d1c6af82d" containerName="payload-processing"
Oct 30 01:13:00.040274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.040150    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="77aca47d-c426-4970-a1f3-44a22295b2dc" containerName="revoke-keys"
Oct 30 01:13:00.040274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.040173    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="888c42f2-9961-49e5-a8e8-30af506bfe25" containerName="payload-pre-processing"
Oct 30 01:13:00.044912 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.044889    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"
Oct 30 01:13:00.070127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.070089    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:13:00.070127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.070094    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:13:00.070392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.070087    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:13:00.070392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.070194    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:13:00.074806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.074777    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"]
Oct 30 01:13:00.123798 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.123755    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304cd043-6da9-4144-9aca-dc55f326a17a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3b68163-89j6s\" (UID: \"304cd043-6da9-4144-9aca-dc55f326a17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"
Oct 30 01:13:00.124030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.123947    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304cd043-6da9-4144-9aca-dc55f326a17a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3b68163-89j6s\" (UID: \"304cd043-6da9-4144-9aca-dc55f326a17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"
Oct 30 01:13:00.124030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.124005    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfdhl\" (UniqueName: \"kubernetes.io/projected/304cd043-6da9-4144-9aca-dc55f326a17a-kube-api-access-rfdhl\") pod \"maas-api-revoke-sub-f3b68163-89j6s\" (UID: \"304cd043-6da9-4144-9aca-dc55f326a17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"
Oct 30 01:13:00.224858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.224817    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304cd043-6da9-4144-9aca-dc55f326a17a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3b68163-89j6s\" (UID: \"304cd043-6da9-4144-9aca-dc55f326a17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"
Oct 30 01:13:00.225094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.224921    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304cd043-6da9-4144-9aca-dc55f326a17a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3b68163-89j6s\" (UID: \"304cd043-6da9-4144-9aca-dc55f326a17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"
Oct 30 01:13:00.225094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.224956    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfdhl\" (UniqueName: \"kubernetes.io/projected/304cd043-6da9-4144-9aca-dc55f326a17a-kube-api-access-rfdhl\") pod \"maas-api-revoke-sub-f3b68163-89j6s\" (UID: \"304cd043-6da9-4144-9aca-dc55f326a17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"
Oct 30 01:13:00.225718 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.225686    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304cd043-6da9-4144-9aca-dc55f326a17a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3b68163-89j6s\" (UID: \"304cd043-6da9-4144-9aca-dc55f326a17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"
Oct 30 01:13:00.227712 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.227681    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304cd043-6da9-4144-9aca-dc55f326a17a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3b68163-89j6s\" (UID: \"304cd043-6da9-4144-9aca-dc55f326a17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"
Oct 30 01:13:00.246926 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.246890    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfdhl\" (UniqueName: \"kubernetes.io/projected/304cd043-6da9-4144-9aca-dc55f326a17a-kube-api-access-rfdhl\") pod \"maas-api-revoke-sub-f3b68163-89j6s\" (UID: \"304cd043-6da9-4144-9aca-dc55f326a17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"
Oct 30 01:13:00.361901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.361803    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"
Oct 30 01:13:00.667441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:00.667412    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"]
Oct 30 01:13:00.670001 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:13:00.669966    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304cd043_6da9_4144_9aca_dc55f326a17a.slice/crio-1b8083d087a90716936b2eeebc98f805def2ed022ab797dce35a7568bd3f4dc7 WatchSource:0}: Error finding container 1b8083d087a90716936b2eeebc98f805def2ed022ab797dce35a7568bd3f4dc7: Status 404 returned error can't find the container with id 1b8083d087a90716936b2eeebc98f805def2ed022ab797dce35a7568bd3f4dc7
Oct 30 01:13:01.339113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:01.339057    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s" event={"ID":"304cd043-6da9-4144-9aca-dc55f326a17a","Type":"ContainerStarted","Data":"dfd6c7d95dce5564807c7bd24bd432eb9149a2dc2763a70d3d43fd89a830ddba"}
Oct 30 01:13:01.339568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:01.339125    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s" event={"ID":"304cd043-6da9-4144-9aca-dc55f326a17a","Type":"ContainerStarted","Data":"1b8083d087a90716936b2eeebc98f805def2ed022ab797dce35a7568bd3f4dc7"}
Oct 30 01:13:01.377036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:01.376971    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s" podStartSLOduration=1.376954162 podStartE2EDuration="1.376954162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:01.374713839 +0000 UTC m=+1890.124369151" watchObservedRunningTime="2026-10-30 01:13:01.376954162 +0000 UTC m=+1890.126609472"
Oct 30 01:13:02.348688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:02.348648    4425 generic.go:356] "Generic (PLEG): container finished" podID="304cd043-6da9-4144-9aca-dc55f326a17a" containerID="dfd6c7d95dce5564807c7bd24bd432eb9149a2dc2763a70d3d43fd89a830ddba" exitCode=0
Oct 30 01:13:02.349137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:02.348731    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s" event={"ID":"304cd043-6da9-4144-9aca-dc55f326a17a","Type":"ContainerDied","Data":"dfd6c7d95dce5564807c7bd24bd432eb9149a2dc2763a70d3d43fd89a830ddba"}
Oct 30 01:13:03.523672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:03.523642    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"
Oct 30 01:13:03.569279 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:03.569241    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/304cd043-6da9-4144-9aca-dc55f326a17a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304cd043-6da9-4144-9aca-dc55f326a17a-maas-api-xks-ca\") pod \"304cd043-6da9-4144-9aca-dc55f326a17a\" (UID: \"304cd043-6da9-4144-9aca-dc55f326a17a\") "
Oct 30 01:13:03.569460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:03.569399    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/304cd043-6da9-4144-9aca-dc55f326a17a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304cd043-6da9-4144-9aca-dc55f326a17a-maas-api-openshift-ca\") pod \"304cd043-6da9-4144-9aca-dc55f326a17a\" (UID: \"304cd043-6da9-4144-9aca-dc55f326a17a\") "
Oct 30 01:13:03.569460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:03.569425    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/304cd043-6da9-4144-9aca-dc55f326a17a-kube-api-access-rfdhl\" (UniqueName: \"kubernetes.io/projected/304cd043-6da9-4144-9aca-dc55f326a17a-kube-api-access-rfdhl\") pod \"304cd043-6da9-4144-9aca-dc55f326a17a\" (UID: \"304cd043-6da9-4144-9aca-dc55f326a17a\") "
Oct 30 01:13:03.569891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:03.569854    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/304cd043-6da9-4144-9aca-dc55f326a17a-maas-api-openshift-ca" pod "304cd043-6da9-4144-9aca-dc55f326a17a" (UID: "304cd043-6da9-4144-9aca-dc55f326a17a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:03.571847 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:03.571820    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/304cd043-6da9-4144-9aca-dc55f326a17a-maas-api-xks-ca" pod "304cd043-6da9-4144-9aca-dc55f326a17a" (UID: "304cd043-6da9-4144-9aca-dc55f326a17a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:03.572242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:03.572221    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304cd043-6da9-4144-9aca-dc55f326a17a-kube-api-access-rfdhl" pod "304cd043-6da9-4144-9aca-dc55f326a17a" (UID: "304cd043-6da9-4144-9aca-dc55f326a17a"). InnerVolumeSpecName "kube-api-access-rfdhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:03.670597 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:03.670511    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304cd043-6da9-4144-9aca-dc55f326a17a-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:03.670597 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:03.670542    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304cd043-6da9-4144-9aca-dc55f326a17a-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:03.670597 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:03.670554    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfdhl\" (UniqueName: \"kubernetes.io/projected/304cd043-6da9-4144-9aca-dc55f326a17a-kube-api-access-rfdhl\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:04.366563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:04.366529    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"
Oct 30 01:13:04.366563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:04.366557    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s" event={"ID":"304cd043-6da9-4144-9aca-dc55f326a17a","Type":"ContainerDied","Data":"1b8083d087a90716936b2eeebc98f805def2ed022ab797dce35a7568bd3f4dc7"}
Oct 30 01:13:04.366832 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:04.366609    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b8083d087a90716936b2eeebc98f805def2ed022ab797dce35a7568bd3f4dc7"
Oct 30 01:13:06.532033 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.531990    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"]
Oct 30 01:13:06.532748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.532724    4425 cpu_manager.go:414] "containerMap: removing container" podUID="304cd043-6da9-4144-9aca-dc55f326a17a" containerName="revoke-keys"
Oct 30 01:13:06.532895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.532749    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304cd043-6da9-4144-9aca-dc55f326a17a" containerName="revoke-keys"
Oct 30 01:13:06.532996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.532980    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="304cd043-6da9-4144-9aca-dc55f326a17a" containerName="revoke-keys"
Oct 30 01:13:06.536869 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.536846    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"
Oct 30 01:13:06.556094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.556054    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"]
Oct 30 01:13:06.562058 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.562027    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:13:06.562260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.562081    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:13:06.562260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.562101    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:13:06.562260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.562226    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:13:06.611595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.611555    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f3c083-0082-47eb-b638-acb89ee0f865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3827c40-kckhw\" (UID: \"76f3c083-0082-47eb-b638-acb89ee0f865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"
Oct 30 01:13:06.611785 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.611617    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f3c083-0082-47eb-b638-acb89ee0f865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3827c40-kckhw\" (UID: \"76f3c083-0082-47eb-b638-acb89ee0f865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"
Oct 30 01:13:06.611882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.611860    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7g7m\" (UniqueName: \"kubernetes.io/projected/76f3c083-0082-47eb-b638-acb89ee0f865-kube-api-access-z7g7m\") pod \"maas-api-revoke-sub-b3827c40-kckhw\" (UID: \"76f3c083-0082-47eb-b638-acb89ee0f865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"
Oct 30 01:13:06.713637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.713602    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7g7m\" (UniqueName: \"kubernetes.io/projected/76f3c083-0082-47eb-b638-acb89ee0f865-kube-api-access-z7g7m\") pod \"maas-api-revoke-sub-b3827c40-kckhw\" (UID: \"76f3c083-0082-47eb-b638-acb89ee0f865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"
Oct 30 01:13:06.713843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.713717    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f3c083-0082-47eb-b638-acb89ee0f865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3827c40-kckhw\" (UID: \"76f3c083-0082-47eb-b638-acb89ee0f865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"
Oct 30 01:13:06.713843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.713773    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f3c083-0082-47eb-b638-acb89ee0f865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3827c40-kckhw\" (UID: \"76f3c083-0082-47eb-b638-acb89ee0f865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"
Oct 30 01:13:06.714653 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.714628    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f3c083-0082-47eb-b638-acb89ee0f865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3827c40-kckhw\" (UID: \"76f3c083-0082-47eb-b638-acb89ee0f865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"
Oct 30 01:13:06.716775 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.716746    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f3c083-0082-47eb-b638-acb89ee0f865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3827c40-kckhw\" (UID: \"76f3c083-0082-47eb-b638-acb89ee0f865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"
Oct 30 01:13:06.734922 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.734884    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7g7m\" (UniqueName: \"kubernetes.io/projected/76f3c083-0082-47eb-b638-acb89ee0f865-kube-api-access-z7g7m\") pod \"maas-api-revoke-sub-b3827c40-kckhw\" (UID: \"76f3c083-0082-47eb-b638-acb89ee0f865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"
Oct 30 01:13:06.852220 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:06.852122    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"
Oct 30 01:13:07.159824 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:07.159789    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"]
Oct 30 01:13:07.161619 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:13:07.161579    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f3c083_0082_47eb_b638_acb89ee0f865.slice/crio-bd1d1dac24efc7eccd78bea2812cb4e2ca108bd3c40971af54545d8024970902 WatchSource:0}: Error finding container bd1d1dac24efc7eccd78bea2812cb4e2ca108bd3c40971af54545d8024970902: Status 404 returned error can't find the container with id bd1d1dac24efc7eccd78bea2812cb4e2ca108bd3c40971af54545d8024970902
Oct 30 01:13:07.389937 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:07.389897    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw" event={"ID":"76f3c083-0082-47eb-b638-acb89ee0f865","Type":"ContainerStarted","Data":"bd1d1dac24efc7eccd78bea2812cb4e2ca108bd3c40971af54545d8024970902"}
Oct 30 01:13:07.559048 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:13:07.558998    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:13:08.035480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:08.035438    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"]
Oct 30 01:13:08.039728 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:08.039704    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"
Oct 30 01:13:08.063261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:08.063223    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"]
Oct 30 01:13:08.130808 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:08.130766    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2127eacf-p8d77\" (UID: \"a352527e-bda1-46c6-96c9-e6d63cb0fa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"
Oct 30 01:13:08.130998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:08.130972    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2127eacf-p8d77\" (UID: \"a352527e-bda1-46c6-96c9-e6d63cb0fa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"
Oct 30 01:13:08.131077 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:08.131061    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwmgp\" (UniqueName: \"kubernetes.io/projected/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-kube-api-access-vwmgp\") pod \"maas-api-revoke-sub-2127eacf-p8d77\" (UID: \"a352527e-bda1-46c6-96c9-e6d63cb0fa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"
Oct 30 01:13:08.231961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:08.231920    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2127eacf-p8d77\" (UID: \"a352527e-bda1-46c6-96c9-e6d63cb0fa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"
Oct 30 01:13:08.232216 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:08.232011    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwmgp\" (UniqueName: \"kubernetes.io/projected/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-kube-api-access-vwmgp\") pod \"maas-api-revoke-sub-2127eacf-p8d77\" (UID: \"a352527e-bda1-46c6-96c9-e6d63cb0fa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"
Oct 30 01:13:08.232216 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:08.232076    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2127eacf-p8d77\" (UID: \"a352527e-bda1-46c6-96c9-e6d63cb0fa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"
Oct 30 01:13:08.232798 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:08.232769    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2127eacf-p8d77\" (UID: \"a352527e-bda1-46c6-96c9-e6d63cb0fa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"
Oct 30 01:13:08.234604 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:08.234580    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2127eacf-p8d77\" (UID: \"a352527e-bda1-46c6-96c9-e6d63cb0fa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"
Oct 30 01:13:08.253261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:08.253227    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwmgp\" (UniqueName: \"kubernetes.io/projected/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-kube-api-access-vwmgp\") pod \"maas-api-revoke-sub-2127eacf-p8d77\" (UID: \"a352527e-bda1-46c6-96c9-e6d63cb0fa0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"
Oct 30 01:13:08.359461 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:08.359364    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"
Oct 30 01:13:08.403529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:08.403491    4425 generic.go:356] "Generic (PLEG): container finished" podID="76f3c083-0082-47eb-b638-acb89ee0f865" containerID="78e90bfae287c9df02872f0e29e44fa989f99aea3a83e9038a86c5608346b023" exitCode=0
Oct 30 01:13:08.403713 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:08.403565    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw" event={"ID":"76f3c083-0082-47eb-b638-acb89ee0f865","Type":"ContainerDied","Data":"78e90bfae287c9df02872f0e29e44fa989f99aea3a83e9038a86c5608346b023"}
Oct 30 01:13:08.671872 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:08.671521    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"]
Oct 30 01:13:08.673095 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:13:08.673057    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda352527e_bda1_46c6_96c9_e6d63cb0fa0d.slice/crio-dee879f547e238c429816939d6ce4a9024dc0fbda1a93c1035af259568c33a45 WatchSource:0}: Error finding container dee879f547e238c429816939d6ce4a9024dc0fbda1a93c1035af259568c33a45: Status 404 returned error can't find the container with id dee879f547e238c429816939d6ce4a9024dc0fbda1a93c1035af259568c33a45
Oct 30 01:13:09.413717 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:09.413667    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77" event={"ID":"a352527e-bda1-46c6-96c9-e6d63cb0fa0d","Type":"ContainerStarted","Data":"c372065e8ec675ac1f0531e77aabc4b1e4ed499e3b36616d4e2df122381e2019"}
Oct 30 01:13:09.413892 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:09.413733    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77" event={"ID":"a352527e-bda1-46c6-96c9-e6d63cb0fa0d","Type":"ContainerStarted","Data":"dee879f547e238c429816939d6ce4a9024dc0fbda1a93c1035af259568c33a45"}
Oct 30 01:13:09.452795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:09.452735    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77" podStartSLOduration=1.452714009 podStartE2EDuration="1.452714009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:09.449133077 +0000 UTC m=+1898.198788410" watchObservedRunningTime="2026-10-30 01:13:09.452714009 +0000 UTC m=+1898.202369319"
Oct 30 01:13:09.581726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:09.581694    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"
Oct 30 01:13:09.651090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:09.650620    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76f3c083-0082-47eb-b638-acb89ee0f865-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f3c083-0082-47eb-b638-acb89ee0f865-maas-api-xks-ca\") pod \"76f3c083-0082-47eb-b638-acb89ee0f865\" (UID: \"76f3c083-0082-47eb-b638-acb89ee0f865\") "
Oct 30 01:13:09.651090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:09.650679    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f3c083-0082-47eb-b638-acb89ee0f865-kube-api-access-z7g7m\" (UniqueName: \"kubernetes.io/projected/76f3c083-0082-47eb-b638-acb89ee0f865-kube-api-access-z7g7m\") pod \"76f3c083-0082-47eb-b638-acb89ee0f865\" (UID: \"76f3c083-0082-47eb-b638-acb89ee0f865\") "
Oct 30 01:13:09.651090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:09.650843    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76f3c083-0082-47eb-b638-acb89ee0f865-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f3c083-0082-47eb-b638-acb89ee0f865-maas-api-openshift-ca\") pod \"76f3c083-0082-47eb-b638-acb89ee0f865\" (UID: \"76f3c083-0082-47eb-b638-acb89ee0f865\") "
Oct 30 01:13:09.651482 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:09.651420    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f3c083-0082-47eb-b638-acb89ee0f865-maas-api-openshift-ca" pod "76f3c083-0082-47eb-b638-acb89ee0f865" (UID: "76f3c083-0082-47eb-b638-acb89ee0f865"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:09.653032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:09.652270    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76f3c083-0082-47eb-b638-acb89ee0f865-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.654234 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:09.654198    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f3c083-0082-47eb-b638-acb89ee0f865-kube-api-access-z7g7m" pod "76f3c083-0082-47eb-b638-acb89ee0f865" (UID: "76f3c083-0082-47eb-b638-acb89ee0f865"). InnerVolumeSpecName "kube-api-access-z7g7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:09.654634 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:09.654607    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76f3c083-0082-47eb-b638-acb89ee0f865-maas-api-xks-ca" pod "76f3c083-0082-47eb-b638-acb89ee0f865" (UID: "76f3c083-0082-47eb-b638-acb89ee0f865"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:09.753484 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:09.753451    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76f3c083-0082-47eb-b638-acb89ee0f865-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:09.753484 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:09.753478    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7g7m\" (UniqueName: \"kubernetes.io/projected/76f3c083-0082-47eb-b638-acb89ee0f865-kube-api-access-z7g7m\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:10.423274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:10.423234    4425 generic.go:356] "Generic (PLEG): container finished" podID="a352527e-bda1-46c6-96c9-e6d63cb0fa0d" containerID="c372065e8ec675ac1f0531e77aabc4b1e4ed499e3b36616d4e2df122381e2019" exitCode=0
Oct 30 01:13:10.423498 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:10.423272    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77" event={"ID":"a352527e-bda1-46c6-96c9-e6d63cb0fa0d","Type":"ContainerDied","Data":"c372065e8ec675ac1f0531e77aabc4b1e4ed499e3b36616d4e2df122381e2019"}
Oct 30 01:13:10.424948 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:10.424926    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"
Oct 30 01:13:10.425093 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:10.424924    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw" event={"ID":"76f3c083-0082-47eb-b638-acb89ee0f865","Type":"ContainerDied","Data":"bd1d1dac24efc7eccd78bea2812cb4e2ca108bd3c40971af54545d8024970902"}
Oct 30 01:13:10.425093 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:10.425056    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd1d1dac24efc7eccd78bea2812cb4e2ca108bd3c40971af54545d8024970902"
Oct 30 01:13:11.603882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:11.603849    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"
Oct 30 01:13:11.787890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:11.787858    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-maas-api-openshift-ca\") pod \"a352527e-bda1-46c6-96c9-e6d63cb0fa0d\" (UID: \"a352527e-bda1-46c6-96c9-e6d63cb0fa0d\") "
Oct 30 01:13:11.788105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:11.787952    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-maas-api-xks-ca\") pod \"a352527e-bda1-46c6-96c9-e6d63cb0fa0d\" (UID: \"a352527e-bda1-46c6-96c9-e6d63cb0fa0d\") "
Oct 30 01:13:11.788105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:11.788051    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-kube-api-access-vwmgp\" (UniqueName: \"kubernetes.io/projected/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-kube-api-access-vwmgp\") pod \"a352527e-bda1-46c6-96c9-e6d63cb0fa0d\" (UID: \"a352527e-bda1-46c6-96c9-e6d63cb0fa0d\") "
Oct 30 01:13:11.788433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:11.788401    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-maas-api-openshift-ca" pod "a352527e-bda1-46c6-96c9-e6d63cb0fa0d" (UID: "a352527e-bda1-46c6-96c9-e6d63cb0fa0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:11.788742 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:11.788714    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:11.790646 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:11.790621    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-maas-api-xks-ca" pod "a352527e-bda1-46c6-96c9-e6d63cb0fa0d" (UID: "a352527e-bda1-46c6-96c9-e6d63cb0fa0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:11.790918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:11.790902    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-kube-api-access-vwmgp" pod "a352527e-bda1-46c6-96c9-e6d63cb0fa0d" (UID: "a352527e-bda1-46c6-96c9-e6d63cb0fa0d"). InnerVolumeSpecName "kube-api-access-vwmgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:11.890268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:11.890224    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwmgp\" (UniqueName: \"kubernetes.io/projected/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-kube-api-access-vwmgp\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:11.890268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:11.890258    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a352527e-bda1-46c6-96c9-e6d63cb0fa0d-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:12.444230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:12.444191    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77" event={"ID":"a352527e-bda1-46c6-96c9-e6d63cb0fa0d","Type":"ContainerDied","Data":"dee879f547e238c429816939d6ce4a9024dc0fbda1a93c1035af259568c33a45"}
Oct 30 01:13:12.444430 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:12.444245    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"
Oct 30 01:13:12.444430 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:12.444257    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dee879f547e238c429816939d6ce4a9024dc0fbda1a93c1035af259568c33a45"
Oct 30 01:13:20.364465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.364418    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"]
Oct 30 01:13:20.365244 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.365219    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a352527e-bda1-46c6-96c9-e6d63cb0fa0d" containerName="revoke-keys"
Oct 30 01:13:20.365372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.365244    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a352527e-bda1-46c6-96c9-e6d63cb0fa0d" containerName="revoke-keys"
Oct 30 01:13:20.365372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.365266    4425 cpu_manager.go:414] "containerMap: removing container" podUID="76f3c083-0082-47eb-b638-acb89ee0f865" containerName="revoke-keys"
Oct 30 01:13:20.365372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.365274    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f3c083-0082-47eb-b638-acb89ee0f865" containerName="revoke-keys"
Oct 30 01:13:20.365574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.365464    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a352527e-bda1-46c6-96c9-e6d63cb0fa0d" containerName="revoke-keys"
Oct 30 01:13:20.365574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.365479    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f3c083-0082-47eb-b638-acb89ee0f865" containerName="revoke-keys"
Oct 30 01:13:20.370286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.370255    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"
Oct 30 01:13:20.387955 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.387918    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:13:20.387955 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.387918    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:13:20.388189 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.388008    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:13:20.388189 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.388079    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:13:20.390643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.390615    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"]
Oct 30 01:13:20.431566 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.431528    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18b6bb0f-2j97k\" (UID: \"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"
Oct 30 01:13:20.431761 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.431693    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18b6bb0f-2j97k\" (UID: \"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"
Oct 30 01:13:20.431809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.431759    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l9dq\" (UniqueName: \"kubernetes.io/projected/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-kube-api-access-5l9dq\") pod \"maas-api-revoke-sub-18b6bb0f-2j97k\" (UID: \"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"
Oct 30 01:13:20.532924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.532883    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l9dq\" (UniqueName: \"kubernetes.io/projected/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-kube-api-access-5l9dq\") pod \"maas-api-revoke-sub-18b6bb0f-2j97k\" (UID: \"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"
Oct 30 01:13:20.533138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.532989    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18b6bb0f-2j97k\" (UID: \"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"
Oct 30 01:13:20.533138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.533097    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18b6bb0f-2j97k\" (UID: \"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"
Oct 30 01:13:20.533810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.533775    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18b6bb0f-2j97k\" (UID: \"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"
Oct 30 01:13:20.535558 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.535535    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18b6bb0f-2j97k\" (UID: \"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"
Oct 30 01:13:20.554983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.554949    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l9dq\" (UniqueName: \"kubernetes.io/projected/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-kube-api-access-5l9dq\") pod \"maas-api-revoke-sub-18b6bb0f-2j97k\" (UID: \"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"
Oct 30 01:13:20.685070 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.684974    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"
Oct 30 01:13:20.994271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:20.994241    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"]
Oct 30 01:13:20.996516 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:13:20.996478    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a5a9d7_8c0d_46ac_9db9_1ab3af8f728e.slice/crio-644cd5ced962afc1d9a86962385bf095719eec1e85176064fb053dc960ffe633 WatchSource:0}: Error finding container 644cd5ced962afc1d9a86962385bf095719eec1e85176064fb053dc960ffe633: Status 404 returned error can't find the container with id 644cd5ced962afc1d9a86962385bf095719eec1e85176064fb053dc960ffe633
Oct 30 01:13:21.525729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:21.525624    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k" event={"ID":"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e","Type":"ContainerStarted","Data":"8a239bb36b7c0a3f2722838ceb3e716958fcbd1584fd00249c9b84dd6cd064bf"}
Oct 30 01:13:21.525729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:21.525692    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k" event={"ID":"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e","Type":"ContainerStarted","Data":"644cd5ced962afc1d9a86962385bf095719eec1e85176064fb053dc960ffe633"}
Oct 30 01:13:22.208693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:22.208470    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k" podStartSLOduration=2.208447774 podStartE2EDuration="2.208447774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:21.577442528 +0000 UTC m=+1910.327097836" watchObservedRunningTime="2026-10-30 01:13:22.208447774 +0000 UTC m=+1910.958103084"
Oct 30 01:13:22.210982 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:22.210947    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"]
Oct 30 01:13:22.218697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:22.218665    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"
Oct 30 01:13:22.236480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:22.235687    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"]
Oct 30 01:13:22.254365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:22.254322    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41cc0607-960c-4a25-bd81-9c3352304048-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11ad4bf4-9zclv\" (UID: \"41cc0607-960c-4a25-bd81-9c3352304048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"
Oct 30 01:13:22.254579 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:22.254385    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41cc0607-960c-4a25-bd81-9c3352304048-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11ad4bf4-9zclv\" (UID: \"41cc0607-960c-4a25-bd81-9c3352304048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"
Oct 30 01:13:22.254579 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:22.254526    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p262\" (UniqueName: \"kubernetes.io/projected/41cc0607-960c-4a25-bd81-9c3352304048-kube-api-access-5p262\") pod \"maas-api-revoke-sub-11ad4bf4-9zclv\" (UID: \"41cc0607-960c-4a25-bd81-9c3352304048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"
Oct 30 01:13:22.356265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:22.356215    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p262\" (UniqueName: \"kubernetes.io/projected/41cc0607-960c-4a25-bd81-9c3352304048-kube-api-access-5p262\") pod \"maas-api-revoke-sub-11ad4bf4-9zclv\" (UID: \"41cc0607-960c-4a25-bd81-9c3352304048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"
Oct 30 01:13:22.356491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:22.356414    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41cc0607-960c-4a25-bd81-9c3352304048-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11ad4bf4-9zclv\" (UID: \"41cc0607-960c-4a25-bd81-9c3352304048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"
Oct 30 01:13:22.356491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:22.356441    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41cc0607-960c-4a25-bd81-9c3352304048-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11ad4bf4-9zclv\" (UID: \"41cc0607-960c-4a25-bd81-9c3352304048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"
Oct 30 01:13:22.357113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:22.357093    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41cc0607-960c-4a25-bd81-9c3352304048-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11ad4bf4-9zclv\" (UID: \"41cc0607-960c-4a25-bd81-9c3352304048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"
Oct 30 01:13:22.359051 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:22.359026    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41cc0607-960c-4a25-bd81-9c3352304048-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11ad4bf4-9zclv\" (UID: \"41cc0607-960c-4a25-bd81-9c3352304048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"
Oct 30 01:13:22.378442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:22.378401    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p262\" (UniqueName: \"kubernetes.io/projected/41cc0607-960c-4a25-bd81-9c3352304048-kube-api-access-5p262\") pod \"maas-api-revoke-sub-11ad4bf4-9zclv\" (UID: \"41cc0607-960c-4a25-bd81-9c3352304048\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"
Oct 30 01:13:22.534987 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:22.534947    4425 generic.go:356] "Generic (PLEG): container finished" podID="48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e" containerID="8a239bb36b7c0a3f2722838ceb3e716958fcbd1584fd00249c9b84dd6cd064bf" exitCode=0
Oct 30 01:13:22.535464 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:22.535029    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k" event={"ID":"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e","Type":"ContainerDied","Data":"8a239bb36b7c0a3f2722838ceb3e716958fcbd1584fd00249c9b84dd6cd064bf"}
Oct 30 01:13:22.535464 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:22.535220    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"
Oct 30 01:13:22.844654 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:22.844620    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"]
Oct 30 01:13:22.845355 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:13:22.845320    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41cc0607_960c_4a25_bd81_9c3352304048.slice/crio-9b48c34cdc198c725649a4f7ddf85b9baf56e5e4a89c884ab55d59319f95ac96 WatchSource:0}: Error finding container 9b48c34cdc198c725649a4f7ddf85b9baf56e5e4a89c884ab55d59319f95ac96: Status 404 returned error can't find the container with id 9b48c34cdc198c725649a4f7ddf85b9baf56e5e4a89c884ab55d59319f95ac96
Oct 30 01:13:23.544526 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:23.544484    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv" event={"ID":"41cc0607-960c-4a25-bd81-9c3352304048","Type":"ContainerStarted","Data":"8ec53a13ead2b00c9662fba6d67d93a85103928bf7011d3abcf7832bbc826ef9"}
Oct 30 01:13:23.545018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:23.544538    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv" event={"ID":"41cc0607-960c-4a25-bd81-9c3352304048","Type":"ContainerStarted","Data":"9b48c34cdc198c725649a4f7ddf85b9baf56e5e4a89c884ab55d59319f95ac96"}
Oct 30 01:13:23.584709 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:23.584650    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv" podStartSLOduration=1.584634813 podStartE2EDuration="1.584634813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:23.58404512 +0000 UTC m=+1912.333700431" watchObservedRunningTime="2026-10-30 01:13:23.584634813 +0000 UTC m=+1912.334290121"
Oct 30 01:13:23.698096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:23.698061    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"
Oct 30 01:13:23.775353 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:23.775299    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-kube-api-access-5l9dq\" (UniqueName: \"kubernetes.io/projected/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-kube-api-access-5l9dq\") pod \"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e\" (UID: \"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e\") "
Oct 30 01:13:23.775550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:23.775418    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-maas-api-openshift-ca\") pod \"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e\" (UID: \"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e\") "
Oct 30 01:13:23.775721 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:23.775702    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-maas-api-xks-ca\") pod \"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e\" (UID: \"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e\") "
Oct 30 01:13:23.775823 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:23.775795    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-maas-api-openshift-ca" pod "48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e" (UID: "48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:23.776256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:23.776231    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:23.777767 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:23.777736    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-kube-api-access-5l9dq" pod "48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e" (UID: "48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e"). InnerVolumeSpecName "kube-api-access-5l9dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:23.778061 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:23.778033    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-maas-api-xks-ca" pod "48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e" (UID: "48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:23.877609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:23.877571    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:23.877609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:23.877609    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l9dq\" (UniqueName: \"kubernetes.io/projected/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e-kube-api-access-5l9dq\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:24.553501 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:24.553467    4425 generic.go:356] "Generic (PLEG): container finished" podID="41cc0607-960c-4a25-bd81-9c3352304048" containerID="8ec53a13ead2b00c9662fba6d67d93a85103928bf7011d3abcf7832bbc826ef9" exitCode=0
Oct 30 01:13:24.553976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:24.553561    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv" event={"ID":"41cc0607-960c-4a25-bd81-9c3352304048","Type":"ContainerDied","Data":"8ec53a13ead2b00c9662fba6d67d93a85103928bf7011d3abcf7832bbc826ef9"}
Oct 30 01:13:24.555217 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:24.555147    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"
Oct 30 01:13:24.555217 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:24.555181    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k" event={"ID":"48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e","Type":"ContainerDied","Data":"644cd5ced962afc1d9a86962385bf095719eec1e85176064fb053dc960ffe633"}
Oct 30 01:13:24.555427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:24.555231    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="644cd5ced962afc1d9a86962385bf095719eec1e85176064fb053dc960ffe633"
Oct 30 01:13:25.745222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:25.745187    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"
Oct 30 01:13:25.803143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:25.803097    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41cc0607-960c-4a25-bd81-9c3352304048-kube-api-access-5p262\" (UniqueName: \"kubernetes.io/projected/41cc0607-960c-4a25-bd81-9c3352304048-kube-api-access-5p262\") pod \"41cc0607-960c-4a25-bd81-9c3352304048\" (UID: \"41cc0607-960c-4a25-bd81-9c3352304048\") "
Oct 30 01:13:25.803394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:25.803318    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41cc0607-960c-4a25-bd81-9c3352304048-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41cc0607-960c-4a25-bd81-9c3352304048-maas-api-xks-ca\") pod \"41cc0607-960c-4a25-bd81-9c3352304048\" (UID: \"41cc0607-960c-4a25-bd81-9c3352304048\") "
Oct 30 01:13:25.803394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:25.803366    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41cc0607-960c-4a25-bd81-9c3352304048-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41cc0607-960c-4a25-bd81-9c3352304048-maas-api-openshift-ca\") pod \"41cc0607-960c-4a25-bd81-9c3352304048\" (UID: \"41cc0607-960c-4a25-bd81-9c3352304048\") "
Oct 30 01:13:25.804307 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:25.804244    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41cc0607-960c-4a25-bd81-9c3352304048-maas-api-openshift-ca" pod "41cc0607-960c-4a25-bd81-9c3352304048" (UID: "41cc0607-960c-4a25-bd81-9c3352304048"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:25.805954 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:25.805924    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41cc0607-960c-4a25-bd81-9c3352304048-maas-api-xks-ca" pod "41cc0607-960c-4a25-bd81-9c3352304048" (UID: "41cc0607-960c-4a25-bd81-9c3352304048"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:25.806425 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:25.806349    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cc0607-960c-4a25-bd81-9c3352304048-kube-api-access-5p262" pod "41cc0607-960c-4a25-bd81-9c3352304048" (UID: "41cc0607-960c-4a25-bd81-9c3352304048"). InnerVolumeSpecName "kube-api-access-5p262". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:25.905729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:25.905628    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41cc0607-960c-4a25-bd81-9c3352304048-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:25.905729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:25.905667    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41cc0607-960c-4a25-bd81-9c3352304048-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:25.905729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:25.905679    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p262\" (UniqueName: \"kubernetes.io/projected/41cc0607-960c-4a25-bd81-9c3352304048-kube-api-access-5p262\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:26.578336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:26.578305    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"
Oct 30 01:13:26.578535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:26.578339    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv" event={"ID":"41cc0607-960c-4a25-bd81-9c3352304048","Type":"ContainerDied","Data":"9b48c34cdc198c725649a4f7ddf85b9baf56e5e4a89c884ab55d59319f95ac96"}
Oct 30 01:13:26.578535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:26.578391    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b48c34cdc198c725649a4f7ddf85b9baf56e5e4a89c884ab55d59319f95ac96"
Oct 30 01:13:27.570576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:27.570525    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:13:28.149490 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.149453    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"]
Oct 30 01:13:28.159376 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.159339    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ce5996-smfcf"]
Oct 30 01:13:28.577772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.577726    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"]
Oct 30 01:13:28.578521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.578496    4425 cpu_manager.go:414] "containerMap: removing container" podUID="41cc0607-960c-4a25-bd81-9c3352304048" containerName="revoke-keys"
Oct 30 01:13:28.578637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.578522    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41cc0607-960c-4a25-bd81-9c3352304048" containerName="revoke-keys"
Oct 30 01:13:28.578637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.578548    4425 cpu_manager.go:414] "containerMap: removing container" podUID="48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e" containerName="revoke-keys"
Oct 30 01:13:28.578637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.578556    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e" containerName="revoke-keys"
Oct 30 01:13:28.578780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.578736    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="41cc0607-960c-4a25-bd81-9c3352304048" containerName="revoke-keys"
Oct 30 01:13:28.578780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.578755    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e" containerName="revoke-keys"
Oct 30 01:13:28.582736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.582708    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.611257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.611219    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"]
Oct 30 01:13:28.642548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.642506    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.642792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.642574    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4598591-0f1b-4498-8b16-9353767f087f-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.642792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.642679    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.642792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.642726    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.642792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.642789    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.643033 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.642942    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4598591-0f1b-4498-8b16-9353767f087f-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.643093 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.643038    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znjkk\" (UniqueName: \"kubernetes.io/projected/a4598591-0f1b-4498-8b16-9353767f087f-kube-api-access-znjkk\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.643093 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.643078    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.643252 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.643100    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4598591-0f1b-4498-8b16-9353767f087f-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.744397 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.744353    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znjkk\" (UniqueName: \"kubernetes.io/projected/a4598591-0f1b-4498-8b16-9353767f087f-kube-api-access-znjkk\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.744397 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.744405    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.744680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.744428    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4598591-0f1b-4498-8b16-9353767f087f-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.744680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.744485    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.744680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.744509    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4598591-0f1b-4498-8b16-9353767f087f-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.744680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.744531    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.744680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.744564    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.744680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.744622    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.745007 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.744712    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4598591-0f1b-4498-8b16-9353767f087f-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.745074 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.745036    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-istio-data\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.745138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.745083    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-workload-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.745222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.745182    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-credential-socket\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.745327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.745306    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-workload-certs\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.745459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.745434    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4598591-0f1b-4498-8b16-9353767f087f-istiod-ca-cert\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.747105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.747081    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-istio-envoy\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.747473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.747450    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4598591-0f1b-4498-8b16-9353767f087f-istio-podinfo\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.767149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.767106    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4598591-0f1b-4498-8b16-9353767f087f-istio-token\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.767883 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.767796    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znjkk\" (UniqueName: \"kubernetes.io/projected/a4598591-0f1b-4498-8b16-9353767f087f-kube-api-access-znjkk\") pod \"e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") " pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:28.901064 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:28.900935    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:29.287343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:29.287257    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"]
Oct 30 01:13:29.289718 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:29.289672    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:29.289852 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:29.289774    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:29.289852 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:29.289817    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:29.523145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:29.523047    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4" path="/var/lib/kubelet/pods/8be4a7e8-85ef-4ee1-8dd9-4fa4c73434b4/volumes"
Oct 30 01:13:29.700379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:29.700272    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz" event={"ID":"a4598591-0f1b-4498-8b16-9353767f087f","Type":"ContainerStarted","Data":"e376804362c700266225c7697c8fd3348ecb6a9d8e1e46f34dda0819115dcefb"}
Oct 30 01:13:29.701269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:29.701238    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz" event={"ID":"a4598591-0f1b-4498-8b16-9353767f087f","Type":"ContainerStarted","Data":"6b301d5d9eb000920534d829b65870d2797dcf0fe3925d52ee5839f93a3d5c97"}
Oct 30 01:13:29.775283 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:29.775213    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz" podStartSLOduration=1.7751870269999999 podStartE2EDuration="1.775187027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:29.751091963 +0000 UTC m=+1918.500747273" watchObservedRunningTime="2026-10-30 01:13:29.775187027 +0000 UTC m=+1918.524842337"
Oct 30 01:13:29.907321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:29.907284    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:30.599329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:30.599267    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8" podUID="99499942-631e-4844-b1c4-1c7059fb5ece" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:30.765182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:30.763408    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:30.830184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:30.823417    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w" podUID="66a19982-1f16-4a03-a629-bf9b7d6b50d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 01:13:30.906245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:30.904663    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 01:13:30.906245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:30.904753    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz" podUID="a4598591-0f1b-4498-8b16-9353767f087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 01:13:31.005182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:30.996291    4425 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w" podUID="66a19982-1f16-4a03-a629-bf9b7d6b50d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:31.240596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:31.239253    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" podUID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:31.240596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:31.239253    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2" podUID="13f688db-25e0-4a88-9d1e-c3fd78b7f659" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:31.355258 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:31.337251    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z" podUID="b4ed47ff-0328-4645-be5f-6a6d51a94fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 01:13:31.617192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:31.601265    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8" podUID="99499942-631e-4844-b1c4-1c7059fb5ece" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:31.617192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:31.603029    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:13:31.770182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:31.763733    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 30 01:13:31.770182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:31.764580    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 01:13:31.860035 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:31.859951    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w" podUID="66a19982-1f16-4a03-a629-bf9b7d6b50d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 01:13:31.860459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:31.860427    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7" podUID="834166ed-5c63-4854-9e89-c45041553630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:31.861479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:31.861456    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 01:13:31.937263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:31.907796    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 01:13:31.937263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:31.907872    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz" podUID="a4598591-0f1b-4498-8b16-9353767f087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 01:13:32.167589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.050320    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw" podUID="4d3d278e-ecd0-46ee-a203-33003899a369" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:32.267265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.237293    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc" podUID="c8d1f900-b8c4-43dc-814e-9491e560ea3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:32.267265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.241007    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2" podUID="13f688db-25e0-4a88-9d1e-c3fd78b7f659" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 01:13:32.286577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.276753    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" podUID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 30 01:13:32.286577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.277565    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 01:13:32.286577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.277631    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 01:13:32.359007 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.358960    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s" podUID="b8b27e40-2516-4d18-8c84-5e0925b3f4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:32.359580 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.359546    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z" podUID="b4ed47ff-0328-4645-be5f-6a6d51a94fc8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 01:13:32.360061 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.360041    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 01:13:32.662245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.656259    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8" podUID="99499942-631e-4844-b1c4-1c7059fb5ece" containerName="main" probeResult="failure" output="Get \"https://10.133.0.149:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:32.663455 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.663409    4425 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7" podUID="834166ed-5c63-4854-9e89-c45041553630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:32.815190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.812784    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:32.868181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.867314    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7" podUID="834166ed-5c63-4854-9e89-c45041553630" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 01:13:32.906189 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.905619    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 01:13:32.906189 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.905706    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz" podUID="a4598591-0f1b-4498-8b16-9353767f087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 01:13:32.979240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.976387    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w" podUID="66a19982-1f16-4a03-a629-bf9b7d6b50d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded"
Oct 30 01:13:32.979240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.978487    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:13:32.979240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:32.978548    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 01:13:33.000737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.000393    4425 scope.go:122] "RemoveContainer" containerID="0a9bab0b6a4cc2eca0468b734317b4f1ba338152d6f7478b4635a5b1de4c3148"
Oct 30 01:13:33.046362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.046271    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"]
Oct 30 01:13:33.055431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.055392    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"]
Oct 30 01:13:33.055803 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.055779    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"
Oct 30 01:13:33.085812 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.084560    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 01:13:33.111020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.110976    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:13:33.112073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.111936    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:13:33.112073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.112050    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:13:33.112324 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.112173    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:13:33.112766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.112721    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 01:13:33.117347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.115729    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 01:13:33.120323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.118146    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw" podUID="4d3d278e-ecd0-46ee-a203-33003899a369" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:33.120323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.119071    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 01:13:33.175091 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.175042    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 01:13:33.226184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.226119    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4mgs\" (UniqueName: \"kubernetes.io/projected/8184aa82-c3ff-48b6-975f-0293efcd9f20-kube-api-access-v4mgs\") pod \"maas-api-revoke-sub-43631f79-7z9s8\" (UID: \"8184aa82-c3ff-48b6-975f-0293efcd9f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"
Oct 30 01:13:33.230550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.230456    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8184aa82-c3ff-48b6-975f-0293efcd9f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43631f79-7z9s8\" (UID: \"8184aa82-c3ff-48b6-975f-0293efcd9f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"
Oct 30 01:13:33.231083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.231058    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8184aa82-c3ff-48b6-975f-0293efcd9f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43631f79-7z9s8\" (UID: \"8184aa82-c3ff-48b6-975f-0293efcd9f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"
Oct 30 01:13:33.243083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.243038    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2" podUID="13f688db-25e0-4a88-9d1e-c3fd78b7f659" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 01:13:33.293465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.293412    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" podUID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:33.295541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.295368    4425 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" podUID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:13:33.318742 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.318711    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 01:13:33.319976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.319953    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 01:13:33.340923 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.340876    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8184aa82-c3ff-48b6-975f-0293efcd9f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43631f79-7z9s8\" (UID: \"8184aa82-c3ff-48b6-975f-0293efcd9f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"
Oct 30 01:13:33.341090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.340959    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4mgs\" (UniqueName: \"kubernetes.io/projected/8184aa82-c3ff-48b6-975f-0293efcd9f20-kube-api-access-v4mgs\") pod \"maas-api-revoke-sub-43631f79-7z9s8\" (UID: \"8184aa82-c3ff-48b6-975f-0293efcd9f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"
Oct 30 01:13:33.341090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.341026    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8184aa82-c3ff-48b6-975f-0293efcd9f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43631f79-7z9s8\" (UID: \"8184aa82-c3ff-48b6-975f-0293efcd9f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"
Oct 30 01:13:33.344116 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.344082    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 01:13:33.346813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.345508    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8184aa82-c3ff-48b6-975f-0293efcd9f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43631f79-7z9s8\" (UID: \"8184aa82-c3ff-48b6-975f-0293efcd9f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"
Oct 30 01:13:33.346813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.346753    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8184aa82-c3ff-48b6-975f-0293efcd9f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43631f79-7z9s8\" (UID: \"8184aa82-c3ff-48b6-975f-0293efcd9f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"
Oct 30 01:13:33.397854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.397823    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4mgs\" (UniqueName: \"kubernetes.io/projected/8184aa82-c3ff-48b6-975f-0293efcd9f20-kube-api-access-v4mgs\") pod \"maas-api-revoke-sub-43631f79-7z9s8\" (UID: \"8184aa82-c3ff-48b6-975f-0293efcd9f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"
Oct 30 01:13:33.472187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.470826    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"
Oct 30 01:13:33.596941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.596898    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"]
Oct 30 01:13:33.843197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.843129    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"]
Oct 30 01:13:33.844583 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:13:33.844539    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8184aa82_c3ff_48b6_975f_0293efcd9f20.slice/crio-2687c0b7c9a94a539d582e90bb8cb309d05a8a09aefe4aede9254940835d8a10 WatchSource:0}: Error finding container 2687c0b7c9a94a539d582e90bb8cb309d05a8a09aefe4aede9254940835d8a10: Status 404 returned error can't find the container with id 2687c0b7c9a94a539d582e90bb8cb309d05a8a09aefe4aede9254940835d8a10
Oct 30 01:13:33.902438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.902400    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 01:13:33.902614 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:33.902472    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz" podUID="a4598591-0f1b-4498-8b16-9353767f087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 01:13:34.327249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:34.327211    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8" event={"ID":"8184aa82-c3ff-48b6-975f-0293efcd9f20","Type":"ContainerStarted","Data":"18e129718b99df06a437197d2fa573c6a00e4e20a94df974af95b48a84cc9915"}
Oct 30 01:13:34.327447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:34.327268    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8" event={"ID":"8184aa82-c3ff-48b6-975f-0293efcd9f20","Type":"ContainerStarted","Data":"2687c0b7c9a94a539d582e90bb8cb309d05a8a09aefe4aede9254940835d8a10"}
Oct 30 01:13:34.375249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:34.373000    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8" podStartSLOduration=2.372975251 podStartE2EDuration="2.372975251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:34.366795768 +0000 UTC m=+1923.116451107" watchObservedRunningTime="2026-10-30 01:13:34.372975251 +0000 UTC m=+1923.122630562"
Oct 30 01:13:34.901411 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:34.901370    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 01:13:34.901830 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:34.901431    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz" podUID="a4598591-0f1b-4498-8b16-9353767f087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 01:13:35.178314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.178193    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"]
Oct 30 01:13:35.182909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.182876    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"
Oct 30 01:13:35.198303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.198250    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"]
Oct 30 01:13:35.294686 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.294636    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b4cb78-804f-406f-a7d3-2a869c09282c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ce417c4-ppxzl\" (UID: \"b1b4cb78-804f-406f-a7d3-2a869c09282c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"
Oct 30 01:13:35.294889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.294828    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np2mb\" (UniqueName: \"kubernetes.io/projected/b1b4cb78-804f-406f-a7d3-2a869c09282c-kube-api-access-np2mb\") pod \"maas-api-revoke-sub-5ce417c4-ppxzl\" (UID: \"b1b4cb78-804f-406f-a7d3-2a869c09282c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"
Oct 30 01:13:35.294987 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.294970    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b4cb78-804f-406f-a7d3-2a869c09282c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ce417c4-ppxzl\" (UID: \"b1b4cb78-804f-406f-a7d3-2a869c09282c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"
Oct 30 01:13:35.335904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.335862    4425 generic.go:356] "Generic (PLEG): container finished" podID="8184aa82-c3ff-48b6-975f-0293efcd9f20" containerID="18e129718b99df06a437197d2fa573c6a00e4e20a94df974af95b48a84cc9915" exitCode=0
Oct 30 01:13:35.336117 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.335962    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8" event={"ID":"8184aa82-c3ff-48b6-975f-0293efcd9f20","Type":"ContainerDied","Data":"18e129718b99df06a437197d2fa573c6a00e4e20a94df974af95b48a84cc9915"}
Oct 30 01:13:35.396173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.396124    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b4cb78-804f-406f-a7d3-2a869c09282c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ce417c4-ppxzl\" (UID: \"b1b4cb78-804f-406f-a7d3-2a869c09282c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"
Oct 30 01:13:35.396413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.396312    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b4cb78-804f-406f-a7d3-2a869c09282c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ce417c4-ppxzl\" (UID: \"b1b4cb78-804f-406f-a7d3-2a869c09282c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"
Oct 30 01:13:35.396480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.396425    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np2mb\" (UniqueName: \"kubernetes.io/projected/b1b4cb78-804f-406f-a7d3-2a869c09282c-kube-api-access-np2mb\") pod \"maas-api-revoke-sub-5ce417c4-ppxzl\" (UID: \"b1b4cb78-804f-406f-a7d3-2a869c09282c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"
Oct 30 01:13:35.397061 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.397027    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b4cb78-804f-406f-a7d3-2a869c09282c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ce417c4-ppxzl\" (UID: \"b1b4cb78-804f-406f-a7d3-2a869c09282c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"
Oct 30 01:13:35.398811 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.398786    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b4cb78-804f-406f-a7d3-2a869c09282c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ce417c4-ppxzl\" (UID: \"b1b4cb78-804f-406f-a7d3-2a869c09282c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"
Oct 30 01:13:35.417419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.417387    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np2mb\" (UniqueName: \"kubernetes.io/projected/b1b4cb78-804f-406f-a7d3-2a869c09282c-kube-api-access-np2mb\") pod \"maas-api-revoke-sub-5ce417c4-ppxzl\" (UID: \"b1b4cb78-804f-406f-a7d3-2a869c09282c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"
Oct 30 01:13:35.499748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.499643    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"
Oct 30 01:13:35.821464 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.821429    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"]
Oct 30 01:13:35.822551 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:13:35.822528    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b4cb78_804f_406f_a7d3_2a869c09282c.slice/crio-868a70506c5b53fa89be419107043ba8ea87b5ac9216659c5bf991874131e0c3 WatchSource:0}: Error finding container 868a70506c5b53fa89be419107043ba8ea87b5ac9216659c5bf991874131e0c3: Status 404 returned error can't find the container with id 868a70506c5b53fa89be419107043ba8ea87b5ac9216659c5bf991874131e0c3
Oct 30 01:13:35.902081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.902031    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 01:13:35.902647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:35.902091    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz" podUID="a4598591-0f1b-4498-8b16-9353767f087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 01:13:36.344986 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:36.344937    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl" event={"ID":"b1b4cb78-804f-406f-a7d3-2a869c09282c","Type":"ContainerStarted","Data":"17b9ec166c4e239d95822aada815ea8e443e52de522e1f7fc132776632e97d40"}
Oct 30 01:13:36.345197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:36.345014    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl" event={"ID":"b1b4cb78-804f-406f-a7d3-2a869c09282c","Type":"ContainerStarted","Data":"868a70506c5b53fa89be419107043ba8ea87b5ac9216659c5bf991874131e0c3"}
Oct 30 01:13:36.404024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:36.403897    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl" podStartSLOduration=1.4038727579999999 podStartE2EDuration="1.403872758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:36.399217355 +0000 UTC m=+1925.148872661" watchObservedRunningTime="2026-10-30 01:13:36.403872758 +0000 UTC m=+1925.153528067"
Oct 30 01:13:36.591932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:36.591900    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"
Oct 30 01:13:36.617808 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:36.617765    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8184aa82-c3ff-48b6-975f-0293efcd9f20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8184aa82-c3ff-48b6-975f-0293efcd9f20-maas-api-openshift-ca\") pod \"8184aa82-c3ff-48b6-975f-0293efcd9f20\" (UID: \"8184aa82-c3ff-48b6-975f-0293efcd9f20\") "
Oct 30 01:13:36.617998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:36.617863    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8184aa82-c3ff-48b6-975f-0293efcd9f20-kube-api-access-v4mgs\" (UniqueName: \"kubernetes.io/projected/8184aa82-c3ff-48b6-975f-0293efcd9f20-kube-api-access-v4mgs\") pod \"8184aa82-c3ff-48b6-975f-0293efcd9f20\" (UID: \"8184aa82-c3ff-48b6-975f-0293efcd9f20\") "
Oct 30 01:13:36.617998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:36.617903    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8184aa82-c3ff-48b6-975f-0293efcd9f20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8184aa82-c3ff-48b6-975f-0293efcd9f20-maas-api-xks-ca\") pod \"8184aa82-c3ff-48b6-975f-0293efcd9f20\" (UID: \"8184aa82-c3ff-48b6-975f-0293efcd9f20\") "
Oct 30 01:13:36.618252 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:36.618225    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8184aa82-c3ff-48b6-975f-0293efcd9f20-maas-api-openshift-ca" pod "8184aa82-c3ff-48b6-975f-0293efcd9f20" (UID: "8184aa82-c3ff-48b6-975f-0293efcd9f20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:36.618459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:36.618439    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8184aa82-c3ff-48b6-975f-0293efcd9f20-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:36.620382 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:36.620353    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8184aa82-c3ff-48b6-975f-0293efcd9f20-kube-api-access-v4mgs" pod "8184aa82-c3ff-48b6-975f-0293efcd9f20" (UID: "8184aa82-c3ff-48b6-975f-0293efcd9f20"). InnerVolumeSpecName "kube-api-access-v4mgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:36.620502 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:36.620397    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8184aa82-c3ff-48b6-975f-0293efcd9f20-maas-api-xks-ca" pod "8184aa82-c3ff-48b6-975f-0293efcd9f20" (UID: "8184aa82-c3ff-48b6-975f-0293efcd9f20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:36.720134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:36.720040    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4mgs\" (UniqueName: \"kubernetes.io/projected/8184aa82-c3ff-48b6-975f-0293efcd9f20-kube-api-access-v4mgs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:36.720134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:36.720070    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8184aa82-c3ff-48b6-975f-0293efcd9f20-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:36.901641 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:36.901594    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 01:13:36.901837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:36.901659    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz" podUID="a4598591-0f1b-4498-8b16-9353767f087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 01:13:37.356687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:37.356578    4425 generic.go:356] "Generic (PLEG): container finished" podID="b1b4cb78-804f-406f-a7d3-2a869c09282c" containerID="17b9ec166c4e239d95822aada815ea8e443e52de522e1f7fc132776632e97d40" exitCode=0
Oct 30 01:13:37.356687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:37.356655    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl" event={"ID":"b1b4cb78-804f-406f-a7d3-2a869c09282c","Type":"ContainerDied","Data":"17b9ec166c4e239d95822aada815ea8e443e52de522e1f7fc132776632e97d40"}
Oct 30 01:13:37.358443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:37.358417    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"
Oct 30 01:13:37.358443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:37.358430    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8" event={"ID":"8184aa82-c3ff-48b6-975f-0293efcd9f20","Type":"ContainerDied","Data":"2687c0b7c9a94a539d582e90bb8cb309d05a8a09aefe4aede9254940835d8a10"}
Oct 30 01:13:37.358611 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:37.358485    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2687c0b7c9a94a539d582e90bb8cb309d05a8a09aefe4aede9254940835d8a10"
Oct 30 01:13:37.560443 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:13:37.560401    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:13:38.230246 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:38.230213    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:38.230637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:38.230612    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:38.245499 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:38.245412    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:13:38.533609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:38.533578    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"
Oct 30 01:13:38.650845 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:38.650811    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1b4cb78-804f-406f-a7d3-2a869c09282c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b4cb78-804f-406f-a7d3-2a869c09282c-maas-api-xks-ca\") pod \"b1b4cb78-804f-406f-a7d3-2a869c09282c\" (UID: \"b1b4cb78-804f-406f-a7d3-2a869c09282c\") "
Oct 30 01:13:38.650845 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:38.650850    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b4cb78-804f-406f-a7d3-2a869c09282c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b4cb78-804f-406f-a7d3-2a869c09282c-maas-api-openshift-ca\") pod \"b1b4cb78-804f-406f-a7d3-2a869c09282c\" (UID: \"b1b4cb78-804f-406f-a7d3-2a869c09282c\") "
Oct 30 01:13:38.651100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:38.650915    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b4cb78-804f-406f-a7d3-2a869c09282c-kube-api-access-np2mb\" (UniqueName: \"kubernetes.io/projected/b1b4cb78-804f-406f-a7d3-2a869c09282c-kube-api-access-np2mb\") pod \"b1b4cb78-804f-406f-a7d3-2a869c09282c\" (UID: \"b1b4cb78-804f-406f-a7d3-2a869c09282c\") "
Oct 30 01:13:38.651265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:38.651240    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b4cb78-804f-406f-a7d3-2a869c09282c-maas-api-openshift-ca" pod "b1b4cb78-804f-406f-a7d3-2a869c09282c" (UID: "b1b4cb78-804f-406f-a7d3-2a869c09282c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:38.651493 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:38.651477    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b4cb78-804f-406f-a7d3-2a869c09282c-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:38.653212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:38.653143    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b4cb78-804f-406f-a7d3-2a869c09282c-maas-api-xks-ca" pod "b1b4cb78-804f-406f-a7d3-2a869c09282c" (UID: "b1b4cb78-804f-406f-a7d3-2a869c09282c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:38.653212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:38.653201    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b4cb78-804f-406f-a7d3-2a869c09282c-kube-api-access-np2mb" pod "b1b4cb78-804f-406f-a7d3-2a869c09282c" (UID: "b1b4cb78-804f-406f-a7d3-2a869c09282c"). InnerVolumeSpecName "kube-api-access-np2mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:38.753326 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:38.753227    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b4cb78-804f-406f-a7d3-2a869c09282c-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:38.753326 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:38.753264    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np2mb\" (UniqueName: \"kubernetes.io/projected/b1b4cb78-804f-406f-a7d3-2a869c09282c-kube-api-access-np2mb\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:39.379231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:39.379199    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"
Oct 30 01:13:39.379231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:39.379211    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl" event={"ID":"b1b4cb78-804f-406f-a7d3-2a869c09282c","Type":"ContainerDied","Data":"868a70506c5b53fa89be419107043ba8ea87b5ac9216659c5bf991874131e0c3"}
Oct 30 01:13:39.379487 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:39.379269    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="868a70506c5b53fa89be419107043ba8ea87b5ac9216659c5bf991874131e0c3"
Oct 30 01:13:41.271514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.271466    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"]
Oct 30 01:13:41.272288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.272263    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8184aa82-c3ff-48b6-975f-0293efcd9f20" containerName="revoke-keys"
Oct 30 01:13:41.272421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.272288    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8184aa82-c3ff-48b6-975f-0293efcd9f20" containerName="revoke-keys"
Oct 30 01:13:41.272421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.272352    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b1b4cb78-804f-406f-a7d3-2a869c09282c" containerName="revoke-keys"
Oct 30 01:13:41.272421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.272361    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b4cb78-804f-406f-a7d3-2a869c09282c" containerName="revoke-keys"
Oct 30 01:13:41.272986 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.272522    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b4cb78-804f-406f-a7d3-2a869c09282c" containerName="revoke-keys"
Oct 30 01:13:41.272986 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.272542    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8184aa82-c3ff-48b6-975f-0293efcd9f20" containerName="revoke-keys"
Oct 30 01:13:41.276130 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.276105    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"
Oct 30 01:13:41.290885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.290855    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:13:41.300710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.300657    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:13:41.300922 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.300713    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:13:41.300922 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.300794    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:13:41.305504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.305465    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"]
Oct 30 01:13:41.387919 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.387874    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz\" (UID: \"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"
Oct 30 01:13:41.388134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.388053    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v525d\" (UniqueName: \"kubernetes.io/projected/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-kube-api-access-v525d\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz\" (UID: \"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"
Oct 30 01:13:41.388269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.388211    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz\" (UID: \"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"
Oct 30 01:13:41.489858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.489785    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v525d\" (UniqueName: \"kubernetes.io/projected/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-kube-api-access-v525d\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz\" (UID: \"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"
Oct 30 01:13:41.490095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.489953    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz\" (UID: \"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"
Oct 30 01:13:41.490236 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.490125    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz\" (UID: \"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"
Oct 30 01:13:41.490904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.490881    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz\" (UID: \"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"
Oct 30 01:13:41.492753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.492725    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz\" (UID: \"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"
Oct 30 01:13:41.511646 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.511609    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v525d\" (UniqueName: \"kubernetes.io/projected/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-kube-api-access-v525d\") pod \"maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz\" (UID: \"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"
Oct 30 01:13:41.590939 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.590840    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"
Oct 30 01:13:41.898034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:41.897982    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"]
Oct 30 01:13:41.899053 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:13:41.899025    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64d7b5c_fdf1_4942_ba3a_b13edc03f4b6.slice/crio-d6e4cd08c5e0cd2e128d54a97c423de4e6b0bd77e9753107ee07902676e8ff80 WatchSource:0}: Error finding container d6e4cd08c5e0cd2e128d54a97c423de4e6b0bd77e9753107ee07902676e8ff80: Status 404 returned error can't find the container with id d6e4cd08c5e0cd2e128d54a97c423de4e6b0bd77e9753107ee07902676e8ff80
Oct 30 01:13:42.409574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:42.409530    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz" event={"ID":"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6","Type":"ContainerStarted","Data":"65333d10df1ecf7f4be6efefe5d49b68c2be587ae6299e1e0341f9900396729a"}
Oct 30 01:13:42.410104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:42.409590    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz" event={"ID":"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6","Type":"ContainerStarted","Data":"d6e4cd08c5e0cd2e128d54a97c423de4e6b0bd77e9753107ee07902676e8ff80"}
Oct 30 01:13:42.449832 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:42.449773    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz" podStartSLOduration=1.449755765 podStartE2EDuration="1.449755765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:42.445773408 +0000 UTC m=+1931.195428718" watchObservedRunningTime="2026-10-30 01:13:42.449755765 +0000 UTC m=+1931.199411101"
Oct 30 01:13:42.667957 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:42.667906    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"]
Oct 30 01:13:42.672348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:42.672317    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"
Oct 30 01:13:42.693129 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:42.693087    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"]
Oct 30 01:13:42.707831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:42.707781    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ece684-decd-4047-98f5-c925eb2c2004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3bf48ae-8nsqd\" (UID: \"a5ece684-decd-4047-98f5-c925eb2c2004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"
Oct 30 01:13:42.708027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:42.707930    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ece684-decd-4047-98f5-c925eb2c2004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3bf48ae-8nsqd\" (UID: \"a5ece684-decd-4047-98f5-c925eb2c2004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"
Oct 30 01:13:42.708118 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:42.708101    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qf2m\" (UniqueName: \"kubernetes.io/projected/a5ece684-decd-4047-98f5-c925eb2c2004-kube-api-access-7qf2m\") pod \"maas-api-revoke-sub-a3bf48ae-8nsqd\" (UID: \"a5ece684-decd-4047-98f5-c925eb2c2004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"
Oct 30 01:13:42.809996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:42.809958    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qf2m\" (UniqueName: \"kubernetes.io/projected/a5ece684-decd-4047-98f5-c925eb2c2004-kube-api-access-7qf2m\") pod \"maas-api-revoke-sub-a3bf48ae-8nsqd\" (UID: \"a5ece684-decd-4047-98f5-c925eb2c2004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"
Oct 30 01:13:42.810255 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:42.810116    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ece684-decd-4047-98f5-c925eb2c2004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3bf48ae-8nsqd\" (UID: \"a5ece684-decd-4047-98f5-c925eb2c2004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"
Oct 30 01:13:42.810255 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:42.810231    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ece684-decd-4047-98f5-c925eb2c2004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3bf48ae-8nsqd\" (UID: \"a5ece684-decd-4047-98f5-c925eb2c2004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"
Oct 30 01:13:42.811106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:42.811080    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ece684-decd-4047-98f5-c925eb2c2004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3bf48ae-8nsqd\" (UID: \"a5ece684-decd-4047-98f5-c925eb2c2004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"
Oct 30 01:13:42.812719 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:42.812701    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ece684-decd-4047-98f5-c925eb2c2004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3bf48ae-8nsqd\" (UID: \"a5ece684-decd-4047-98f5-c925eb2c2004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"
Oct 30 01:13:42.831244 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:42.831205    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qf2m\" (UniqueName: \"kubernetes.io/projected/a5ece684-decd-4047-98f5-c925eb2c2004-kube-api-access-7qf2m\") pod \"maas-api-revoke-sub-a3bf48ae-8nsqd\" (UID: \"a5ece684-decd-4047-98f5-c925eb2c2004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"
Oct 30 01:13:42.990782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:42.990674    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"
Oct 30 01:13:43.302213 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:43.302180    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"]
Oct 30 01:13:43.303821 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:13:43.303779    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ece684_decd_4047_98f5_c925eb2c2004.slice/crio-7c62f08508367d2333f8aed67227efe9e7b24c08c74e4b85a49697138a2caeb3 WatchSource:0}: Error finding container 7c62f08508367d2333f8aed67227efe9e7b24c08c74e4b85a49697138a2caeb3: Status 404 returned error can't find the container with id 7c62f08508367d2333f8aed67227efe9e7b24c08c74e4b85a49697138a2caeb3
Oct 30 01:13:43.418933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:43.418897    4425 generic.go:356] "Generic (PLEG): container finished" podID="b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6" containerID="65333d10df1ecf7f4be6efefe5d49b68c2be587ae6299e1e0341f9900396729a" exitCode=0
Oct 30 01:13:43.423419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:43.423384    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz" event={"ID":"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6","Type":"ContainerDied","Data":"65333d10df1ecf7f4be6efefe5d49b68c2be587ae6299e1e0341f9900396729a"}
Oct 30 01:13:43.423602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:43.423440    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd" event={"ID":"a5ece684-decd-4047-98f5-c925eb2c2004","Type":"ContainerStarted","Data":"7c62f08508367d2333f8aed67227efe9e7b24c08c74e4b85a49697138a2caeb3"}
Oct 30 01:13:44.431196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:44.431037    4425 generic.go:356] "Generic (PLEG): container finished" podID="a5ece684-decd-4047-98f5-c925eb2c2004" containerID="16bca5ddd362acd3f03f8791029df8f6ddf3e8e7cbafb4deb70b3bf71298c917" exitCode=0
Oct 30 01:13:44.431196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:44.431110    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd" event={"ID":"a5ece684-decd-4047-98f5-c925eb2c2004","Type":"ContainerDied","Data":"16bca5ddd362acd3f03f8791029df8f6ddf3e8e7cbafb4deb70b3bf71298c917"}
Oct 30 01:13:44.607631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:44.607596    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"
Oct 30 01:13:44.733858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:44.733758    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-maas-api-openshift-ca\") pod \"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6\" (UID: \"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6\") "
Oct 30 01:13:44.734035 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:44.733870    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-kube-api-access-v525d\" (UniqueName: \"kubernetes.io/projected/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-kube-api-access-v525d\") pod \"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6\" (UID: \"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6\") "
Oct 30 01:13:44.734035 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:44.733947    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-maas-api-xks-ca\") pod \"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6\" (UID: \"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6\") "
Oct 30 01:13:44.734271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:44.734235    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-maas-api-openshift-ca" pod "b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6" (UID: "b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:44.734525 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:44.734498    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:44.736401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:44.736364    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-maas-api-xks-ca" pod "b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6" (UID: "b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:44.736529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:44.736443    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-kube-api-access-v525d" pod "b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6" (UID: "b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6"). InnerVolumeSpecName "kube-api-access-v525d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:44.835977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:44.835930    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v525d\" (UniqueName: \"kubernetes.io/projected/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-kube-api-access-v525d\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:44.835977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:44.835972    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:45.440018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:45.439988    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"
Oct 30 01:13:45.440018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:45.440003    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz" event={"ID":"b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6","Type":"ContainerDied","Data":"d6e4cd08c5e0cd2e128d54a97c423de4e6b0bd77e9753107ee07902676e8ff80"}
Oct 30 01:13:45.440506 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:45.440064    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6e4cd08c5e0cd2e128d54a97c423de4e6b0bd77e9753107ee07902676e8ff80"
Oct 30 01:13:45.607925 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:45.607894    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"
Oct 30 01:13:45.746003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:45.745906    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ece684-decd-4047-98f5-c925eb2c2004-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ece684-decd-4047-98f5-c925eb2c2004-maas-api-openshift-ca\") pod \"a5ece684-decd-4047-98f5-c925eb2c2004\" (UID: \"a5ece684-decd-4047-98f5-c925eb2c2004\") "
Oct 30 01:13:45.746234 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:45.746176    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5ece684-decd-4047-98f5-c925eb2c2004-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ece684-decd-4047-98f5-c925eb2c2004-maas-api-xks-ca\") pod \"a5ece684-decd-4047-98f5-c925eb2c2004\" (UID: \"a5ece684-decd-4047-98f5-c925eb2c2004\") "
Oct 30 01:13:45.746333 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:45.746279    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ece684-decd-4047-98f5-c925eb2c2004-kube-api-access-7qf2m\" (UniqueName: \"kubernetes.io/projected/a5ece684-decd-4047-98f5-c925eb2c2004-kube-api-access-7qf2m\") pod \"a5ece684-decd-4047-98f5-c925eb2c2004\" (UID: \"a5ece684-decd-4047-98f5-c925eb2c2004\") "
Oct 30 01:13:45.746476 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:45.746451    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ece684-decd-4047-98f5-c925eb2c2004-maas-api-openshift-ca" pod "a5ece684-decd-4047-98f5-c925eb2c2004" (UID: "a5ece684-decd-4047-98f5-c925eb2c2004"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:45.746897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:45.746877    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ece684-decd-4047-98f5-c925eb2c2004-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:45.748528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:45.748499    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5ece684-decd-4047-98f5-c925eb2c2004-maas-api-xks-ca" pod "a5ece684-decd-4047-98f5-c925eb2c2004" (UID: "a5ece684-decd-4047-98f5-c925eb2c2004"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:45.748631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:45.748552    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ece684-decd-4047-98f5-c925eb2c2004-kube-api-access-7qf2m" pod "a5ece684-decd-4047-98f5-c925eb2c2004" (UID: "a5ece684-decd-4047-98f5-c925eb2c2004"). InnerVolumeSpecName "kube-api-access-7qf2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:45.848394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:45.848355    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ece684-decd-4047-98f5-c925eb2c2004-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:45.848394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:45.848388    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qf2m\" (UniqueName: \"kubernetes.io/projected/a5ece684-decd-4047-98f5-c925eb2c2004-kube-api-access-7qf2m\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:46.450235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.450201    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"
Oct 30 01:13:46.450781 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.450202    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd" event={"ID":"a5ece684-decd-4047-98f5-c925eb2c2004","Type":"ContainerDied","Data":"7c62f08508367d2333f8aed67227efe9e7b24c08c74e4b85a49697138a2caeb3"}
Oct 30 01:13:46.450781 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.450347    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c62f08508367d2333f8aed67227efe9e7b24c08c74e4b85a49697138a2caeb3"
Oct 30 01:13:46.850928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.850887    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"]
Oct 30 01:13:46.851514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.851497    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a5ece684-decd-4047-98f5-c925eb2c2004" containerName="revoke-keys"
Oct 30 01:13:46.851589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.851515    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ece684-decd-4047-98f5-c925eb2c2004" containerName="revoke-keys"
Oct 30 01:13:46.851589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.851540    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6" containerName="revoke-keys"
Oct 30 01:13:46.851589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.851545    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6" containerName="revoke-keys"
Oct 30 01:13:46.851687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.851653    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ece684-decd-4047-98f5-c925eb2c2004" containerName="revoke-keys"
Oct 30 01:13:46.851687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.851663    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6" containerName="revoke-keys"
Oct 30 01:13:46.855504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.855480    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"
Oct 30 01:13:46.870425 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.870390    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:13:46.870599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.870428    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:13:46.870599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.870457    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:13:46.870599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.870463    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:13:46.875302 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.875270    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"]
Oct 30 01:13:46.965756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.965720    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc5z7\" (UniqueName: \"kubernetes.io/projected/b381e5c3-4609-4569-805e-349e19a53cce-kube-api-access-hc5z7\") pod \"maas-api-revoke-sub-2185bf49-vz9mm\" (UID: \"b381e5c3-4609-4569-805e-349e19a53cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"
Oct 30 01:13:46.965951 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.965922    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b381e5c3-4609-4569-805e-349e19a53cce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2185bf49-vz9mm\" (UID: \"b381e5c3-4609-4569-805e-349e19a53cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"
Oct 30 01:13:46.966069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:46.966053    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b381e5c3-4609-4569-805e-349e19a53cce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2185bf49-vz9mm\" (UID: \"b381e5c3-4609-4569-805e-349e19a53cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"
Oct 30 01:13:47.068022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:47.067982    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b381e5c3-4609-4569-805e-349e19a53cce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2185bf49-vz9mm\" (UID: \"b381e5c3-4609-4569-805e-349e19a53cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"
Oct 30 01:13:47.068301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:47.068092    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b381e5c3-4609-4569-805e-349e19a53cce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2185bf49-vz9mm\" (UID: \"b381e5c3-4609-4569-805e-349e19a53cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"
Oct 30 01:13:47.068301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:47.068125    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc5z7\" (UniqueName: \"kubernetes.io/projected/b381e5c3-4609-4569-805e-349e19a53cce-kube-api-access-hc5z7\") pod \"maas-api-revoke-sub-2185bf49-vz9mm\" (UID: \"b381e5c3-4609-4569-805e-349e19a53cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"
Oct 30 01:13:47.068740 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:47.068721    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b381e5c3-4609-4569-805e-349e19a53cce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2185bf49-vz9mm\" (UID: \"b381e5c3-4609-4569-805e-349e19a53cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"
Oct 30 01:13:47.070905 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:47.070875    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b381e5c3-4609-4569-805e-349e19a53cce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2185bf49-vz9mm\" (UID: \"b381e5c3-4609-4569-805e-349e19a53cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"
Oct 30 01:13:47.089931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:47.089890    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc5z7\" (UniqueName: \"kubernetes.io/projected/b381e5c3-4609-4569-805e-349e19a53cce-kube-api-access-hc5z7\") pod \"maas-api-revoke-sub-2185bf49-vz9mm\" (UID: \"b381e5c3-4609-4569-805e-349e19a53cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"
Oct 30 01:13:47.171003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:47.170913    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"
Oct 30 01:13:47.480909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:47.480873    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"]
Oct 30 01:13:47.482797 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:13:47.482751    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb381e5c3_4609_4569_805e_349e19a53cce.slice/crio-b2a23c9bac7c2e113dfced349c36d57ed547e32e16f6a946c29454f75ed784fb WatchSource:0}: Error finding container b2a23c9bac7c2e113dfced349c36d57ed547e32e16f6a946c29454f75ed784fb: Status 404 returned error can't find the container with id b2a23c9bac7c2e113dfced349c36d57ed547e32e16f6a946c29454f75ed784fb
Oct 30 01:13:48.105733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.105635    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"]
Oct 30 01:13:48.109939 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.109915    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"
Oct 30 01:13:48.133394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.133356    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"]
Oct 30 01:13:48.191478 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.191437    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-486ch\" (UniqueName: \"kubernetes.io/projected/0cbd41ec-10c0-4677-b9bf-6134b50df573-kube-api-access-486ch\") pod \"maas-api-revoke-sub-e3d775c9-79v8p\" (UID: \"0cbd41ec-10c0-4677-b9bf-6134b50df573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"
Oct 30 01:13:48.191672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.191486    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbd41ec-10c0-4677-b9bf-6134b50df573-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d775c9-79v8p\" (UID: \"0cbd41ec-10c0-4677-b9bf-6134b50df573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"
Oct 30 01:13:48.191738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.191717    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbd41ec-10c0-4677-b9bf-6134b50df573-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d775c9-79v8p\" (UID: \"0cbd41ec-10c0-4677-b9bf-6134b50df573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"
Oct 30 01:13:48.293221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.293135    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbd41ec-10c0-4677-b9bf-6134b50df573-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d775c9-79v8p\" (UID: \"0cbd41ec-10c0-4677-b9bf-6134b50df573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"
Oct 30 01:13:48.293467 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.293335    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-486ch\" (UniqueName: \"kubernetes.io/projected/0cbd41ec-10c0-4677-b9bf-6134b50df573-kube-api-access-486ch\") pod \"maas-api-revoke-sub-e3d775c9-79v8p\" (UID: \"0cbd41ec-10c0-4677-b9bf-6134b50df573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"
Oct 30 01:13:48.293467 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.293369    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbd41ec-10c0-4677-b9bf-6134b50df573-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d775c9-79v8p\" (UID: \"0cbd41ec-10c0-4677-b9bf-6134b50df573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"
Oct 30 01:13:48.294138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.294098    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbd41ec-10c0-4677-b9bf-6134b50df573-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d775c9-79v8p\" (UID: \"0cbd41ec-10c0-4677-b9bf-6134b50df573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"
Oct 30 01:13:48.295980 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.295946    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbd41ec-10c0-4677-b9bf-6134b50df573-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d775c9-79v8p\" (UID: \"0cbd41ec-10c0-4677-b9bf-6134b50df573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"
Oct 30 01:13:48.314640 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.314607    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-486ch\" (UniqueName: \"kubernetes.io/projected/0cbd41ec-10c0-4677-b9bf-6134b50df573-kube-api-access-486ch\") pod \"maas-api-revoke-sub-e3d775c9-79v8p\" (UID: \"0cbd41ec-10c0-4677-b9bf-6134b50df573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"
Oct 30 01:13:48.434995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.434910    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"
Oct 30 01:13:48.471850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.471814    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm" event={"ID":"b381e5c3-4609-4569-805e-349e19a53cce","Type":"ContainerStarted","Data":"ff82a86a964dadf2f8bb78de944a13847d6cace75069f8e1fca1d3ffe72dce1d"}
Oct 30 01:13:48.472026 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.471870    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm" event={"ID":"b381e5c3-4609-4569-805e-349e19a53cce","Type":"ContainerStarted","Data":"b2a23c9bac7c2e113dfced349c36d57ed547e32e16f6a946c29454f75ed784fb"}
Oct 30 01:13:48.512274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.509110    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm" podStartSLOduration=2.509079818 podStartE2EDuration="2.509079818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:48.50759349 +0000 UTC m=+1937.257248800" watchObservedRunningTime="2026-10-30 01:13:48.509079818 +0000 UTC m=+1937.258735128"
Oct 30 01:13:48.633049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.633004    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8" podUID="99499942-631e-4844-b1c4-1c7059fb5ece" containerName="main" containerID="cri-o://c2039bbf8b8223e6c3d2af287175b625c1b9afdf67e62c78f8aca95c98f08566" gracePeriod=45
Oct 30 01:13:48.744731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.744691    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"]
Oct 30 01:13:48.744925 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:13:48.744900    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbd41ec_10c0_4677_b9bf_6134b50df573.slice/crio-9b958e5db13043e548a151eaf2a13f404752ee1ff30f08ccc0453a2dc09eac29 WatchSource:0}: Error finding container 9b958e5db13043e548a151eaf2a13f404752ee1ff30f08ccc0453a2dc09eac29: Status 404 returned error can't find the container with id 9b958e5db13043e548a151eaf2a13f404752ee1ff30f08ccc0453a2dc09eac29
Oct 30 01:13:48.892896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:48.892864    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:13:49.004702 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.004602    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99499942-631e-4844-b1c4-1c7059fb5ece-kube-api-access-2d6l5\" (UniqueName: \"kubernetes.io/projected/99499942-631e-4844-b1c4-1c7059fb5ece-kube-api-access-2d6l5\") pod \"99499942-631e-4844-b1c4-1c7059fb5ece\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") "
Oct 30 01:13:49.004702 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.004669    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-model-cache\") pod \"99499942-631e-4844-b1c4-1c7059fb5ece\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") "
Oct 30 01:13:49.004979 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.004706    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-tmp-dir\") pod \"99499942-631e-4844-b1c4-1c7059fb5ece\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") "
Oct 30 01:13:49.004979 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.004751    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-home\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-home\") pod \"99499942-631e-4844-b1c4-1c7059fb5ece\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") "
Oct 30 01:13:49.005099 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.004971    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-model-cache" pod "99499942-631e-4844-b1c4-1c7059fb5ece" (UID: "99499942-631e-4844-b1c4-1c7059fb5ece"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:49.005099 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.004988    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-tmp-dir" pod "99499942-631e-4844-b1c4-1c7059fb5ece" (UID: "99499942-631e-4844-b1c4-1c7059fb5ece"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:49.005099 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.005002    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-dshm\") pod \"99499942-631e-4844-b1c4-1c7059fb5ece\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") "
Oct 30 01:13:49.005099 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.005066    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99499942-631e-4844-b1c4-1c7059fb5ece-tls-certs\" (UniqueName: \"kubernetes.io/secret/99499942-631e-4844-b1c4-1c7059fb5ece-tls-certs\") pod \"99499942-631e-4844-b1c4-1c7059fb5ece\" (UID: \"99499942-631e-4844-b1c4-1c7059fb5ece\") "
Oct 30 01:13:49.005336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.005145    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-home" pod "99499942-631e-4844-b1c4-1c7059fb5ece" (UID: "99499942-631e-4844-b1c4-1c7059fb5ece"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:49.005567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.005547    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:49.005567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.005568    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:49.005751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.005577    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:49.007231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.007198    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99499942-631e-4844-b1c4-1c7059fb5ece-kube-api-access-2d6l5" pod "99499942-631e-4844-b1c4-1c7059fb5ece" (UID: "99499942-631e-4844-b1c4-1c7059fb5ece"). InnerVolumeSpecName "kube-api-access-2d6l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:49.007347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.007247    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99499942-631e-4844-b1c4-1c7059fb5ece-tls-certs" pod "99499942-631e-4844-b1c4-1c7059fb5ece" (UID: "99499942-631e-4844-b1c4-1c7059fb5ece"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:49.007481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.007461    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-dshm" pod "99499942-631e-4844-b1c4-1c7059fb5ece" (UID: "99499942-631e-4844-b1c4-1c7059fb5ece"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:49.106738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.106693    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99499942-631e-4844-b1c4-1c7059fb5ece-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:49.106738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.106734    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99499942-631e-4844-b1c4-1c7059fb5ece-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:49.106973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.106751    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d6l5\" (UniqueName: \"kubernetes.io/projected/99499942-631e-4844-b1c4-1c7059fb5ece-kube-api-access-2d6l5\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:49.480163 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.480128    4425 generic.go:356] "Generic (PLEG): container finished" podID="99499942-631e-4844-b1c4-1c7059fb5ece" containerID="c2039bbf8b8223e6c3d2af287175b625c1b9afdf67e62c78f8aca95c98f08566" exitCode=0
Oct 30 01:13:49.480385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.480204    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"
Oct 30 01:13:49.480385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.480222    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8" event={"ID":"99499942-631e-4844-b1c4-1c7059fb5ece","Type":"ContainerDied","Data":"c2039bbf8b8223e6c3d2af287175b625c1b9afdf67e62c78f8aca95c98f08566"}
Oct 30 01:13:49.480385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.480265    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8" event={"ID":"99499942-631e-4844-b1c4-1c7059fb5ece","Type":"ContainerDied","Data":"600a3962de9cc690f8d07dd620b1abb038dcbd8a212f56bc9b3fca552e132b25"}
Oct 30 01:13:49.480385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.480284    4425 scope.go:122] "RemoveContainer" containerID="c2039bbf8b8223e6c3d2af287175b625c1b9afdf67e62c78f8aca95c98f08566"
Oct 30 01:13:49.482448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.482421    4425 generic.go:356] "Generic (PLEG): container finished" podID="b381e5c3-4609-4569-805e-349e19a53cce" containerID="ff82a86a964dadf2f8bb78de944a13847d6cace75069f8e1fca1d3ffe72dce1d" exitCode=0
Oct 30 01:13:49.482665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.482616    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm" event={"ID":"b381e5c3-4609-4569-805e-349e19a53cce","Type":"ContainerDied","Data":"ff82a86a964dadf2f8bb78de944a13847d6cace75069f8e1fca1d3ffe72dce1d"}
Oct 30 01:13:49.484642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.484611    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p" event={"ID":"0cbd41ec-10c0-4677-b9bf-6134b50df573","Type":"ContainerStarted","Data":"5ec95f3a138311130ba11975114b39f1133b71cb337c89ff08185d5c34270507"}
Oct 30 01:13:49.484778 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.484664    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p" event={"ID":"0cbd41ec-10c0-4677-b9bf-6134b50df573","Type":"ContainerStarted","Data":"9b958e5db13043e548a151eaf2a13f404752ee1ff30f08ccc0453a2dc09eac29"}
Oct 30 01:13:49.492286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.492234    4425 scope.go:122] "RemoveContainer" containerID="c2039bbf8b8223e6c3d2af287175b625c1b9afdf67e62c78f8aca95c98f08566"
Oct 30 01:13:49.493014 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:13:49.492990    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2039bbf8b8223e6c3d2af287175b625c1b9afdf67e62c78f8aca95c98f08566\": container with ID starting with c2039bbf8b8223e6c3d2af287175b625c1b9afdf67e62c78f8aca95c98f08566 not found: ID does not exist" containerID="c2039bbf8b8223e6c3d2af287175b625c1b9afdf67e62c78f8aca95c98f08566"
Oct 30 01:13:49.493116 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.493022    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2039bbf8b8223e6c3d2af287175b625c1b9afdf67e62c78f8aca95c98f08566"} err="failed to get container status \"c2039bbf8b8223e6c3d2af287175b625c1b9afdf67e62c78f8aca95c98f08566\": rpc error: code = NotFound desc = could not find container \"c2039bbf8b8223e6c3d2af287175b625c1b9afdf67e62c78f8aca95c98f08566\": container with ID starting with c2039bbf8b8223e6c3d2af287175b625c1b9afdf67e62c78f8aca95c98f08566 not found: ID does not exist"
Oct 30 01:13:49.564518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.564469    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"]
Oct 30 01:13:49.568716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.568675    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-ce5996/praxis-route-ce5996-kserve-6b7d54c547-84lv8"]
Oct 30 01:13:49.598209 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:49.598110    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p" podStartSLOduration=1.598088213 podStartE2EDuration="1.598088213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:49.595769211 +0000 UTC m=+1938.345424519" watchObservedRunningTime="2026-10-30 01:13:49.598088213 +0000 UTC m=+1938.347743526"
Oct 30 01:13:50.494301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:50.494257    4425 generic.go:356] "Generic (PLEG): container finished" podID="0cbd41ec-10c0-4677-b9bf-6134b50df573" containerID="5ec95f3a138311130ba11975114b39f1133b71cb337c89ff08185d5c34270507" exitCode=0
Oct 30 01:13:50.494560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:50.494344    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p" event={"ID":"0cbd41ec-10c0-4677-b9bf-6134b50df573","Type":"ContainerDied","Data":"5ec95f3a138311130ba11975114b39f1133b71cb337c89ff08185d5c34270507"}
Oct 30 01:13:50.656025 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:50.655997    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"
Oct 30 01:13:50.727710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:50.727666    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b381e5c3-4609-4569-805e-349e19a53cce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b381e5c3-4609-4569-805e-349e19a53cce-maas-api-xks-ca\") pod \"b381e5c3-4609-4569-805e-349e19a53cce\" (UID: \"b381e5c3-4609-4569-805e-349e19a53cce\") "
Oct 30 01:13:50.727937 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:50.727830    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b381e5c3-4609-4569-805e-349e19a53cce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b381e5c3-4609-4569-805e-349e19a53cce-maas-api-openshift-ca\") pod \"b381e5c3-4609-4569-805e-349e19a53cce\" (UID: \"b381e5c3-4609-4569-805e-349e19a53cce\") "
Oct 30 01:13:50.727937 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:50.727853    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b381e5c3-4609-4569-805e-349e19a53cce-kube-api-access-hc5z7\" (UniqueName: \"kubernetes.io/projected/b381e5c3-4609-4569-805e-349e19a53cce-kube-api-access-hc5z7\") pod \"b381e5c3-4609-4569-805e-349e19a53cce\" (UID: \"b381e5c3-4609-4569-805e-349e19a53cce\") "
Oct 30 01:13:50.728287 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:50.728258    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b381e5c3-4609-4569-805e-349e19a53cce-maas-api-openshift-ca" pod "b381e5c3-4609-4569-805e-349e19a53cce" (UID: "b381e5c3-4609-4569-805e-349e19a53cce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:50.728508 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:50.728489    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b381e5c3-4609-4569-805e-349e19a53cce-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:50.730238 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:50.730137    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b381e5c3-4609-4569-805e-349e19a53cce-kube-api-access-hc5z7" pod "b381e5c3-4609-4569-805e-349e19a53cce" (UID: "b381e5c3-4609-4569-805e-349e19a53cce"). InnerVolumeSpecName "kube-api-access-hc5z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:50.730520 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:50.730494    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b381e5c3-4609-4569-805e-349e19a53cce-maas-api-xks-ca" pod "b381e5c3-4609-4569-805e-349e19a53cce" (UID: "b381e5c3-4609-4569-805e-349e19a53cce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:50.830122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:50.830080    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b381e5c3-4609-4569-805e-349e19a53cce-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:50.830122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:50.830117    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc5z7\" (UniqueName: \"kubernetes.io/projected/b381e5c3-4609-4569-805e-349e19a53cce-kube-api-access-hc5z7\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:51.423395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:51.423356    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99499942-631e-4844-b1c4-1c7059fb5ece" path="/var/lib/kubelet/pods/99499942-631e-4844-b1c4-1c7059fb5ece/volumes"
Oct 30 01:13:51.504718 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:51.504683    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"
Oct 30 01:13:51.504919 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:51.504712    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm" event={"ID":"b381e5c3-4609-4569-805e-349e19a53cce","Type":"ContainerDied","Data":"b2a23c9bac7c2e113dfced349c36d57ed547e32e16f6a946c29454f75ed784fb"}
Oct 30 01:13:51.504919 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:51.504762    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2a23c9bac7c2e113dfced349c36d57ed547e32e16f6a946c29454f75ed784fb"
Oct 30 01:13:51.671550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:51.671514    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"
Oct 30 01:13:51.843003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:51.842964    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cbd41ec-10c0-4677-b9bf-6134b50df573-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbd41ec-10c0-4677-b9bf-6134b50df573-maas-api-openshift-ca\") pod \"0cbd41ec-10c0-4677-b9bf-6134b50df573\" (UID: \"0cbd41ec-10c0-4677-b9bf-6134b50df573\") "
Oct 30 01:13:51.844652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:51.844609    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cbd41ec-10c0-4677-b9bf-6134b50df573-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbd41ec-10c0-4677-b9bf-6134b50df573-maas-api-xks-ca\") pod \"0cbd41ec-10c0-4677-b9bf-6134b50df573\" (UID: \"0cbd41ec-10c0-4677-b9bf-6134b50df573\") "
Oct 30 01:13:51.844908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:51.844657    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbd41ec-10c0-4677-b9bf-6134b50df573-maas-api-openshift-ca" pod "0cbd41ec-10c0-4677-b9bf-6134b50df573" (UID: "0cbd41ec-10c0-4677-b9bf-6134b50df573"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:13:51.844908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:51.844772    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbd41ec-10c0-4677-b9bf-6134b50df573-kube-api-access-486ch\" (UniqueName: \"kubernetes.io/projected/0cbd41ec-10c0-4677-b9bf-6134b50df573-kube-api-access-486ch\") pod \"0cbd41ec-10c0-4677-b9bf-6134b50df573\" (UID: \"0cbd41ec-10c0-4677-b9bf-6134b50df573\") "
Oct 30 01:13:51.845459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:51.845430    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cbd41ec-10c0-4677-b9bf-6134b50df573-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:51.854426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:51.853615    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cbd41ec-10c0-4677-b9bf-6134b50df573-maas-api-xks-ca" pod "0cbd41ec-10c0-4677-b9bf-6134b50df573" (UID: "0cbd41ec-10c0-4677-b9bf-6134b50df573"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:51.854426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:51.853792    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbd41ec-10c0-4677-b9bf-6134b50df573-kube-api-access-486ch" pod "0cbd41ec-10c0-4677-b9bf-6134b50df573" (UID: "0cbd41ec-10c0-4677-b9bf-6134b50df573"). InnerVolumeSpecName "kube-api-access-486ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:51.946920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:51.946878    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cbd41ec-10c0-4677-b9bf-6134b50df573-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:51.946920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:51.946914    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-486ch\" (UniqueName: \"kubernetes.io/projected/0cbd41ec-10c0-4677-b9bf-6134b50df573-kube-api-access-486ch\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:13:52.514273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:52.514228    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p" event={"ID":"0cbd41ec-10c0-4677-b9bf-6134b50df573","Type":"ContainerDied","Data":"9b958e5db13043e548a151eaf2a13f404752ee1ff30f08ccc0453a2dc09eac29"}
Oct 30 01:13:52.514273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:52.514259    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"
Oct 30 01:13:52.514474 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:52.514297    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b958e5db13043e548a151eaf2a13f404752ee1ff30f08ccc0453a2dc09eac29"
Oct 30 01:13:53.382423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.382369    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"]
Oct 30 01:13:53.393516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.392565    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b381e5c3-4609-4569-805e-349e19a53cce" containerName="revoke-keys"
Oct 30 01:13:53.393516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.392596    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b381e5c3-4609-4569-805e-349e19a53cce" containerName="revoke-keys"
Oct 30 01:13:53.393516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.392619    4425 cpu_manager.go:414] "containerMap: removing container" podUID="0cbd41ec-10c0-4677-b9bf-6134b50df573" containerName="revoke-keys"
Oct 30 01:13:53.393516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.392627    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbd41ec-10c0-4677-b9bf-6134b50df573" containerName="revoke-keys"
Oct 30 01:13:53.393516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.392647    4425 cpu_manager.go:414] "containerMap: removing container" podUID="99499942-631e-4844-b1c4-1c7059fb5ece" containerName="main"
Oct 30 01:13:53.393516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.392655    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99499942-631e-4844-b1c4-1c7059fb5ece" containerName="main"
Oct 30 01:13:53.393516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.392789    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b381e5c3-4609-4569-805e-349e19a53cce" containerName="revoke-keys"
Oct 30 01:13:53.393516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.392802    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbd41ec-10c0-4677-b9bf-6134b50df573" containerName="revoke-keys"
Oct 30 01:13:53.393516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.392815    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="99499942-631e-4844-b1c4-1c7059fb5ece" containerName="main"
Oct 30 01:13:53.398456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.397908    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.420521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.413732    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"]
Oct 30 01:13:53.575828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.575787    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45r8r\" (UniqueName: \"kubernetes.io/projected/2591e8d9-706f-4362-816a-14376d116652-kube-api-access-45r8r\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.576044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.575950    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2591e8d9-706f-4362-816a-14376d116652-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.576044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.575991    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2591e8d9-706f-4362-816a-14376d116652-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.576208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.576184    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.576271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.576234    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.576313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.576303    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.576376 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.576358    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.576418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.576405    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2591e8d9-706f-4362-816a-14376d116652-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.576459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.576443    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.677689 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.677573    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2591e8d9-706f-4362-816a-14376d116652-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.677689 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.677621    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2591e8d9-706f-4362-816a-14376d116652-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.677960 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.677723    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.677960 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.677758    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.677960 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.677814    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.677960 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.677886    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.677960 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.677917    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2591e8d9-706f-4362-816a-14376d116652-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.677960 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.677941    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.678290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.678031    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45r8r\" (UniqueName: \"kubernetes.io/projected/2591e8d9-706f-4362-816a-14376d116652-kube-api-access-45r8r\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.678290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.678249    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.678406 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.678366    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.678459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.678414    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.678604 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.678580    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.678857 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.678832    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2591e8d9-706f-4362-816a-14376d116652-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.680506 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.680484    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2591e8d9-706f-4362-816a-14376d116652-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.680601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.680529    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.702788 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.702749    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45r8r\" (UniqueName: \"kubernetes.io/projected/2591e8d9-706f-4362-816a-14376d116652-kube-api-access-45r8r\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.702788 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.702746    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2591e8d9-706f-4362-816a-14376d116652-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:53.724794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:53.724753    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:54.061769 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:54.061731    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"]
Oct 30 01:13:54.069676 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:54.069633    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:54.069835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:54.069731    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:54.069835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:54.069778    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:13:54.537513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:54.536276    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4" event={"ID":"2591e8d9-706f-4362-816a-14376d116652","Type":"ContainerStarted","Data":"54c7fb25f4e17215a36f6ce236ab883a75301bcf905c5c7f752695a9ff64917f"}
Oct 30 01:13:54.537513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:54.536351    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4" event={"ID":"2591e8d9-706f-4362-816a-14376d116652","Type":"ContainerStarted","Data":"bcdc0ef99c03b366e29ba7506e02477a64248882e8597ab2b14347a5ddc6e726"}
Oct 30 01:13:54.604931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:54.604870    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4" podStartSLOduration=1.604854028 podStartE2EDuration="1.604854028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:54.592795304 +0000 UTC m=+1943.342450636" watchObservedRunningTime="2026-10-30 01:13:54.604854028 +0000 UTC m=+1943.354509394"
Oct 30 01:13:54.725351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:54.725293    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:13:55.730182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:55.729885    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 01:13:55.730182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:55.729948    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4" podUID="2591e8d9-706f-4362-816a-14376d116652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 01:13:56.725386 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:56.725319    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 01:13:56.725703 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:56.725669    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4" podUID="2591e8d9-706f-4362-816a-14376d116652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 01:13:57.726007 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:57.725953    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 01:13:57.726511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:57.726039    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4" podUID="2591e8d9-706f-4362-816a-14376d116652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 01:13:58.262181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.262119    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"]
Oct 30 01:13:58.280485 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.280450    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"
Oct 30 01:13:58.284041 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.284002    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"]
Oct 30 01:13:58.294891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.293767    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:13:58.303494 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.303455    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:13:58.303647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.303507    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:13:58.303647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.303518    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:13:58.421616 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.421580    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2b4db3-0cca-4726-ab42-a62d7337034a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422de6cd-d5f5w\" (UID: \"5d2b4db3-0cca-4726-ab42-a62d7337034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"
Oct 30 01:13:58.421828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.421653    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcq8k\" (UniqueName: \"kubernetes.io/projected/5d2b4db3-0cca-4726-ab42-a62d7337034a-kube-api-access-jcq8k\") pod \"maas-api-revoke-sub-422de6cd-d5f5w\" (UID: \"5d2b4db3-0cca-4726-ab42-a62d7337034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"
Oct 30 01:13:58.421897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.421818    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2b4db3-0cca-4726-ab42-a62d7337034a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422de6cd-d5f5w\" (UID: \"5d2b4db3-0cca-4726-ab42-a62d7337034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"
Oct 30 01:13:58.523730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.523631    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2b4db3-0cca-4726-ab42-a62d7337034a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422de6cd-d5f5w\" (UID: \"5d2b4db3-0cca-4726-ab42-a62d7337034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"
Oct 30 01:13:58.523948 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.523807    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2b4db3-0cca-4726-ab42-a62d7337034a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422de6cd-d5f5w\" (UID: \"5d2b4db3-0cca-4726-ab42-a62d7337034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"
Oct 30 01:13:58.523948 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.523843    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcq8k\" (UniqueName: \"kubernetes.io/projected/5d2b4db3-0cca-4726-ab42-a62d7337034a-kube-api-access-jcq8k\") pod \"maas-api-revoke-sub-422de6cd-d5f5w\" (UID: \"5d2b4db3-0cca-4726-ab42-a62d7337034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"
Oct 30 01:13:58.524546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.524517    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2b4db3-0cca-4726-ab42-a62d7337034a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-422de6cd-d5f5w\" (UID: \"5d2b4db3-0cca-4726-ab42-a62d7337034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"
Oct 30 01:13:58.526496 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.526468    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2b4db3-0cca-4726-ab42-a62d7337034a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-422de6cd-d5f5w\" (UID: \"5d2b4db3-0cca-4726-ab42-a62d7337034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"
Oct 30 01:13:58.540083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.540047    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"]
Oct 30 01:13:58.544429 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.544408    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"
Oct 30 01:13:58.546106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.546078    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcq8k\" (UniqueName: \"kubernetes.io/projected/5d2b4db3-0cca-4726-ab42-a62d7337034a-kube-api-access-jcq8k\") pod \"maas-api-revoke-sub-422de6cd-d5f5w\" (UID: \"5d2b4db3-0cca-4726-ab42-a62d7337034a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"
Oct 30 01:13:58.592340 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.592305    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"]
Oct 30 01:13:58.600254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.600224    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"
Oct 30 01:13:58.726006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.725969    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 01:13:58.726216 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.726025    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4" podUID="2591e8d9-706f-4362-816a-14376d116652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 01:13:58.726593 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.726300    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-466885d2-5rgwq\" (UID: \"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"
Oct 30 01:13:58.726593 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.726343    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-466885d2-5rgwq\" (UID: \"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"
Oct 30 01:13:58.726593 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.726496    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwcfc\" (UniqueName: \"kubernetes.io/projected/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-kube-api-access-kwcfc\") pod \"maas-api-revoke-sub-466885d2-5rgwq\" (UID: \"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"
Oct 30 01:13:58.828227 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.828088    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwcfc\" (UniqueName: \"kubernetes.io/projected/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-kube-api-access-kwcfc\") pod \"maas-api-revoke-sub-466885d2-5rgwq\" (UID: \"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"
Oct 30 01:13:58.828425 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.828288    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-466885d2-5rgwq\" (UID: \"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"
Oct 30 01:13:58.828425 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.828323    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-466885d2-5rgwq\" (UID: \"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"
Oct 30 01:13:58.829122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.829097    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-466885d2-5rgwq\" (UID: \"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"
Oct 30 01:13:58.830993 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.830965    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-466885d2-5rgwq\" (UID: \"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"
Oct 30 01:13:58.849896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.849857    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwcfc\" (UniqueName: \"kubernetes.io/projected/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-kube-api-access-kwcfc\") pod \"maas-api-revoke-sub-466885d2-5rgwq\" (UID: \"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"
Oct 30 01:13:58.902619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.902576    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"
Oct 30 01:13:58.944839 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:58.944797    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"]
Oct 30 01:13:58.946363 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:13:58.946335    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2b4db3_0cca_4726_ab42_a62d7337034a.slice/crio-fddf8798d77621bf5d1ce50adc548d281c38698cfa80aa90706905c0a25b9ba2 WatchSource:0}: Error finding container fddf8798d77621bf5d1ce50adc548d281c38698cfa80aa90706905c0a25b9ba2: Status 404 returned error can't find the container with id fddf8798d77621bf5d1ce50adc548d281c38698cfa80aa90706905c0a25b9ba2
Oct 30 01:13:59.244857 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:59.244695    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"]
Oct 30 01:13:59.698826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:59.697826    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq" event={"ID":"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e","Type":"ContainerStarted","Data":"6c08741e2d913ee219b74ef27004bdcb56cec68dbfb10c041f095fe95ef94de4"}
Oct 30 01:13:59.698826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:59.697905    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq" event={"ID":"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e","Type":"ContainerStarted","Data":"177fc0a20cd127d40cfed5787e0b023195cff8a3de88696c8a566107066f527f"}
Oct 30 01:13:59.700091 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:59.700058    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w" event={"ID":"5d2b4db3-0cca-4726-ab42-a62d7337034a","Type":"ContainerStarted","Data":"b099bd53bd0c97cb0081631658ef32c40ec80d7f2cec20c9b4ed4ba10838ea60"}
Oct 30 01:13:59.700230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:59.700111    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w" event={"ID":"5d2b4db3-0cca-4726-ab42-a62d7337034a","Type":"ContainerStarted","Data":"fddf8798d77621bf5d1ce50adc548d281c38698cfa80aa90706905c0a25b9ba2"}
Oct 30 01:13:59.725578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:59.725535    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 01:13:59.725809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:59.725608    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4" podUID="2591e8d9-706f-4362-816a-14376d116652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 01:13:59.794233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:59.789231    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq" podStartSLOduration=1.789204286 podStartE2EDuration="1.789204286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:59.745428757 +0000 UTC m=+1948.495084077" watchObservedRunningTime="2026-10-30 01:13:59.789204286 +0000 UTC m=+1948.538859596"
Oct 30 01:13:59.824609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:13:59.824535    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w" podStartSLOduration=1.824513262 podStartE2EDuration="1.824513262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:59.819674992 +0000 UTC m=+1948.569330303" watchObservedRunningTime="2026-10-30 01:13:59.824513262 +0000 UTC m=+1948.574168573"
Oct 30 01:14:00.710056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:00.709942    4425 generic.go:356] "Generic (PLEG): container finished" podID="a8ac87d8-e7ed-4f22-98c8-798328b4cb1e" containerID="6c08741e2d913ee219b74ef27004bdcb56cec68dbfb10c041f095fe95ef94de4" exitCode=0
Oct 30 01:14:00.710056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:00.710020    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq" event={"ID":"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e","Type":"ContainerDied","Data":"6c08741e2d913ee219b74ef27004bdcb56cec68dbfb10c041f095fe95ef94de4"}
Oct 30 01:14:00.711919 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:00.711887    4425 generic.go:356] "Generic (PLEG): container finished" podID="5d2b4db3-0cca-4726-ab42-a62d7337034a" containerID="b099bd53bd0c97cb0081631658ef32c40ec80d7f2cec20c9b4ed4ba10838ea60" exitCode=0
Oct 30 01:14:00.712091 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:00.711966    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w" event={"ID":"5d2b4db3-0cca-4726-ab42-a62d7337034a","Type":"ContainerDied","Data":"b099bd53bd0c97cb0081631658ef32c40ec80d7f2cec20c9b4ed4ba10838ea60"}
Oct 30 01:14:00.725721 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:00.725679    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 01:14:00.725971 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:00.725750    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4" podUID="2591e8d9-706f-4362-816a-14376d116652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 01:14:01.725401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:01.725365    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 01:14:01.725875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:01.725432    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4" podUID="2591e8d9-706f-4362-816a-14376d116652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 01:14:01.992731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:01.992701    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"
Oct 30 01:14:01.999050 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:01.999014    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"
Oct 30 01:14:02.017366 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.017322    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d2b4db3-0cca-4726-ab42-a62d7337034a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2b4db3-0cca-4726-ab42-a62d7337034a-maas-api-xks-ca\") pod \"5d2b4db3-0cca-4726-ab42-a62d7337034a\" (UID: \"5d2b4db3-0cca-4726-ab42-a62d7337034a\") "
Oct 30 01:14:02.017574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.017417    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d2b4db3-0cca-4726-ab42-a62d7337034a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2b4db3-0cca-4726-ab42-a62d7337034a-maas-api-openshift-ca\") pod \"5d2b4db3-0cca-4726-ab42-a62d7337034a\" (UID: \"5d2b4db3-0cca-4726-ab42-a62d7337034a\") "
Oct 30 01:14:02.017574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.017491    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-kube-api-access-kwcfc\" (UniqueName: \"kubernetes.io/projected/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-kube-api-access-kwcfc\") pod \"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e\" (UID: \"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e\") "
Oct 30 01:14:02.017574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.017527    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d2b4db3-0cca-4726-ab42-a62d7337034a-kube-api-access-jcq8k\" (UniqueName: \"kubernetes.io/projected/5d2b4db3-0cca-4726-ab42-a62d7337034a-kube-api-access-jcq8k\") pod \"5d2b4db3-0cca-4726-ab42-a62d7337034a\" (UID: \"5d2b4db3-0cca-4726-ab42-a62d7337034a\") "
Oct 30 01:14:02.017757 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.017645    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-maas-api-openshift-ca\") pod \"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e\" (UID: \"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e\") "
Oct 30 01:14:02.017757 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.017737    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-maas-api-xks-ca\") pod \"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e\" (UID: \"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e\") "
Oct 30 01:14:02.018106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.017940    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d2b4db3-0cca-4726-ab42-a62d7337034a-maas-api-openshift-ca" pod "5d2b4db3-0cca-4726-ab42-a62d7337034a" (UID: "5d2b4db3-0cca-4726-ab42-a62d7337034a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:02.018453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.018423    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-maas-api-openshift-ca" pod "a8ac87d8-e7ed-4f22-98c8-798328b4cb1e" (UID: "a8ac87d8-e7ed-4f22-98c8-798328b4cb1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:02.018565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.018521    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2b4db3-0cca-4726-ab42-a62d7337034a-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:14:02.020610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.020579    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d2b4db3-0cca-4726-ab42-a62d7337034a-maas-api-xks-ca" pod "5d2b4db3-0cca-4726-ab42-a62d7337034a" (UID: "5d2b4db3-0cca-4726-ab42-a62d7337034a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:02.020723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.020665    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-maas-api-xks-ca" pod "a8ac87d8-e7ed-4f22-98c8-798328b4cb1e" (UID: "a8ac87d8-e7ed-4f22-98c8-798328b4cb1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:02.021114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.021077    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-kube-api-access-kwcfc" pod "a8ac87d8-e7ed-4f22-98c8-798328b4cb1e" (UID: "a8ac87d8-e7ed-4f22-98c8-798328b4cb1e"). InnerVolumeSpecName "kube-api-access-kwcfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:02.021219 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.021200    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2b4db3-0cca-4726-ab42-a62d7337034a-kube-api-access-jcq8k" pod "5d2b4db3-0cca-4726-ab42-a62d7337034a" (UID: "5d2b4db3-0cca-4726-ab42-a62d7337034a"). InnerVolumeSpecName "kube-api-access-jcq8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:02.119690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.119645    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:14:02.119690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.119689    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2b4db3-0cca-4726-ab42-a62d7337034a-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:14:02.119971 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.119704    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwcfc\" (UniqueName: \"kubernetes.io/projected/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-kube-api-access-kwcfc\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:14:02.119971 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.119716    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcq8k\" (UniqueName: \"kubernetes.io/projected/5d2b4db3-0cca-4726-ab42-a62d7337034a-kube-api-access-jcq8k\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:14:02.119971 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.119730    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:14:02.730106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.730078    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"
Oct 30 01:14:02.730652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.730107    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq" event={"ID":"a8ac87d8-e7ed-4f22-98c8-798328b4cb1e","Type":"ContainerDied","Data":"177fc0a20cd127d40cfed5787e0b023195cff8a3de88696c8a566107066f527f"}
Oct 30 01:14:02.730652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.730199    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="177fc0a20cd127d40cfed5787e0b023195cff8a3de88696c8a566107066f527f"
Oct 30 01:14:02.730795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.730744    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:14:02.731196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.731146    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:14:02.732067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.732046    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"
Oct 30 01:14:02.732194 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.732042    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w" event={"ID":"5d2b4db3-0cca-4726-ab42-a62d7337034a","Type":"ContainerDied","Data":"fddf8798d77621bf5d1ce50adc548d281c38698cfa80aa90706905c0a25b9ba2"}
Oct 30 01:14:02.732281 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.732198    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fddf8798d77621bf5d1ce50adc548d281c38698cfa80aa90706905c0a25b9ba2"
Oct 30 01:14:02.732325 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:02.732293    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:14:05.930271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:05.930222    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"]
Oct 30 01:14:05.930844 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:05.930825    4425 cpu_manager.go:414] "containerMap: removing container" podUID="5d2b4db3-0cca-4726-ab42-a62d7337034a" containerName="revoke-keys"
Oct 30 01:14:05.930909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:05.930845    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2b4db3-0cca-4726-ab42-a62d7337034a" containerName="revoke-keys"
Oct 30 01:14:05.930909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:05.930880    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a8ac87d8-e7ed-4f22-98c8-798328b4cb1e" containerName="revoke-keys"
Oct 30 01:14:05.930909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:05.930886    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ac87d8-e7ed-4f22-98c8-798328b4cb1e" containerName="revoke-keys"
Oct 30 01:14:05.931011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:05.930968    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ac87d8-e7ed-4f22-98c8-798328b4cb1e" containerName="revoke-keys"
Oct 30 01:14:05.931011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:05.930980    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d2b4db3-0cca-4726-ab42-a62d7337034a" containerName="revoke-keys"
Oct 30 01:14:05.935728 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:05.935705    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"
Oct 30 01:14:05.960012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:05.959975    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-0e1a98-dockercfg-gsfkp\""
Oct 30 01:14:05.960251 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:05.960059    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-0e1a98\""
Oct 30 01:14:05.966827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:05.966779    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"]
Oct 30 01:14:05.970847 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:05.970814    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpxxk\" (UniqueName: \"kubernetes.io/projected/aaf97e80-fd90-4db8-84a9-872db881b7c7-kube-api-access-bpxxk\") pod \"payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl\" (UID: \"aaf97e80-fd90-4db8-84a9-872db881b7c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"
Oct 30 01:14:05.970981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:05.970863    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf97e80-fd90-4db8-84a9-872db881b7c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl\" (UID: \"aaf97e80-fd90-4db8-84a9-872db881b7c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"
Oct 30 01:14:06.059203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.056697    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"]
Oct 30 01:14:06.078920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.078879    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"
Oct 30 01:14:06.095417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.085247    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpxxk\" (UniqueName: \"kubernetes.io/projected/aaf97e80-fd90-4db8-84a9-872db881b7c7-kube-api-access-bpxxk\") pod \"payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl\" (UID: \"aaf97e80-fd90-4db8-84a9-872db881b7c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"
Oct 30 01:14:06.095417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.085307    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf97e80-fd90-4db8-84a9-872db881b7c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl\" (UID: \"aaf97e80-fd90-4db8-84a9-872db881b7c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"
Oct 30 01:14:06.095417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.086553    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf97e80-fd90-4db8-84a9-872db881b7c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl\" (UID: \"aaf97e80-fd90-4db8-84a9-872db881b7c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"
Oct 30 01:14:06.128336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.128299    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"]
Oct 30 01:14:06.133412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.133371    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpxxk\" (UniqueName: \"kubernetes.io/projected/aaf97e80-fd90-4db8-84a9-872db881b7c7-kube-api-access-bpxxk\") pod \"payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl\" (UID: \"aaf97e80-fd90-4db8-84a9-872db881b7c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"
Oct 30 01:14:06.186978 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.186871    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36566e9-c18e-4026-b593-467e3491e95a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl\" (UID: \"d36566e9-c18e-4026-b593-467e3491e95a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"
Oct 30 01:14:06.186978 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.186938    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47slg\" (UniqueName: \"kubernetes.io/projected/d36566e9-c18e-4026-b593-467e3491e95a-kube-api-access-47slg\") pod \"payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl\" (UID: \"d36566e9-c18e-4026-b593-467e3491e95a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"
Oct 30 01:14:06.250429 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.250389    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"
Oct 30 01:14:06.288547 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.288508    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36566e9-c18e-4026-b593-467e3491e95a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl\" (UID: \"d36566e9-c18e-4026-b593-467e3491e95a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"
Oct 30 01:14:06.288705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.288571    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47slg\" (UniqueName: \"kubernetes.io/projected/d36566e9-c18e-4026-b593-467e3491e95a-kube-api-access-47slg\") pod \"payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl\" (UID: \"d36566e9-c18e-4026-b593-467e3491e95a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"
Oct 30 01:14:06.289640 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.289614    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36566e9-c18e-4026-b593-467e3491e95a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl\" (UID: \"d36566e9-c18e-4026-b593-467e3491e95a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"
Oct 30 01:14:06.312457 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.312423    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47slg\" (UniqueName: \"kubernetes.io/projected/d36566e9-c18e-4026-b593-467e3491e95a-kube-api-access-47slg\") pod \"payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl\" (UID: \"d36566e9-c18e-4026-b593-467e3491e95a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"
Oct 30 01:14:06.412361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.412318    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"
Oct 30 01:14:06.571882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.571844    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"]
Oct 30 01:14:06.572532 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:14:06.572507    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf97e80_fd90_4db8_84a9_872db881b7c7.slice/crio-52f874fc487a54b9a8cc72a6a6200fd157e47c9f9387c7ef0cce38799adafa72 WatchSource:0}: Error finding container 52f874fc487a54b9a8cc72a6a6200fd157e47c9f9387c7ef0cce38799adafa72: Status 404 returned error can't find the container with id 52f874fc487a54b9a8cc72a6a6200fd157e47c9f9387c7ef0cce38799adafa72
Oct 30 01:14:06.725412 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:14:06.725368    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36566e9_c18e_4026_b593_467e3491e95a.slice/crio-b56fed4ede55e3c54fdb6c45d0f0a24231dc4d0bea86320b0477682234341faf WatchSource:0}: Error finding container b56fed4ede55e3c54fdb6c45d0f0a24231dc4d0bea86320b0477682234341faf: Status 404 returned error can't find the container with id b56fed4ede55e3c54fdb6c45d0f0a24231dc4d0bea86320b0477682234341faf
Oct 30 01:14:06.725975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.725946    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"]
Oct 30 01:14:06.765452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.765409    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl" event={"ID":"aaf97e80-fd90-4db8-84a9-872db881b7c7","Type":"ContainerStarted","Data":"4215f23058a3c9b739702d1e19f7dab71b8d0e39e2d76dd72a2db762b9ada5d0"}
Oct 30 01:14:06.765632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.765491    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"
Oct 30 01:14:06.765632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.765516    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl" event={"ID":"aaf97e80-fd90-4db8-84a9-872db881b7c7","Type":"ContainerStarted","Data":"52f874fc487a54b9a8cc72a6a6200fd157e47c9f9387c7ef0cce38799adafa72"}
Oct 30 01:14:06.767112 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.767072    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl" event={"ID":"d36566e9-c18e-4026-b593-467e3491e95a","Type":"ContainerStarted","Data":"b56fed4ede55e3c54fdb6c45d0f0a24231dc4d0bea86320b0477682234341faf"}
Oct 30 01:14:06.806526 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:06.806467    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl" podStartSLOduration=1.806448897 podStartE2EDuration="1.806448897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:06.800833085 +0000 UTC m=+1955.550488393" watchObservedRunningTime="2026-10-30 01:14:06.806448897 +0000 UTC m=+1955.556104206"
Oct 30 01:14:07.558516 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:14:07.558471    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:14:07.776416 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:07.776379    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl" event={"ID":"d36566e9-c18e-4026-b593-467e3491e95a","Type":"ContainerStarted","Data":"081cd735a10af86c406dbe3edfd6575895f2b2247ce78d6a63233522af925a42"}
Oct 30 01:14:07.776581 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:07.776484    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"
Oct 30 01:14:07.812531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:07.812412    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl" podStartSLOduration=2.812396339 podStartE2EDuration="2.812396339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:07.811926277 +0000 UTC m=+1956.561581596" watchObservedRunningTime="2026-10-30 01:14:07.812396339 +0000 UTC m=+1956.562051648"
Oct 30 01:14:08.881036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:08.880994    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"]
Oct 30 01:14:08.888188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:08.888132    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d3c1fa-64gpz"]
Oct 30 01:14:09.423779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:09.423744    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6" path="/var/lib/kubelet/pods/b64d7b5c-fdf1-4942-ba3a-b13edc03f4b6/volumes"
Oct 30 01:14:16.710594 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:16.710553    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"]
Oct 30 01:14:16.716474 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:16.716442    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"
Oct 30 01:14:16.737366 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:16.737326    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"]
Oct 30 01:14:16.741862 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:16.741836    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-8ae41e\""
Oct 30 01:14:16.742024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:16.741956    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-8ae41e-dockercfg-g7pmw\""
Oct 30 01:14:16.796009 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:16.795937    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"]
Oct 30 01:14:16.824177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:16.804044    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"
Oct 30 01:14:16.844206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:16.844144    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c46fb47-d71a-4831-bf35-2acc1bb4c935-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g\" (UID: \"4c46fb47-d71a-4831-bf35-2acc1bb4c935\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"
Oct 30 01:14:16.844427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:16.844382    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thzgx\" (UniqueName: \"kubernetes.io/projected/4c46fb47-d71a-4831-bf35-2acc1bb4c935-kube-api-access-thzgx\") pod \"payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g\" (UID: \"4c46fb47-d71a-4831-bf35-2acc1bb4c935\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"
Oct 30 01:14:16.855182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:16.851545    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"]
Oct 30 01:14:16.946047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:16.945996    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gdg8\" (UniqueName: \"kubernetes.io/projected/3ac3ee9c-d077-44ac-8be5-ed5555679412-kube-api-access-2gdg8\") pod \"payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x\" (UID: \"3ac3ee9c-d077-44ac-8be5-ed5555679412\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"
Oct 30 01:14:16.946294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:16.946172    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c46fb47-d71a-4831-bf35-2acc1bb4c935-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g\" (UID: \"4c46fb47-d71a-4831-bf35-2acc1bb4c935\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"
Oct 30 01:14:16.946377 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:16.946295    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac3ee9c-d077-44ac-8be5-ed5555679412-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x\" (UID: \"3ac3ee9c-d077-44ac-8be5-ed5555679412\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"
Oct 30 01:14:16.946377 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:16.946331    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thzgx\" (UniqueName: \"kubernetes.io/projected/4c46fb47-d71a-4831-bf35-2acc1bb4c935-kube-api-access-thzgx\") pod \"payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g\" (UID: \"4c46fb47-d71a-4831-bf35-2acc1bb4c935\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"
Oct 30 01:14:16.947537 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:16.947509    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c46fb47-d71a-4831-bf35-2acc1bb4c935-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g\" (UID: \"4c46fb47-d71a-4831-bf35-2acc1bb4c935\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"
Oct 30 01:14:16.970342 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:16.969870    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thzgx\" (UniqueName: \"kubernetes.io/projected/4c46fb47-d71a-4831-bf35-2acc1bb4c935-kube-api-access-thzgx\") pod \"payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g\" (UID: \"4c46fb47-d71a-4831-bf35-2acc1bb4c935\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"
Oct 30 01:14:17.032902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:17.032857    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"
Oct 30 01:14:17.048024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:17.047987    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac3ee9c-d077-44ac-8be5-ed5555679412-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x\" (UID: \"3ac3ee9c-d077-44ac-8be5-ed5555679412\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"
Oct 30 01:14:17.048229 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:17.048094    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gdg8\" (UniqueName: \"kubernetes.io/projected/3ac3ee9c-d077-44ac-8be5-ed5555679412-kube-api-access-2gdg8\") pod \"payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x\" (UID: \"3ac3ee9c-d077-44ac-8be5-ed5555679412\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"
Oct 30 01:14:17.049619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:17.049586    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac3ee9c-d077-44ac-8be5-ed5555679412-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x\" (UID: \"3ac3ee9c-d077-44ac-8be5-ed5555679412\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"
Oct 30 01:14:17.076106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:17.076069    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gdg8\" (UniqueName: \"kubernetes.io/projected/3ac3ee9c-d077-44ac-8be5-ed5555679412-kube-api-access-2gdg8\") pod \"payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x\" (UID: \"3ac3ee9c-d077-44ac-8be5-ed5555679412\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"
Oct 30 01:14:17.169371 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:17.153082    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"
Oct 30 01:14:17.359308 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:14:17.359264    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c46fb47_d71a_4831_bf35_2acc1bb4c935.slice/crio-fc3d730dc1d55f6b427daefce85514a71a88288434aacf885eda40ec6ad8820e WatchSource:0}: Error finding container fc3d730dc1d55f6b427daefce85514a71a88288434aacf885eda40ec6ad8820e: Status 404 returned error can't find the container with id fc3d730dc1d55f6b427daefce85514a71a88288434aacf885eda40ec6ad8820e
Oct 30 01:14:17.360737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:17.360702    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"]
Oct 30 01:14:17.506021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:17.505987    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"]
Oct 30 01:14:17.507124 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:14:17.507089    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac3ee9c_d077_44ac_8be5_ed5555679412.slice/crio-0e2737fcac31df6faab28c081e7650ac702a7a22a0620a4f02fcd461fe29953f WatchSource:0}: Error finding container 0e2737fcac31df6faab28c081e7650ac702a7a22a0620a4f02fcd461fe29953f: Status 404 returned error can't find the container with id 0e2737fcac31df6faab28c081e7650ac702a7a22a0620a4f02fcd461fe29953f
Oct 30 01:14:17.780339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:17.780299    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"
Oct 30 01:14:17.894880 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:17.894825    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g" event={"ID":"4c46fb47-d71a-4831-bf35-2acc1bb4c935","Type":"ContainerStarted","Data":"7441f9119204455033681f3e0b544d35642fdee66bcf70e330c206b9a16bed77"}
Oct 30 01:14:17.895778 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:17.895751    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"
Oct 30 01:14:17.896012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:17.895984    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g" event={"ID":"4c46fb47-d71a-4831-bf35-2acc1bb4c935","Type":"ContainerStarted","Data":"fc3d730dc1d55f6b427daefce85514a71a88288434aacf885eda40ec6ad8820e"}
Oct 30 01:14:17.898576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:17.898546    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x" event={"ID":"3ac3ee9c-d077-44ac-8be5-ed5555679412","Type":"ContainerStarted","Data":"53ef72fdb7e25171fd039f365071887731d6e12a9ddf56266e5a8ab2dff03f9f"}
Oct 30 01:14:17.898710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:17.898602    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x" event={"ID":"3ac3ee9c-d077-44ac-8be5-ed5555679412","Type":"ContainerStarted","Data":"0e2737fcac31df6faab28c081e7650ac702a7a22a0620a4f02fcd461fe29953f"}
Oct 30 01:14:17.898710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:17.898638    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"
Oct 30 01:14:17.935666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:17.935598    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g" podStartSLOduration=1.9355795009999999 podStartE2EDuration="1.935579501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:17.930767903 +0000 UTC m=+1966.680423211" watchObservedRunningTime="2026-10-30 01:14:17.935579501 +0000 UTC m=+1966.685234809"
Oct 30 01:14:17.975346 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:17.975284    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x" podStartSLOduration=1.975266827 podStartE2EDuration="1.975266827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:17.968516975 +0000 UTC m=+1966.718172284" watchObservedRunningTime="2026-10-30 01:14:17.975266827 +0000 UTC m=+1966.724922143"
Oct 30 01:14:18.310952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:18.310910    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"]
Oct 30 01:14:18.316322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:18.316270    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"
Oct 30 01:14:18.330000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:18.329968    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-df4db1\""
Oct 30 01:14:18.339039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:18.339004    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"]
Oct 30 01:14:18.474533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:18.474487    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1addf2b0-5855-4a72-acba-a10a5bcc5ec0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx\" (UID: \"1addf2b0-5855-4a72-acba-a10a5bcc5ec0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"
Oct 30 01:14:18.474533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:18.474541    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmnm7\" (UniqueName: \"kubernetes.io/projected/1addf2b0-5855-4a72-acba-a10a5bcc5ec0-kube-api-access-wmnm7\") pod \"payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx\" (UID: \"1addf2b0-5855-4a72-acba-a10a5bcc5ec0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"
Oct 30 01:14:18.587240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:18.577070    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1addf2b0-5855-4a72-acba-a10a5bcc5ec0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx\" (UID: \"1addf2b0-5855-4a72-acba-a10a5bcc5ec0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"
Oct 30 01:14:18.587240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:18.586263    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmnm7\" (UniqueName: \"kubernetes.io/projected/1addf2b0-5855-4a72-acba-a10a5bcc5ec0-kube-api-access-wmnm7\") pod \"payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx\" (UID: \"1addf2b0-5855-4a72-acba-a10a5bcc5ec0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"
Oct 30 01:14:18.587240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:18.578640    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1addf2b0-5855-4a72-acba-a10a5bcc5ec0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx\" (UID: \"1addf2b0-5855-4a72-acba-a10a5bcc5ec0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"
Oct 30 01:14:18.625283 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:18.625243    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmnm7\" (UniqueName: \"kubernetes.io/projected/1addf2b0-5855-4a72-acba-a10a5bcc5ec0-kube-api-access-wmnm7\") pod \"payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx\" (UID: \"1addf2b0-5855-4a72-acba-a10a5bcc5ec0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"
Oct 30 01:14:18.639558 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:18.639517    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"
Oct 30 01:14:18.789243 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:18.787944    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"
Oct 30 01:14:18.982885 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:14:18.982838    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1addf2b0_5855_4a72_acba_a10a5bcc5ec0.slice/crio-afb0ea4d5ac4a0d748d91cc9a765080fbd301eaf67f70190b7a8e861418d589e WatchSource:0}: Error finding container afb0ea4d5ac4a0d748d91cc9a765080fbd301eaf67f70190b7a8e861418d589e: Status 404 returned error can't find the container with id afb0ea4d5ac4a0d748d91cc9a765080fbd301eaf67f70190b7a8e861418d589e
Oct 30 01:14:18.984637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:18.984601    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"]
Oct 30 01:14:19.057907 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:19.057875    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:14:19.985181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:19.968535    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx" event={"ID":"1addf2b0-5855-4a72-acba-a10a5bcc5ec0","Type":"ContainerStarted","Data":"a641f90cd456e23891883c5fd9059ea39a467a76eaa9caabf844504b64c49bfe"}
Oct 30 01:14:19.985181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:19.968614    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx" event={"ID":"1addf2b0-5855-4a72-acba-a10a5bcc5ec0","Type":"ContainerStarted","Data":"afb0ea4d5ac4a0d748d91cc9a765080fbd301eaf67f70190b7a8e861418d589e"}
Oct 30 01:14:19.985181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:19.968666    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"
Oct 30 01:14:20.023208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:20.019319    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx" podStartSLOduration=2.019295967 podStartE2EDuration="2.019295967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:20.016632582 +0000 UTC m=+1968.766287895" watchObservedRunningTime="2026-10-30 01:14:20.019295967 +0000 UTC m=+1968.768951280"
Oct 30 01:14:21.237106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:21.204891    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"]
Oct 30 01:14:21.251214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:21.246632    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"]
Oct 30 01:14:21.251214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:21.247847    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"
Oct 30 01:14:21.327072 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:21.324620    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:14:21.327072 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:21.324872    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:14:21.327072 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:21.325595    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:14:21.434283 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:21.430947    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:14:21.493243 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:21.471928    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cctgq\" (UniqueName: \"kubernetes.io/projected/919e1033-a55b-4991-8de3-3671260b100e-kube-api-access-cctgq\") pod \"maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2\" (UID: \"919e1033-a55b-4991-8de3-3671260b100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"
Oct 30 01:14:21.500181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:21.499373    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/919e1033-a55b-4991-8de3-3671260b100e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2\" (UID: \"919e1033-a55b-4991-8de3-3671260b100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"
Oct 30 01:14:21.500181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:21.499602    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/919e1033-a55b-4991-8de3-3671260b100e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2\" (UID: \"919e1033-a55b-4991-8de3-3671260b100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"
Oct 30 01:14:21.619521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:21.617907    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/919e1033-a55b-4991-8de3-3671260b100e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2\" (UID: \"919e1033-a55b-4991-8de3-3671260b100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"
Oct 30 01:14:21.619521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:21.618221    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cctgq\" (UniqueName: \"kubernetes.io/projected/919e1033-a55b-4991-8de3-3671260b100e-kube-api-access-cctgq\") pod \"maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2\" (UID: \"919e1033-a55b-4991-8de3-3671260b100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"
Oct 30 01:14:21.619521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:21.618389    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/919e1033-a55b-4991-8de3-3671260b100e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2\" (UID: \"919e1033-a55b-4991-8de3-3671260b100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"
Oct 30 01:14:21.619886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:21.619679    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/919e1033-a55b-4991-8de3-3671260b100e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2\" (UID: \"919e1033-a55b-4991-8de3-3671260b100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"
Oct 30 01:14:21.621123 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:21.621080    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/919e1033-a55b-4991-8de3-3671260b100e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2\" (UID: \"919e1033-a55b-4991-8de3-3671260b100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"
Oct 30 01:14:21.641004 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:21.640961    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cctgq\" (UniqueName: \"kubernetes.io/projected/919e1033-a55b-4991-8de3-3671260b100e-kube-api-access-cctgq\") pod \"maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2\" (UID: \"919e1033-a55b-4991-8de3-3671260b100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"
Oct 30 01:14:21.762033 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:21.761393    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"
Oct 30 01:14:22.158288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:22.158234    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"]
Oct 30 01:14:22.161866 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:14:22.161825    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919e1033_a55b_4991_8de3_3671260b100e.slice/crio-c4a08d9638903c25393c3b4fe58f1c846c61719f5c0010476d7a7853e3cdc00e WatchSource:0}: Error finding container c4a08d9638903c25393c3b4fe58f1c846c61719f5c0010476d7a7853e3cdc00e: Status 404 returned error can't find the container with id c4a08d9638903c25393c3b4fe58f1c846c61719f5c0010476d7a7853e3cdc00e
Oct 30 01:14:23.053479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:23.053422    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2" event={"ID":"919e1033-a55b-4991-8de3-3671260b100e","Type":"ContainerStarted","Data":"65edeaeadd91e7bc5b8f67e2db7467d2039cd43c71ecc8a07e814653dee9f1b1"}
Oct 30 01:14:23.053979 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:23.053497    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2" event={"ID":"919e1033-a55b-4991-8de3-3671260b100e","Type":"ContainerStarted","Data":"c4a08d9638903c25393c3b4fe58f1c846c61719f5c0010476d7a7853e3cdc00e"}
Oct 30 01:14:23.124656 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:23.124587    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2" podStartSLOduration=2.124570596 podStartE2EDuration="2.124570596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:23.119896022 +0000 UTC m=+1971.869551334" watchObservedRunningTime="2026-10-30 01:14:23.124570596 +0000 UTC m=+1971.874225905"
Oct 30 01:14:24.063909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:24.063870    4425 generic.go:356] "Generic (PLEG): container finished" podID="919e1033-a55b-4991-8de3-3671260b100e" containerID="65edeaeadd91e7bc5b8f67e2db7467d2039cd43c71ecc8a07e814653dee9f1b1" exitCode=0
Oct 30 01:14:24.064361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:24.063963    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2" event={"ID":"919e1033-a55b-4991-8de3-3671260b100e","Type":"ContainerDied","Data":"65edeaeadd91e7bc5b8f67e2db7467d2039cd43c71ecc8a07e814653dee9f1b1"}
Oct 30 01:14:24.129051 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:24.129011    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"]
Oct 30 01:14:24.133869 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:24.133839    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"
Oct 30 01:14:24.145270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:24.145239    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"]
Oct 30 01:14:24.146433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:24.146384    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-35042a\""
Oct 30 01:14:24.146433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:24.146428    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-35042a-dockercfg-k57gv\""
Oct 30 01:14:24.285047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:24.284985    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c8b78c-9e58-4f50-90c6-16684bfe274a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n\" (UID: \"18c8b78c-9e58-4f50-90c6-16684bfe274a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"
Oct 30 01:14:24.285330 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:24.285287    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxm9v\" (UniqueName: \"kubernetes.io/projected/18c8b78c-9e58-4f50-90c6-16684bfe274a-kube-api-access-xxm9v\") pod \"payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n\" (UID: \"18c8b78c-9e58-4f50-90c6-16684bfe274a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"
Oct 30 01:14:24.386932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:24.386831    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxm9v\" (UniqueName: \"kubernetes.io/projected/18c8b78c-9e58-4f50-90c6-16684bfe274a-kube-api-access-xxm9v\") pod \"payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n\" (UID: \"18c8b78c-9e58-4f50-90c6-16684bfe274a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"
Oct 30 01:14:24.387147 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:24.387000    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c8b78c-9e58-4f50-90c6-16684bfe274a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n\" (UID: \"18c8b78c-9e58-4f50-90c6-16684bfe274a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"
Oct 30 01:14:24.387831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:24.387804    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c8b78c-9e58-4f50-90c6-16684bfe274a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n\" (UID: \"18c8b78c-9e58-4f50-90c6-16684bfe274a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"
Oct 30 01:14:24.408126 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:24.408090    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxm9v\" (UniqueName: \"kubernetes.io/projected/18c8b78c-9e58-4f50-90c6-16684bfe274a-kube-api-access-xxm9v\") pod \"payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n\" (UID: \"18c8b78c-9e58-4f50-90c6-16684bfe274a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"
Oct 30 01:14:24.452051 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:24.452010    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"
Oct 30 01:14:24.763837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:24.763801    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"]
Oct 30 01:14:24.764043 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:14:24.764016    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c8b78c_9e58_4f50_90c6_16684bfe274a.slice/crio-c1e9a8b152fba62b9fe29b114601bfa0ff3f26b4d6fa0541ee8b1668d48a7e4c WatchSource:0}: Error finding container c1e9a8b152fba62b9fe29b114601bfa0ff3f26b4d6fa0541ee8b1668d48a7e4c: Status 404 returned error can't find the container with id c1e9a8b152fba62b9fe29b114601bfa0ff3f26b4d6fa0541ee8b1668d48a7e4c
Oct 30 01:14:25.076698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:25.076607    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n" event={"ID":"18c8b78c-9e58-4f50-90c6-16684bfe274a","Type":"ContainerStarted","Data":"f116d49e73054fd253d98977a9e07d6549e4724014e38060be92be1ef212f5f1"}
Oct 30 01:14:25.076698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:25.076669    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n" event={"ID":"18c8b78c-9e58-4f50-90c6-16684bfe274a","Type":"ContainerStarted","Data":"c1e9a8b152fba62b9fe29b114601bfa0ff3f26b4d6fa0541ee8b1668d48a7e4c"}
Oct 30 01:14:25.076698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:25.076698    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"
Oct 30 01:14:25.115544 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:25.115471    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n" podStartSLOduration=1.115449627 podStartE2EDuration="1.115449627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:25.111617643 +0000 UTC m=+1973.861272955" watchObservedRunningTime="2026-10-30 01:14:25.115449627 +0000 UTC m=+1973.865104938"
Oct 30 01:14:25.225136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:25.225098    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"
Oct 30 01:14:25.299399 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:25.299352    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/919e1033-a55b-4991-8de3-3671260b100e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/919e1033-a55b-4991-8de3-3671260b100e-maas-api-openshift-ca\") pod \"919e1033-a55b-4991-8de3-3671260b100e\" (UID: \"919e1033-a55b-4991-8de3-3671260b100e\") "
Oct 30 01:14:25.299628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:25.299409    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/919e1033-a55b-4991-8de3-3671260b100e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/919e1033-a55b-4991-8de3-3671260b100e-maas-api-xks-ca\") pod \"919e1033-a55b-4991-8de3-3671260b100e\" (UID: \"919e1033-a55b-4991-8de3-3671260b100e\") "
Oct 30 01:14:25.299628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:25.299599    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919e1033-a55b-4991-8de3-3671260b100e-kube-api-access-cctgq\" (UniqueName: \"kubernetes.io/projected/919e1033-a55b-4991-8de3-3671260b100e-kube-api-access-cctgq\") pod \"919e1033-a55b-4991-8de3-3671260b100e\" (UID: \"919e1033-a55b-4991-8de3-3671260b100e\") "
Oct 30 01:14:25.300239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:25.300210    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919e1033-a55b-4991-8de3-3671260b100e-maas-api-openshift-ca" pod "919e1033-a55b-4991-8de3-3671260b100e" (UID: "919e1033-a55b-4991-8de3-3671260b100e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:25.302411 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:25.302377    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/919e1033-a55b-4991-8de3-3671260b100e-maas-api-xks-ca" pod "919e1033-a55b-4991-8de3-3671260b100e" (UID: "919e1033-a55b-4991-8de3-3671260b100e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:25.302714 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:25.302692    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919e1033-a55b-4991-8de3-3671260b100e-kube-api-access-cctgq" pod "919e1033-a55b-4991-8de3-3671260b100e" (UID: "919e1033-a55b-4991-8de3-3671260b100e"). InnerVolumeSpecName "kube-api-access-cctgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:25.401705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:25.401611    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cctgq\" (UniqueName: \"kubernetes.io/projected/919e1033-a55b-4991-8de3-3671260b100e-kube-api-access-cctgq\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:14:25.401705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:25.401645    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/919e1033-a55b-4991-8de3-3671260b100e-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:14:25.401705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:25.401657    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/919e1033-a55b-4991-8de3-3671260b100e-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:14:26.088271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:26.088237    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"
Oct 30 01:14:26.088805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:26.088321    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2" event={"ID":"919e1033-a55b-4991-8de3-3671260b100e","Type":"ContainerDied","Data":"c4a08d9638903c25393c3b4fe58f1c846c61719f5c0010476d7a7853e3cdc00e"}
Oct 30 01:14:26.088805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:26.088365    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4a08d9638903c25393c3b4fe58f1c846c61719f5c0010476d7a7853e3cdc00e"
Oct 30 01:14:28.911385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:28.911350    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"
Oct 30 01:14:29.971215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:29.971181    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"
Oct 30 01:14:31.372293 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:31.372253    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"]
Oct 30 01:14:31.403725 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:31.403685    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"]
Oct 30 01:14:32.049629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:32.047988    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"
Oct 30 01:14:33.438649 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:33.438614    4425 scope.go:122] "RemoveContainer" containerID="d3fe407865ed89aa8928612918129d46a59494500d30e2a7bfc754ef489ff20d"
Oct 30 01:14:36.092436 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:36.092403    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"
Oct 30 01:14:36.373642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:36.373507    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn" podUID="3e79b33e-4d54-4f80-8363-70da08b12fdc" containerName="payload-processing" containerID="cri-o://ede835f6d8cbd06ae7c5758fb6a5485a3e9f3c26f779dbc21e12bdcd325d82d5" gracePeriod=25
Oct 30 01:14:36.404350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:36.404294    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l" podUID="a7aae476-59c5-4012-b346-ce106823d157" containerName="payload-pre-processing" containerID="cri-o://18c9d60fda30c8eaf610603666f47e30cd70aa7f19cb66a0a9a2f6bb0bfc8c83" gracePeriod=25
Oct 30 01:14:36.702981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:36.702947    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"
Oct 30 01:14:36.767799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:36.767766    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"
Oct 30 01:14:36.783603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:36.783555    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e79b33e-4d54-4f80-8363-70da08b12fdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e79b33e-4d54-4f80-8363-70da08b12fdc-plugins-config-volume\") pod \"3e79b33e-4d54-4f80-8363-70da08b12fdc\" (UID: \"3e79b33e-4d54-4f80-8363-70da08b12fdc\") "
Oct 30 01:14:36.783833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:36.783801    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e79b33e-4d54-4f80-8363-70da08b12fdc-kube-api-access-xjfth\" (UniqueName: \"kubernetes.io/projected/3e79b33e-4d54-4f80-8363-70da08b12fdc-kube-api-access-xjfth\") pod \"3e79b33e-4d54-4f80-8363-70da08b12fdc\" (UID: \"3e79b33e-4d54-4f80-8363-70da08b12fdc\") "
Oct 30 01:14:36.784102 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:36.784069    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e79b33e-4d54-4f80-8363-70da08b12fdc-plugins-config-volume" pod "3e79b33e-4d54-4f80-8363-70da08b12fdc" (UID: "3e79b33e-4d54-4f80-8363-70da08b12fdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:36.786015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:36.785981    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e79b33e-4d54-4f80-8363-70da08b12fdc-kube-api-access-xjfth" pod "3e79b33e-4d54-4f80-8363-70da08b12fdc" (UID: "3e79b33e-4d54-4f80-8363-70da08b12fdc"). InnerVolumeSpecName "kube-api-access-xjfth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:36.884814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:36.884770    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7aae476-59c5-4012-b346-ce106823d157-kube-api-access-sfr8s\" (UniqueName: \"kubernetes.io/projected/a7aae476-59c5-4012-b346-ce106823d157-kube-api-access-sfr8s\") pod \"a7aae476-59c5-4012-b346-ce106823d157\" (UID: \"a7aae476-59c5-4012-b346-ce106823d157\") "
Oct 30 01:14:36.885011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:36.884859    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7aae476-59c5-4012-b346-ce106823d157-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7aae476-59c5-4012-b346-ce106823d157-plugins-config-volume\") pod \"a7aae476-59c5-4012-b346-ce106823d157\" (UID: \"a7aae476-59c5-4012-b346-ce106823d157\") "
Oct 30 01:14:36.885327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:36.885290    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7aae476-59c5-4012-b346-ce106823d157-plugins-config-volume" pod "a7aae476-59c5-4012-b346-ce106823d157" (UID: "a7aae476-59c5-4012-b346-ce106823d157"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:36.885327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:36.885308    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjfth\" (UniqueName: \"kubernetes.io/projected/3e79b33e-4d54-4f80-8363-70da08b12fdc-kube-api-access-xjfth\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:14:36.885327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:36.885326    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e79b33e-4d54-4f80-8363-70da08b12fdc-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:14:36.887023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:36.886999    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7aae476-59c5-4012-b346-ce106823d157-kube-api-access-sfr8s" pod "a7aae476-59c5-4012-b346-ce106823d157" (UID: "a7aae476-59c5-4012-b346-ce106823d157"). InnerVolumeSpecName "kube-api-access-sfr8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:36.987048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:36.986946    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfr8s\" (UniqueName: \"kubernetes.io/projected/a7aae476-59c5-4012-b346-ce106823d157-kube-api-access-sfr8s\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:14:36.987048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:36.986985    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7aae476-59c5-4012-b346-ce106823d157-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:14:37.190990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.190946    4425 generic.go:356] "Generic (PLEG): container finished" podID="a7aae476-59c5-4012-b346-ce106823d157" containerID="18c9d60fda30c8eaf610603666f47e30cd70aa7f19cb66a0a9a2f6bb0bfc8c83" exitCode=0
Oct 30 01:14:37.190990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.190988    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"
Oct 30 01:14:37.191588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.191024    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l" event={"ID":"a7aae476-59c5-4012-b346-ce106823d157","Type":"ContainerDied","Data":"18c9d60fda30c8eaf610603666f47e30cd70aa7f19cb66a0a9a2f6bb0bfc8c83"}
Oct 30 01:14:37.191588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.191081    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l" event={"ID":"a7aae476-59c5-4012-b346-ce106823d157","Type":"ContainerDied","Data":"649a2eef2105e4482b06302647c4608e57c84c3c900f4cabe2a58f3ff1036b08"}
Oct 30 01:14:37.191588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.191114    4425 scope.go:122] "RemoveContainer" containerID="18c9d60fda30c8eaf610603666f47e30cd70aa7f19cb66a0a9a2f6bb0bfc8c83"
Oct 30 01:14:37.193017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.192985    4425 generic.go:356] "Generic (PLEG): container finished" podID="3e79b33e-4d54-4f80-8363-70da08b12fdc" containerID="ede835f6d8cbd06ae7c5758fb6a5485a3e9f3c26f779dbc21e12bdcd325d82d5" exitCode=0
Oct 30 01:14:37.193139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.193037    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"
Oct 30 01:14:37.193139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.193080    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn" event={"ID":"3e79b33e-4d54-4f80-8363-70da08b12fdc","Type":"ContainerDied","Data":"ede835f6d8cbd06ae7c5758fb6a5485a3e9f3c26f779dbc21e12bdcd325d82d5"}
Oct 30 01:14:37.193255 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.193149    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn" event={"ID":"3e79b33e-4d54-4f80-8363-70da08b12fdc","Type":"ContainerDied","Data":"996e343d117316913455ac315281c568bc8543b8c706ff252f8b1bc9d8f648ee"}
Oct 30 01:14:37.205094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.205056    4425 scope.go:122] "RemoveContainer" containerID="18c9d60fda30c8eaf610603666f47e30cd70aa7f19cb66a0a9a2f6bb0bfc8c83"
Oct 30 01:14:37.205488 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:14:37.205459    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c9d60fda30c8eaf610603666f47e30cd70aa7f19cb66a0a9a2f6bb0bfc8c83\": container with ID starting with 18c9d60fda30c8eaf610603666f47e30cd70aa7f19cb66a0a9a2f6bb0bfc8c83 not found: ID does not exist" containerID="18c9d60fda30c8eaf610603666f47e30cd70aa7f19cb66a0a9a2f6bb0bfc8c83"
Oct 30 01:14:37.205598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.205502    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c9d60fda30c8eaf610603666f47e30cd70aa7f19cb66a0a9a2f6bb0bfc8c83"} err="failed to get container status \"18c9d60fda30c8eaf610603666f47e30cd70aa7f19cb66a0a9a2f6bb0bfc8c83\": rpc error: code = NotFound desc = could not find container \"18c9d60fda30c8eaf610603666f47e30cd70aa7f19cb66a0a9a2f6bb0bfc8c83\": container with ID starting with 18c9d60fda30c8eaf610603666f47e30cd70aa7f19cb66a0a9a2f6bb0bfc8c83 not found: ID does not exist"
Oct 30 01:14:37.205598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.205524    4425 scope.go:122] "RemoveContainer" containerID="ede835f6d8cbd06ae7c5758fb6a5485a3e9f3c26f779dbc21e12bdcd325d82d5"
Oct 30 01:14:37.216975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.216949    4425 scope.go:122] "RemoveContainer" containerID="ede835f6d8cbd06ae7c5758fb6a5485a3e9f3c26f779dbc21e12bdcd325d82d5"
Oct 30 01:14:37.217407 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:14:37.217376    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede835f6d8cbd06ae7c5758fb6a5485a3e9f3c26f779dbc21e12bdcd325d82d5\": container with ID starting with ede835f6d8cbd06ae7c5758fb6a5485a3e9f3c26f779dbc21e12bdcd325d82d5 not found: ID does not exist" containerID="ede835f6d8cbd06ae7c5758fb6a5485a3e9f3c26f779dbc21e12bdcd325d82d5"
Oct 30 01:14:37.217545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.217410    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede835f6d8cbd06ae7c5758fb6a5485a3e9f3c26f779dbc21e12bdcd325d82d5"} err="failed to get container status \"ede835f6d8cbd06ae7c5758fb6a5485a3e9f3c26f779dbc21e12bdcd325d82d5\": rpc error: code = NotFound desc = could not find container \"ede835f6d8cbd06ae7c5758fb6a5485a3e9f3c26f779dbc21e12bdcd325d82d5\": container with ID starting with ede835f6d8cbd06ae7c5758fb6a5485a3e9f3c26f779dbc21e12bdcd325d82d5 not found: ID does not exist"
Oct 30 01:14:37.252560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.252461    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"]
Oct 30 01:14:37.261918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.261871    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-88a39d-5c554f88c6-9ndtn"]
Oct 30 01:14:37.286680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.286633    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"]
Oct 30 01:14:37.291103 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.291061    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-88a39d-586947f7cf-4q49l"]
Oct 30 01:14:37.423945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.423908    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e79b33e-4d54-4f80-8363-70da08b12fdc" path="/var/lib/kubelet/pods/3e79b33e-4d54-4f80-8363-70da08b12fdc/volumes"
Oct 30 01:14:37.424402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:37.424388    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7aae476-59c5-4012-b346-ce106823d157" path="/var/lib/kubelet/pods/a7aae476-59c5-4012-b346-ce106823d157/volumes"
Oct 30 01:14:37.557809 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:14:37.557772    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:14:38.537414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.537372    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"]
Oct 30 01:14:38.538022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.538006    4425 cpu_manager.go:414] "containerMap: removing container" podUID="3e79b33e-4d54-4f80-8363-70da08b12fdc" containerName="payload-processing"
Oct 30 01:14:38.538065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.538024    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e79b33e-4d54-4f80-8363-70da08b12fdc" containerName="payload-processing"
Oct 30 01:14:38.538065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.538034    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a7aae476-59c5-4012-b346-ce106823d157" containerName="payload-pre-processing"
Oct 30 01:14:38.538065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.538040    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7aae476-59c5-4012-b346-ce106823d157" containerName="payload-pre-processing"
Oct 30 01:14:38.538210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.538067    4425 cpu_manager.go:414] "containerMap: removing container" podUID="919e1033-a55b-4991-8de3-3671260b100e" containerName="revoke-keys"
Oct 30 01:14:38.538210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.538073    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919e1033-a55b-4991-8de3-3671260b100e" containerName="revoke-keys"
Oct 30 01:14:38.538210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.538172    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="919e1033-a55b-4991-8de3-3671260b100e" containerName="revoke-keys"
Oct 30 01:14:38.538210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.538187    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e79b33e-4d54-4f80-8363-70da08b12fdc" containerName="payload-processing"
Oct 30 01:14:38.538210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.538193    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7aae476-59c5-4012-b346-ce106823d157" containerName="payload-pre-processing"
Oct 30 01:14:38.543480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.543448    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.578748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.578709    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"]
Oct 30 01:14:38.608655 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.608600    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.608897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.608758    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.608897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.608807    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.608897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.608843    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4kkk\" (UniqueName: \"kubernetes.io/projected/dabf895c-bcac-422b-96f8-1a7a548415a2-kube-api-access-l4kkk\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.609109 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.608938    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.609109 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.608995    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.609109 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.609064    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dabf895c-bcac-422b-96f8-1a7a548415a2-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.609271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.609119    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.609271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.609214    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.710789 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.710742    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.710992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.710813    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.710992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.710846    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.710992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.710865    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4kkk\" (UniqueName: \"kubernetes.io/projected/dabf895c-bcac-422b-96f8-1a7a548415a2-kube-api-access-l4kkk\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.710992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.710896    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.710992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.710948    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.713106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.711683    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-workload-certs\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.713106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.711755    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dabf895c-bcac-422b-96f8-1a7a548415a2-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.713106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.711805    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.713106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.711844    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.713106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.711750    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-workload-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.713106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.711841    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-credential-socket\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.713106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.712224    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-data\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.713106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.712336    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dabf895c-bcac-422b-96f8-1a7a548415a2-istiod-ca-cert\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.720771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.720732    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-envoy\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.721008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.720982    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-podinfo\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.733637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.733597    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-token\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.733637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.733630    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4kkk\" (UniqueName: \"kubernetes.io/projected/dabf895c-bcac-422b-96f8-1a7a548415a2-kube-api-access-l4kkk\") pod \"e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") " pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:38.861533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:38.861426    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:39.190113 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:14:39.190067    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabf895c_bcac_422b_96f8_1a7a548415a2.slice/crio-6b5f4a3605b184a30cdd2d4c2956db9b4fcf375f7f4641e05e3ed97efd6d359d WatchSource:0}: Error finding container 6b5f4a3605b184a30cdd2d4c2956db9b4fcf375f7f4641e05e3ed97efd6d359d: Status 404 returned error can't find the container with id 6b5f4a3605b184a30cdd2d4c2956db9b4fcf375f7f4641e05e3ed97efd6d359d
Oct 30 01:14:39.190477 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:39.190446    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"]
Oct 30 01:14:39.193345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:39.193302    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:14:39.193466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:39.193383    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:14:39.193466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:39.193423    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:14:39.213937 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:39.213900    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn" event={"ID":"dabf895c-bcac-422b-96f8-1a7a548415a2","Type":"ContainerStarted","Data":"6b5f4a3605b184a30cdd2d4c2956db9b4fcf375f7f4641e05e3ed97efd6d359d"}
Oct 30 01:14:40.226692 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:40.226655    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn" event={"ID":"dabf895c-bcac-422b-96f8-1a7a548415a2","Type":"ContainerStarted","Data":"78fe4f8d25747628e95b836ef2c3c2e73667d19eafc5269714086759081b01f4"}
Oct 30 01:14:40.278701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:40.278635    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn" podStartSLOduration=2.278617968 podStartE2EDuration="2.278617968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:40.276992546 +0000 UTC m=+1989.026647856" watchObservedRunningTime="2026-10-30 01:14:40.278617968 +0000 UTC m=+1989.028273276"
Oct 30 01:14:40.871680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:40.871629    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:40.880398 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:40.880356    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 01:14:40.880711 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:40.880677    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn" podUID="dabf895c-bcac-422b-96f8-1a7a548415a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 01:14:41.875405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:41.875360    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 01:14:41.876423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:41.876377    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn" podUID="dabf895c-bcac-422b-96f8-1a7a548415a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 01:14:42.872184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:42.864658    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 01:14:42.872184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:42.864751    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn" podUID="dabf895c-bcac-422b-96f8-1a7a548415a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 01:14:43.577250 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:43.577187    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:14:43.864751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:43.864641    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 01:14:43.865069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:43.865031    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn" podUID="dabf895c-bcac-422b-96f8-1a7a548415a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 01:14:44.867180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:44.862755    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 01:14:44.867180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:44.862865    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn" podUID="dabf895c-bcac-422b-96f8-1a7a548415a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 01:14:45.864179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:45.863834    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 01:14:45.864179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:45.863997    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn" podUID="dabf895c-bcac-422b-96f8-1a7a548415a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 01:14:46.862657 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:46.862615    4425 patch_prober.go:29] interesting pod/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 30 01:14:46.863059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:46.862678    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn" podUID="dabf895c-bcac-422b-96f8-1a7a548415a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 30 01:14:47.867826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:47.867791    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:47.868442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:47.868336    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:47.869049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:47.869028    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:14:50.401618 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:50.401576    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"]
Oct 30 01:14:50.410850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:50.410814    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-88a39d-lqxc2"]
Oct 30 01:14:51.251492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.251451    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"]
Oct 30 01:14:51.257595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.257559    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"
Oct 30 01:14:51.276459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.276425    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-1854c4-dockercfg-mwltq\""
Oct 30 01:14:51.279680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.279644    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"]
Oct 30 01:14:51.284276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.284237    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-1854c4\""
Oct 30 01:14:51.307338 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.307303    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"]
Oct 30 01:14:51.313533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.313505    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"
Oct 30 01:14:51.330613 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.330578    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"]
Oct 30 01:14:51.393796 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.393750    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw7tp\" (UniqueName: \"kubernetes.io/projected/e47bac65-a69e-45e0-8a8c-f239e14c2e67-kube-api-access-tw7tp\") pod \"payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs\" (UID: \"e47bac65-a69e-45e0-8a8c-f239e14c2e67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"
Oct 30 01:14:51.394218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.394191    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgt7j\" (UniqueName: \"kubernetes.io/projected/f909610e-c454-4ab6-bdf2-d061ade22e78-kube-api-access-bgt7j\") pod \"payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7\" (UID: \"f909610e-c454-4ab6-bdf2-d061ade22e78\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"
Oct 30 01:14:51.394531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.394457    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f909610e-c454-4ab6-bdf2-d061ade22e78-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7\" (UID: \"f909610e-c454-4ab6-bdf2-d061ade22e78\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"
Oct 30 01:14:51.394817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.394784    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47bac65-a69e-45e0-8a8c-f239e14c2e67-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs\" (UID: \"e47bac65-a69e-45e0-8a8c-f239e14c2e67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"
Oct 30 01:14:51.444911 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.444870    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="919e1033-a55b-4991-8de3-3671260b100e" path="/var/lib/kubelet/pods/919e1033-a55b-4991-8de3-3671260b100e/volumes"
Oct 30 01:14:51.496076 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.496030    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47bac65-a69e-45e0-8a8c-f239e14c2e67-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs\" (UID: \"e47bac65-a69e-45e0-8a8c-f239e14c2e67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"
Oct 30 01:14:51.496347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.496182    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw7tp\" (UniqueName: \"kubernetes.io/projected/e47bac65-a69e-45e0-8a8c-f239e14c2e67-kube-api-access-tw7tp\") pod \"payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs\" (UID: \"e47bac65-a69e-45e0-8a8c-f239e14c2e67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"
Oct 30 01:14:51.496347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.496315    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgt7j\" (UniqueName: \"kubernetes.io/projected/f909610e-c454-4ab6-bdf2-d061ade22e78-kube-api-access-bgt7j\") pod \"payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7\" (UID: \"f909610e-c454-4ab6-bdf2-d061ade22e78\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"
Oct 30 01:14:51.496498 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.496433    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f909610e-c454-4ab6-bdf2-d061ade22e78-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7\" (UID: \"f909610e-c454-4ab6-bdf2-d061ade22e78\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"
Oct 30 01:14:51.497144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.497103    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47bac65-a69e-45e0-8a8c-f239e14c2e67-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs\" (UID: \"e47bac65-a69e-45e0-8a8c-f239e14c2e67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"
Oct 30 01:14:51.497440 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.497415    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f909610e-c454-4ab6-bdf2-d061ade22e78-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7\" (UID: \"f909610e-c454-4ab6-bdf2-d061ade22e78\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"
Oct 30 01:14:51.518263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.518228    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw7tp\" (UniqueName: \"kubernetes.io/projected/e47bac65-a69e-45e0-8a8c-f239e14c2e67-kube-api-access-tw7tp\") pod \"payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs\" (UID: \"e47bac65-a69e-45e0-8a8c-f239e14c2e67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"
Oct 30 01:14:51.518458 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.518231    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgt7j\" (UniqueName: \"kubernetes.io/projected/f909610e-c454-4ab6-bdf2-d061ade22e78-kube-api-access-bgt7j\") pod \"payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7\" (UID: \"f909610e-c454-4ab6-bdf2-d061ade22e78\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"
Oct 30 01:14:51.575419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.575368    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"
Oct 30 01:14:51.634517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.634474    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"
Oct 30 01:14:51.895173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.895127    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"]
Oct 30 01:14:51.896376 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:14:51.896342    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf909610e_c454_4ab6_bdf2_d061ade22e78.slice/crio-e35c6329a8e9dd9edb1864e4de0ce601d1860a97f36fe02584ad794a1e5a39da WatchSource:0}: Error finding container e35c6329a8e9dd9edb1864e4de0ce601d1860a97f36fe02584ad794a1e5a39da: Status 404 returned error can't find the container with id e35c6329a8e9dd9edb1864e4de0ce601d1860a97f36fe02584ad794a1e5a39da
Oct 30 01:14:51.971186 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:51.971135    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"]
Oct 30 01:14:51.972759 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:14:51.972728    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47bac65_a69e_45e0_8a8c_f239e14c2e67.slice/crio-3715bcaaf13991d487fe37d7569c210a3d7b1fd3264bc875580af7a3437c9cc9 WatchSource:0}: Error finding container 3715bcaaf13991d487fe37d7569c210a3d7b1fd3264bc875580af7a3437c9cc9: Status 404 returned error can't find the container with id 3715bcaaf13991d487fe37d7569c210a3d7b1fd3264bc875580af7a3437c9cc9
Oct 30 01:14:52.407417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:52.407369    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7" event={"ID":"f909610e-c454-4ab6-bdf2-d061ade22e78","Type":"ContainerStarted","Data":"e56f1683037c50bec3d8cc9650dfadd3ec083672e4f60703f8052ed26c89f602"}
Oct 30 01:14:52.407645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:52.407447    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7" event={"ID":"f909610e-c454-4ab6-bdf2-d061ade22e78","Type":"ContainerStarted","Data":"e35c6329a8e9dd9edb1864e4de0ce601d1860a97f36fe02584ad794a1e5a39da"}
Oct 30 01:14:52.407645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:52.407483    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"
Oct 30 01:14:52.409515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:52.409484    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs" event={"ID":"e47bac65-a69e-45e0-8a8c-f239e14c2e67","Type":"ContainerStarted","Data":"55eac7f9375154924459baf1df17d10ddff844820bb3c4ac0d023af0d8158628"}
Oct 30 01:14:52.409674 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:52.409532    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs" event={"ID":"e47bac65-a69e-45e0-8a8c-f239e14c2e67","Type":"ContainerStarted","Data":"3715bcaaf13991d487fe37d7569c210a3d7b1fd3264bc875580af7a3437c9cc9"}
Oct 30 01:14:52.409674 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:52.409579    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"
Oct 30 01:14:52.484540 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:52.484465    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7" podStartSLOduration=1.4844464419999999 podStartE2EDuration="1.484446442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:52.447387796 +0000 UTC m=+2001.197043105" watchObservedRunningTime="2026-10-30 01:14:52.484446442 +0000 UTC m=+2001.234101802"
Oct 30 01:14:52.486619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:14:52.486566    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs" podStartSLOduration=1.486553808 podStartE2EDuration="1.486553808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:52.482889637 +0000 UTC m=+2001.232544945" watchObservedRunningTime="2026-10-30 01:14:52.486553808 +0000 UTC m=+2001.236209117"
Oct 30 01:15:00.157788 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.157749    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw"]
Oct 30 01:15:00.162853 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.162826    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw"
Oct 30 01:15:00.164629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.164604    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb"]
Oct 30 01:15:00.168974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.168950    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb"
Oct 30 01:15:00.169174 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.169141    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5"]
Oct 30 01:15:00.173492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.173464    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5"
Oct 30 01:15:00.173882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.173845    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk"]
Oct 30 01:15:00.178242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.178219    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk"
Oct 30 01:15:00.183884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.183854    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:15:00.184090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.184048    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:15:00.190047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.188943    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:15:00.206303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.206255    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw"]
Oct 30 01:15:00.223623 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.222193    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr"]
Oct 30 01:15:00.231893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.228558    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr"
Oct 30 01:15:00.237218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.232802    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb"]
Oct 30 01:15:00.243204 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.239451    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5"]
Oct 30 01:15:00.266540 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.265022    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk"]
Oct 30 01:15:00.279226 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.279129    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4"]
Oct 30 01:15:00.284088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.284059    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4"
Oct 30 01:15:00.284598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.284567    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2"]
Oct 30 01:15:00.290096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.290065    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2"
Oct 30 01:15:00.295763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.295733    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2"]
Oct 30 01:15:00.302433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.301986    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4"]
Oct 30 01:15:00.306806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.306778    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr"]
Oct 30 01:15:00.353786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.353745    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c3da1c-0e85-47da-a3dd-ec211c8faff8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb\" (UID: \"08c3da1c-0e85-47da-a3dd-ec211c8faff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb"
Oct 30 01:15:00.354028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.353890    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42gmc\" (UniqueName: \"kubernetes.io/projected/ceda3eee-1d3b-4306-9943-04197e1833b5-kube-api-access-42gmc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk\" (UID: \"ceda3eee-1d3b-4306-9943-04197e1833b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk"
Oct 30 01:15:00.354028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.353940    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qkhf\" (UniqueName: \"kubernetes.io/projected/b65a9b4b-56e6-4a60-b6b5-44994c564061-kube-api-access-9qkhf\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5\" (UID: \"b65a9b4b-56e6-4a60-b6b5-44994c564061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5"
Oct 30 01:15:00.354028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.353985    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65a9b4b-56e6-4a60-b6b5-44994c564061-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5\" (UID: \"b65a9b4b-56e6-4a60-b6b5-44994c564061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5"
Oct 30 01:15:00.354186 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.354085    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b88s\" (UniqueName: \"kubernetes.io/projected/08c3da1c-0e85-47da-a3dd-ec211c8faff8-kube-api-access-8b88s\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb\" (UID: \"08c3da1c-0e85-47da-a3dd-ec211c8faff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb"
Oct 30 01:15:00.354294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.354271    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052185d0-cc6f-4f21-acb3-c634271726a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw\" (UID: \"052185d0-cc6f-4f21-acb3-c634271726a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw"
Oct 30 01:15:00.354384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.354364    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceda3eee-1d3b-4306-9943-04197e1833b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk\" (UID: \"ceda3eee-1d3b-4306-9943-04197e1833b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk"
Oct 30 01:15:00.354436 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.354404    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/365c7093-230d-487b-a582-1feacf89487d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr\" (UID: \"365c7093-230d-487b-a582-1feacf89487d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr"
Oct 30 01:15:00.354486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.354465    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb5mg\" (UniqueName: \"kubernetes.io/projected/365c7093-230d-487b-a582-1feacf89487d-kube-api-access-zb5mg\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr\" (UID: \"365c7093-230d-487b-a582-1feacf89487d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr"
Oct 30 01:15:00.354525 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.354494    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfjcl\" (UniqueName: \"kubernetes.io/projected/052185d0-cc6f-4f21-acb3-c634271726a6-kube-api-access-nfjcl\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw\" (UID: \"052185d0-cc6f-4f21-acb3-c634271726a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw"
Oct 30 01:15:00.455677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.455554    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5mg\" (UniqueName: \"kubernetes.io/projected/365c7093-230d-487b-a582-1feacf89487d-kube-api-access-zb5mg\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr\" (UID: \"365c7093-230d-487b-a582-1feacf89487d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr"
Oct 30 01:15:00.455677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.455607    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfjcl\" (UniqueName: \"kubernetes.io/projected/052185d0-cc6f-4f21-acb3-c634271726a6-kube-api-access-nfjcl\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw\" (UID: \"052185d0-cc6f-4f21-acb3-c634271726a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw"
Oct 30 01:15:00.455677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.455640    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/315812a7-031f-48c3-a854-9aca8e8c63f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4\" (UID: \"315812a7-031f-48c3-a854-9aca8e8c63f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4"
Oct 30 01:15:00.455996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.455713    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c3da1c-0e85-47da-a3dd-ec211c8faff8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb\" (UID: \"08c3da1c-0e85-47da-a3dd-ec211c8faff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb"
Oct 30 01:15:00.455996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.455743    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hxgn\" (UniqueName: \"kubernetes.io/projected/315812a7-031f-48c3-a854-9aca8e8c63f8-kube-api-access-9hxgn\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4\" (UID: \"315812a7-031f-48c3-a854-9aca8e8c63f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4"
Oct 30 01:15:00.455996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.455820    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gmc\" (UniqueName: \"kubernetes.io/projected/ceda3eee-1d3b-4306-9943-04197e1833b5-kube-api-access-42gmc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk\" (UID: \"ceda3eee-1d3b-4306-9943-04197e1833b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk"
Oct 30 01:15:00.455996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.455854    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qkhf\" (UniqueName: \"kubernetes.io/projected/b65a9b4b-56e6-4a60-b6b5-44994c564061-kube-api-access-9qkhf\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5\" (UID: \"b65a9b4b-56e6-4a60-b6b5-44994c564061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5"
Oct 30 01:15:00.456625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.456595    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65a9b4b-56e6-4a60-b6b5-44994c564061-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5\" (UID: \"b65a9b4b-56e6-4a60-b6b5-44994c564061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5"
Oct 30 01:15:00.456721 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.456677    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pxhj\" (UniqueName: \"kubernetes.io/projected/d696b9da-4e58-4b10-8880-78a0cb2d4e86-kube-api-access-9pxhj\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2\" (UID: \"d696b9da-4e58-4b10-8880-78a0cb2d4e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2"
Oct 30 01:15:00.456780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.456726    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b88s\" (UniqueName: \"kubernetes.io/projected/08c3da1c-0e85-47da-a3dd-ec211c8faff8-kube-api-access-8b88s\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb\" (UID: \"08c3da1c-0e85-47da-a3dd-ec211c8faff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb"
Oct 30 01:15:00.456780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.456754    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c3da1c-0e85-47da-a3dd-ec211c8faff8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb\" (UID: \"08c3da1c-0e85-47da-a3dd-ec211c8faff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb"
Oct 30 01:15:00.456889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.456777    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d696b9da-4e58-4b10-8880-78a0cb2d4e86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2\" (UID: \"d696b9da-4e58-4b10-8880-78a0cb2d4e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2"
Oct 30 01:15:00.456940 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.456914    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052185d0-cc6f-4f21-acb3-c634271726a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw\" (UID: \"052185d0-cc6f-4f21-acb3-c634271726a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw"
Oct 30 01:15:00.456994 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.456978    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceda3eee-1d3b-4306-9943-04197e1833b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk\" (UID: \"ceda3eee-1d3b-4306-9943-04197e1833b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk"
Oct 30 01:15:00.457052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.457009    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/365c7093-230d-487b-a582-1feacf89487d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr\" (UID: \"365c7093-230d-487b-a582-1feacf89487d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr"
Oct 30 01:15:00.457974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.457517    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65a9b4b-56e6-4a60-b6b5-44994c564061-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5\" (UID: \"b65a9b4b-56e6-4a60-b6b5-44994c564061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5"
Oct 30 01:15:00.457974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.457665    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052185d0-cc6f-4f21-acb3-c634271726a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw\" (UID: \"052185d0-cc6f-4f21-acb3-c634271726a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw"
Oct 30 01:15:00.457974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.457798    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/365c7093-230d-487b-a582-1feacf89487d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr\" (UID: \"365c7093-230d-487b-a582-1feacf89487d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr"
Oct 30 01:15:00.458323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.458251    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceda3eee-1d3b-4306-9943-04197e1833b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk\" (UID: \"ceda3eee-1d3b-4306-9943-04197e1833b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk"
Oct 30 01:15:00.477895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.477861    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb5mg\" (UniqueName: \"kubernetes.io/projected/365c7093-230d-487b-a582-1feacf89487d-kube-api-access-zb5mg\") pod \"maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr\" (UID: \"365c7093-230d-487b-a582-1feacf89487d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr"
Oct 30 01:15:00.478072 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.477863    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfjcl\" (UniqueName: \"kubernetes.io/projected/052185d0-cc6f-4f21-acb3-c634271726a6-kube-api-access-nfjcl\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw\" (UID: \"052185d0-cc6f-4f21-acb3-c634271726a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw"
Oct 30 01:15:00.478072 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.477945    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42gmc\" (UniqueName: \"kubernetes.io/projected/ceda3eee-1d3b-4306-9943-04197e1833b5-kube-api-access-42gmc\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk\" (UID: \"ceda3eee-1d3b-4306-9943-04197e1833b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk"
Oct 30 01:15:00.478184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.478119    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qkhf\" (UniqueName: \"kubernetes.io/projected/b65a9b4b-56e6-4a60-b6b5-44994c564061-kube-api-access-9qkhf\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5\" (UID: \"b65a9b4b-56e6-4a60-b6b5-44994c564061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5"
Oct 30 01:15:00.482288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.482265    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw"
Oct 30 01:15:00.483833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.483807    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b88s\" (UniqueName: \"kubernetes.io/projected/08c3da1c-0e85-47da-a3dd-ec211c8faff8-kube-api-access-8b88s\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb\" (UID: \"08c3da1c-0e85-47da-a3dd-ec211c8faff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb"
Oct 30 01:15:00.555046 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.555017    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb"
Oct 30 01:15:00.558514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.558487    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pxhj\" (UniqueName: \"kubernetes.io/projected/d696b9da-4e58-4b10-8880-78a0cb2d4e86-kube-api-access-9pxhj\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2\" (UID: \"d696b9da-4e58-4b10-8880-78a0cb2d4e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2"
Oct 30 01:15:00.558659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.558552    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d696b9da-4e58-4b10-8880-78a0cb2d4e86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2\" (UID: \"d696b9da-4e58-4b10-8880-78a0cb2d4e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2"
Oct 30 01:15:00.558659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.558640    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/315812a7-031f-48c3-a854-9aca8e8c63f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4\" (UID: \"315812a7-031f-48c3-a854-9aca8e8c63f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4"
Oct 30 01:15:00.558730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.558694    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hxgn\" (UniqueName: \"kubernetes.io/projected/315812a7-031f-48c3-a854-9aca8e8c63f8-kube-api-access-9hxgn\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4\" (UID: \"315812a7-031f-48c3-a854-9aca8e8c63f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4"
Oct 30 01:15:00.559505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.559483    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d696b9da-4e58-4b10-8880-78a0cb2d4e86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2\" (UID: \"d696b9da-4e58-4b10-8880-78a0cb2d4e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2"
Oct 30 01:15:00.559684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.559665    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/315812a7-031f-48c3-a854-9aca8e8c63f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4\" (UID: \"315812a7-031f-48c3-a854-9aca8e8c63f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4"
Oct 30 01:15:00.576245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.572968    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5"
Oct 30 01:15:00.576245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.575407    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk"
Oct 30 01:15:00.584703 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.584674    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hxgn\" (UniqueName: \"kubernetes.io/projected/315812a7-031f-48c3-a854-9aca8e8c63f8-kube-api-access-9hxgn\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4\" (UID: \"315812a7-031f-48c3-a854-9aca8e8c63f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4"
Oct 30 01:15:00.584862 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.584765    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pxhj\" (UniqueName: \"kubernetes.io/projected/d696b9da-4e58-4b10-8880-78a0cb2d4e86-kube-api-access-9pxhj\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2\" (UID: \"d696b9da-4e58-4b10-8880-78a0cb2d4e86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2"
Oct 30 01:15:00.586641 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.586615    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr"
Oct 30 01:15:00.603712 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.603665    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4"
Oct 30 01:15:00.611285 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.610996    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2"
Oct 30 01:15:00.712572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.712460    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"]
Oct 30 01:15:00.715059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.713807    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg" podUID="be82dda6-937b-4ab8-bf48-1b3471845c9a" containerName="istio-proxy" containerID="cri-o://96da4152bcc19c5aa137686d2562a071a72f0b4e362d65cc7cb06dae5102dab8" gracePeriod=30
Oct 30 01:15:00.811269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.809379    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw"]
Oct 30 01:15:00.948382 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:15:00.948350    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65a9b4b_56e6_4a60_b6b5_44994c564061.slice/crio-88b6804d1d2308bc23098098cc7acba25432292c16b0ee0a6c3eb51a927e80cd WatchSource:0}: Error finding container 88b6804d1d2308bc23098098cc7acba25432292c16b0ee0a6c3eb51a927e80cd: Status 404 returned error can't find the container with id 88b6804d1d2308bc23098098cc7acba25432292c16b0ee0a6c3eb51a927e80cd
Oct 30 01:15:00.948553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.948528    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5"]
Oct 30 01:15:00.987106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:00.987063    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk"]
Oct 30 01:15:01.027921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:01.027888    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2"]
Oct 30 01:15:01.052405 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:15:01.052368    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceda3eee_1d3b_4306_9943_04197e1833b5.slice/crio-67cd96a22c799038a322b8a644ce9774ea47da0b2d07eb0f71a5b64f80961900 WatchSource:0}: Error finding container 67cd96a22c799038a322b8a644ce9774ea47da0b2d07eb0f71a5b64f80961900: Status 404 returned error can't find the container with id 67cd96a22c799038a322b8a644ce9774ea47da0b2d07eb0f71a5b64f80961900
Oct 30 01:15:01.064679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:01.062520    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4"]
Oct 30 01:15:01.066137 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:15:01.066099    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315812a7_031f_48c3_a854_9aca8e8c63f8.slice/crio-0f28ac014c729ddf93db163656809d0165165c803b7384162fd1de546f3d4491 WatchSource:0}: Error finding container 0f28ac014c729ddf93db163656809d0165165c803b7384162fd1de546f3d4491: Status 404 returned error can't find the container with id 0f28ac014c729ddf93db163656809d0165165c803b7384162fd1de546f3d4491
Oct 30 01:15:01.121696 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:15:01.121635    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c3da1c_0e85_47da_a3dd_ec211c8faff8.slice/crio-aace3236b5b6784e9651e04f56aa166140a05ea586df4aa3c382bb6469d6d5c0 WatchSource:0}: Error finding container aace3236b5b6784e9651e04f56aa166140a05ea586df4aa3c382bb6469d6d5c0: Status 404 returned error can't find the container with id aace3236b5b6784e9651e04f56aa166140a05ea586df4aa3c382bb6469d6d5c0
Oct 30 01:15:01.124516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:01.124483    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb"]
Oct 30 01:15:01.224997 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:01.224953    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr"]
Oct 30 01:15:01.257011 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:15:01.256969    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365c7093_230d_487b_a582_1feacf89487d.slice/crio-8f8dd99c9150f02eb26c8b47015dcc3e9ed226c4c444f24531e0fe936a55506d WatchSource:0}: Error finding container 8f8dd99c9150f02eb26c8b47015dcc3e9ed226c4c444f24531e0fe936a55506d: Status 404 returned error can't find the container with id 8f8dd99c9150f02eb26c8b47015dcc3e9ed226c4c444f24531e0fe936a55506d
Oct 30 01:15:01.515372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:01.515332    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr" event={"ID":"365c7093-230d-487b-a582-1feacf89487d","Type":"ContainerStarted","Data":"8f8dd99c9150f02eb26c8b47015dcc3e9ed226c4c444f24531e0fe936a55506d"}
Oct 30 01:15:01.516814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:01.516780    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4" event={"ID":"315812a7-031f-48c3-a854-9aca8e8c63f8","Type":"ContainerStarted","Data":"0f28ac014c729ddf93db163656809d0165165c803b7384162fd1de546f3d4491"}
Oct 30 01:15:01.518509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:01.518478    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw" event={"ID":"052185d0-cc6f-4f21-acb3-c634271726a6","Type":"ContainerStarted","Data":"be56f968c780d23a6e06a221fefff2a4cfd425e7c05be79f1dacb73de39f4937"}
Oct 30 01:15:01.518658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:01.518529    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw" event={"ID":"052185d0-cc6f-4f21-acb3-c634271726a6","Type":"ContainerStarted","Data":"801a33f1c6102bccb0723f45a33ee45d7e726c84cb63eef2ecf5ad8b02c06915"}
Oct 30 01:15:01.519968 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:01.519940    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2" event={"ID":"d696b9da-4e58-4b10-8880-78a0cb2d4e86","Type":"ContainerStarted","Data":"574c62ee4f259c3c6021c27d13d69e5ab3944b068e5d252d427985678b69b9e3"}
Oct 30 01:15:01.521343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:01.521318    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5" event={"ID":"b65a9b4b-56e6-4a60-b6b5-44994c564061","Type":"ContainerStarted","Data":"88b6804d1d2308bc23098098cc7acba25432292c16b0ee0a6c3eb51a927e80cd"}
Oct 30 01:15:01.522544 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:01.522519    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb" event={"ID":"08c3da1c-0e85-47da-a3dd-ec211c8faff8","Type":"ContainerStarted","Data":"aace3236b5b6784e9651e04f56aa166140a05ea586df4aa3c382bb6469d6d5c0"}
Oct 30 01:15:01.523901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:01.523876    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk" event={"ID":"ceda3eee-1d3b-4306-9943-04197e1833b5","Type":"ContainerStarted","Data":"67cd96a22c799038a322b8a644ce9774ea47da0b2d07eb0f71a5b64f80961900"}
Oct 30 01:15:01.559341 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:01.559276    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw" podStartSLOduration=1.559258558 podStartE2EDuration="1.559258558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:01.554290934 +0000 UTC m=+2010.303946244" watchObservedRunningTime="2026-10-30 01:15:01.559258558 +0000 UTC m=+2010.308913867"
Oct 30 01:15:02.533506 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.533464    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr" event={"ID":"365c7093-230d-487b-a582-1feacf89487d","Type":"ContainerStarted","Data":"3a2090c4b6d4fb157bd205ca2d7f402b6894b31aa2c78451d65e547edd1ae0c7"}
Oct 30 01:15:02.535443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.535414    4425 generic.go:356] "Generic (PLEG): container finished" podID="315812a7-031f-48c3-a854-9aca8e8c63f8" containerID="7156aa5cbf1a8afefed2c97d7ae76fa5cda23847ef67ec6bbc381a8797378b59" exitCode=60
Oct 30 01:15:02.535568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.535462    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4" event={"ID":"315812a7-031f-48c3-a854-9aca8e8c63f8","Type":"ContainerDied","Data":"7156aa5cbf1a8afefed2c97d7ae76fa5cda23847ef67ec6bbc381a8797378b59"}
Oct 30 01:15:02.535839 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.535821    4425 scope.go:122] "RemoveContainer" containerID="7156aa5cbf1a8afefed2c97d7ae76fa5cda23847ef67ec6bbc381a8797378b59"
Oct 30 01:15:02.537328 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.537306    4425 generic.go:356] "Generic (PLEG): container finished" podID="052185d0-cc6f-4f21-acb3-c634271726a6" containerID="be56f968c780d23a6e06a221fefff2a4cfd425e7c05be79f1dacb73de39f4937" exitCode=60
Oct 30 01:15:02.537425 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.537382    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw" event={"ID":"052185d0-cc6f-4f21-acb3-c634271726a6","Type":"ContainerDied","Data":"be56f968c780d23a6e06a221fefff2a4cfd425e7c05be79f1dacb73de39f4937"}
Oct 30 01:15:02.537641 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.537613    4425 scope.go:122] "RemoveContainer" containerID="be56f968c780d23a6e06a221fefff2a4cfd425e7c05be79f1dacb73de39f4937"
Oct 30 01:15:02.539333 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.539309    4425 generic.go:356] "Generic (PLEG): container finished" podID="d696b9da-4e58-4b10-8880-78a0cb2d4e86" containerID="fe3ff1ea009623b0a9afe895854a68c299aff01e6694fb9691af4add63c83f7f" exitCode=60
Oct 30 01:15:02.539424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.539388    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2" event={"ID":"d696b9da-4e58-4b10-8880-78a0cb2d4e86","Type":"ContainerDied","Data":"fe3ff1ea009623b0a9afe895854a68c299aff01e6694fb9691af4add63c83f7f"}
Oct 30 01:15:02.539648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.539612    4425 scope.go:122] "RemoveContainer" containerID="fe3ff1ea009623b0a9afe895854a68c299aff01e6694fb9691af4add63c83f7f"
Oct 30 01:15:02.541612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.541589    4425 generic.go:356] "Generic (PLEG): container finished" podID="b65a9b4b-56e6-4a60-b6b5-44994c564061" containerID="e09d789322280c06e140e52d1ab83f8c16bed3ed6a5cba3eca3e2a00f69fe710" exitCode=60
Oct 30 01:15:02.542365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.541681    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5" event={"ID":"b65a9b4b-56e6-4a60-b6b5-44994c564061","Type":"ContainerDied","Data":"e09d789322280c06e140e52d1ab83f8c16bed3ed6a5cba3eca3e2a00f69fe710"}
Oct 30 01:15:02.542365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.541903    4425 scope.go:122] "RemoveContainer" containerID="e09d789322280c06e140e52d1ab83f8c16bed3ed6a5cba3eca3e2a00f69fe710"
Oct 30 01:15:02.543772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.543749    4425 generic.go:356] "Generic (PLEG): container finished" podID="08c3da1c-0e85-47da-a3dd-ec211c8faff8" containerID="86c8c740f537c47a60dc30e7c246c448e85613612542282c8f098833c2920cf4" exitCode=60
Oct 30 01:15:02.543874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.543784    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb" event={"ID":"08c3da1c-0e85-47da-a3dd-ec211c8faff8","Type":"ContainerDied","Data":"86c8c740f537c47a60dc30e7c246c448e85613612542282c8f098833c2920cf4"}
Oct 30 01:15:02.544090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.544075    4425 scope.go:122] "RemoveContainer" containerID="86c8c740f537c47a60dc30e7c246c448e85613612542282c8f098833c2920cf4"
Oct 30 01:15:02.545687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.545664    4425 generic.go:356] "Generic (PLEG): container finished" podID="ceda3eee-1d3b-4306-9943-04197e1833b5" containerID="343f97dd787b31e62197f7cbeda819e3bbad464d53b9afcf879b679b25708511" exitCode=60
Oct 30 01:15:02.545777 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.545692    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk" event={"ID":"ceda3eee-1d3b-4306-9943-04197e1833b5","Type":"ContainerDied","Data":"343f97dd787b31e62197f7cbeda819e3bbad464d53b9afcf879b679b25708511"}
Oct 30 01:15:02.545963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.545949    4425 scope.go:122] "RemoveContainer" containerID="343f97dd787b31e62197f7cbeda819e3bbad464d53b9afcf879b679b25708511"
Oct 30 01:15:02.575141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:02.575025    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr" podStartSLOduration=2.5750008060000003 podStartE2EDuration="2.575000806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:02.571072784 +0000 UTC m=+2011.320728111" watchObservedRunningTime="2026-10-30 01:15:02.575000806 +0000 UTC m=+2011.324656118"
Oct 30 01:15:03.425424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:03.425335    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"
Oct 30 01:15:03.425424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:03.425416    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"
Oct 30 01:15:03.560365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:03.560330    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb" event={"ID":"08c3da1c-0e85-47da-a3dd-ec211c8faff8","Type":"ContainerStarted","Data":"027b7ef09627a8fe77b16fa859301f8aebdf7b676acb921120c884731f1e85d2"}
Oct 30 01:15:03.562697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:03.562657    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk" event={"ID":"ceda3eee-1d3b-4306-9943-04197e1833b5","Type":"ContainerStarted","Data":"dc42b3d0fe3b3f992ecdcb431fbea09ecfaebdd55ddf7e5783e40f9f3e37ad4f"}
Oct 30 01:15:03.564848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:03.564815    4425 generic.go:356] "Generic (PLEG): container finished" podID="365c7093-230d-487b-a582-1feacf89487d" containerID="3a2090c4b6d4fb157bd205ca2d7f402b6894b31aa2c78451d65e547edd1ae0c7" exitCode=60
Oct 30 01:15:03.565017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:03.564895    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr" event={"ID":"365c7093-230d-487b-a582-1feacf89487d","Type":"ContainerDied","Data":"3a2090c4b6d4fb157bd205ca2d7f402b6894b31aa2c78451d65e547edd1ae0c7"}
Oct 30 01:15:03.565247 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:03.565229    4425 scope.go:122] "RemoveContainer" containerID="3a2090c4b6d4fb157bd205ca2d7f402b6894b31aa2c78451d65e547edd1ae0c7"
Oct 30 01:15:03.567469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:03.567440    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4" event={"ID":"315812a7-031f-48c3-a854-9aca8e8c63f8","Type":"ContainerStarted","Data":"09f5696e2e7541b2d493751f30b24947e45654557b115aae86c6210324756932"}
Oct 30 01:15:03.570345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:03.570300    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw" event={"ID":"052185d0-cc6f-4f21-acb3-c634271726a6","Type":"ContainerStarted","Data":"3b0b906e9e5db26d017d88df21446f8bfbb7089a8359f3b187aef16b07510acf"}
Oct 30 01:15:03.573084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:03.573022    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2" event={"ID":"d696b9da-4e58-4b10-8880-78a0cb2d4e86","Type":"ContainerStarted","Data":"204218ffc644c4466bc44e059a22273fb1c5ebeeaac1c7ad47177e3b551ff856"}
Oct 30 01:15:03.575471 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:03.575446    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5" event={"ID":"b65a9b4b-56e6-4a60-b6b5-44994c564061","Type":"ContainerStarted","Data":"84b4ee85b99012f117d4aaa93f9d280dfb15f36eead1228e1896df6b8f5183b1"}
Oct 30 01:15:03.597674 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:03.597611    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb" podStartSLOduration=3.597595353 podStartE2EDuration="3.597595353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.597417009 +0000 UTC m=+2012.347072318" watchObservedRunningTime="2026-10-30 01:15:03.597595353 +0000 UTC m=+2012.347250663"
Oct 30 01:15:03.633323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:03.633261    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2" podStartSLOduration=3.633244774 podStartE2EDuration="3.633244774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.632023487 +0000 UTC m=+2012.381678796" watchObservedRunningTime="2026-10-30 01:15:03.633244774 +0000 UTC m=+2012.382900087"
Oct 30 01:15:03.707376 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:03.707253    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4" podStartSLOduration=3.70723663 podStartE2EDuration="3.70723663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.70392445 +0000 UTC m=+2012.453579763" watchObservedRunningTime="2026-10-30 01:15:03.70723663 +0000 UTC m=+2012.456891939"
Oct 30 01:15:03.739114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:03.739055    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5" podStartSLOduration=3.73904201 podStartE2EDuration="3.73904201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.738814212 +0000 UTC m=+2012.488469521" watchObservedRunningTime="2026-10-30 01:15:03.73904201 +0000 UTC m=+2012.488697319"
Oct 30 01:15:03.778073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:03.778015    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk" podStartSLOduration=3.77799677 podStartE2EDuration="3.77799677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:03.774058991 +0000 UTC m=+2012.523714302" watchObservedRunningTime="2026-10-30 01:15:03.77799677 +0000 UTC m=+2012.527652078"
Oct 30 01:15:04.328488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.328445    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"]
Oct 30 01:15:04.333022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.332991    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"
Oct 30 01:15:04.355034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.354990    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44e61e3-669d-4ac3-8686-68f72e4df64c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj\" (UID: \"c44e61e3-669d-4ac3-8686-68f72e4df64c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"
Oct 30 01:15:04.355326 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.355297    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvkst\" (UniqueName: \"kubernetes.io/projected/c44e61e3-669d-4ac3-8686-68f72e4df64c-kube-api-access-pvkst\") pod \"payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj\" (UID: \"c44e61e3-669d-4ac3-8686-68f72e4df64c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"
Oct 30 01:15:04.366569 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.366523    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"]
Oct 30 01:15:04.369578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.369544    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c666c2\""
Oct 30 01:15:04.372253 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.372220    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"]
Oct 30 01:15:04.372443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.372352    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"
Oct 30 01:15:04.400064 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.400028    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-c666c2-dockercfg-jhdhr\""
Oct 30 01:15:04.411990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.411207    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"]
Oct 30 01:15:04.464303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.464252    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx\" (UID: \"60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"
Oct 30 01:15:04.464496 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.464385    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvkst\" (UniqueName: \"kubernetes.io/projected/c44e61e3-669d-4ac3-8686-68f72e4df64c-kube-api-access-pvkst\") pod \"payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj\" (UID: \"c44e61e3-669d-4ac3-8686-68f72e4df64c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"
Oct 30 01:15:04.464600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.464505    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qg8f\" (UniqueName: \"kubernetes.io/projected/60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e-kube-api-access-8qg8f\") pod \"payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx\" (UID: \"60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"
Oct 30 01:15:04.464600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.464557    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44e61e3-669d-4ac3-8686-68f72e4df64c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj\" (UID: \"c44e61e3-669d-4ac3-8686-68f72e4df64c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"
Oct 30 01:15:04.465445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.465404    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44e61e3-669d-4ac3-8686-68f72e4df64c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj\" (UID: \"c44e61e3-669d-4ac3-8686-68f72e4df64c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"
Oct 30 01:15:04.485440 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.485408    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvkst\" (UniqueName: \"kubernetes.io/projected/c44e61e3-669d-4ac3-8686-68f72e4df64c-kube-api-access-pvkst\") pod \"payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj\" (UID: \"c44e61e3-669d-4ac3-8686-68f72e4df64c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"
Oct 30 01:15:04.565466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.565359    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx\" (UID: \"60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"
Oct 30 01:15:04.565895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.565510    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qg8f\" (UniqueName: \"kubernetes.io/projected/60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e-kube-api-access-8qg8f\") pod \"payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx\" (UID: \"60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"
Oct 30 01:15:04.566106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.566083    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx\" (UID: \"60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"
Oct 30 01:15:04.586047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.586005    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr" event={"ID":"365c7093-230d-487b-a582-1feacf89487d","Type":"ContainerStarted","Data":"96c4ae46f7081688b8f440a12916e1fe96962f6e8af58de780362f7e1e8f0456"}
Oct 30 01:15:04.588286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.588251    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qg8f\" (UniqueName: \"kubernetes.io/projected/60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e-kube-api-access-8qg8f\") pod \"payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx\" (UID: \"60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"
Oct 30 01:15:04.588638 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.588611    4425 generic.go:356] "Generic (PLEG): container finished" podID="315812a7-031f-48c3-a854-9aca8e8c63f8" containerID="09f5696e2e7541b2d493751f30b24947e45654557b115aae86c6210324756932" exitCode=60
Oct 30 01:15:04.588729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.588690    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4" event={"ID":"315812a7-031f-48c3-a854-9aca8e8c63f8","Type":"ContainerDied","Data":"09f5696e2e7541b2d493751f30b24947e45654557b115aae86c6210324756932"}
Oct 30 01:15:04.588792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.588745    4425 scope.go:122] "RemoveContainer" containerID="7156aa5cbf1a8afefed2c97d7ae76fa5cda23847ef67ec6bbc381a8797378b59"
Oct 30 01:15:04.589050 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.589023    4425 scope.go:122] "RemoveContainer" containerID="09f5696e2e7541b2d493751f30b24947e45654557b115aae86c6210324756932"
Oct 30 01:15:04.589500 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:04.589476    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4_odh-ai-gateway-infra(315812a7-031f-48c3-a854-9aca8e8c63f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4" podUID="315812a7-031f-48c3-a854-9aca8e8c63f8"
Oct 30 01:15:04.591222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.591199    4425 generic.go:356] "Generic (PLEG): container finished" podID="052185d0-cc6f-4f21-acb3-c634271726a6" containerID="3b0b906e9e5db26d017d88df21446f8bfbb7089a8359f3b187aef16b07510acf" exitCode=60
Oct 30 01:15:04.591327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.591301    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw" event={"ID":"052185d0-cc6f-4f21-acb3-c634271726a6","Type":"ContainerDied","Data":"3b0b906e9e5db26d017d88df21446f8bfbb7089a8359f3b187aef16b07510acf"}
Oct 30 01:15:04.591604 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.591589    4425 scope.go:122] "RemoveContainer" containerID="3b0b906e9e5db26d017d88df21446f8bfbb7089a8359f3b187aef16b07510acf"
Oct 30 01:15:04.591973 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:04.591954    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw_odh-ai-gateway-infra(052185d0-cc6f-4f21-acb3-c634271726a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw" podUID="052185d0-cc6f-4f21-acb3-c634271726a6"
Oct 30 01:15:04.593902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.593874    4425 generic.go:356] "Generic (PLEG): container finished" podID="d696b9da-4e58-4b10-8880-78a0cb2d4e86" containerID="204218ffc644c4466bc44e059a22273fb1c5ebeeaac1c7ad47177e3b551ff856" exitCode=60
Oct 30 01:15:04.594051 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.593966    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2" event={"ID":"d696b9da-4e58-4b10-8880-78a0cb2d4e86","Type":"ContainerDied","Data":"204218ffc644c4466bc44e059a22273fb1c5ebeeaac1c7ad47177e3b551ff856"}
Oct 30 01:15:04.594568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.594387    4425 scope.go:122] "RemoveContainer" containerID="204218ffc644c4466bc44e059a22273fb1c5ebeeaac1c7ad47177e3b551ff856"
Oct 30 01:15:04.594871 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:04.594842    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2_odh-ai-gateway-infra(d696b9da-4e58-4b10-8880-78a0cb2d4e86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2" podUID="d696b9da-4e58-4b10-8880-78a0cb2d4e86"
Oct 30 01:15:04.596556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.596529    4425 generic.go:356] "Generic (PLEG): container finished" podID="b65a9b4b-56e6-4a60-b6b5-44994c564061" containerID="84b4ee85b99012f117d4aaa93f9d280dfb15f36eead1228e1896df6b8f5183b1" exitCode=60
Oct 30 01:15:04.596651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.596618    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5" event={"ID":"b65a9b4b-56e6-4a60-b6b5-44994c564061","Type":"ContainerDied","Data":"84b4ee85b99012f117d4aaa93f9d280dfb15f36eead1228e1896df6b8f5183b1"}
Oct 30 01:15:04.597034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.597014    4425 scope.go:122] "RemoveContainer" containerID="84b4ee85b99012f117d4aaa93f9d280dfb15f36eead1228e1896df6b8f5183b1"
Oct 30 01:15:04.597462 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:04.597438    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5_odh-ai-gateway-infra(b65a9b4b-56e6-4a60-b6b5-44994c564061)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5" podUID="b65a9b4b-56e6-4a60-b6b5-44994c564061"
Oct 30 01:15:04.599267 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.599239    4425 generic.go:356] "Generic (PLEG): container finished" podID="08c3da1c-0e85-47da-a3dd-ec211c8faff8" containerID="027b7ef09627a8fe77b16fa859301f8aebdf7b676acb921120c884731f1e85d2" exitCode=60
Oct 30 01:15:04.599364 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.599316    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb" event={"ID":"08c3da1c-0e85-47da-a3dd-ec211c8faff8","Type":"ContainerDied","Data":"027b7ef09627a8fe77b16fa859301f8aebdf7b676acb921120c884731f1e85d2"}
Oct 30 01:15:04.599624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.599608    4425 scope.go:122] "RemoveContainer" containerID="027b7ef09627a8fe77b16fa859301f8aebdf7b676acb921120c884731f1e85d2"
Oct 30 01:15:04.600022 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:04.599993    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb_odh-ai-gateway-infra(08c3da1c-0e85-47da-a3dd-ec211c8faff8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb" podUID="08c3da1c-0e85-47da-a3dd-ec211c8faff8"
Oct 30 01:15:04.601360 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.601325    4425 generic.go:356] "Generic (PLEG): container finished" podID="ceda3eee-1d3b-4306-9943-04197e1833b5" containerID="dc42b3d0fe3b3f992ecdcb431fbea09ecfaebdd55ddf7e5783e40f9f3e37ad4f" exitCode=60
Oct 30 01:15:04.601474 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.601413    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk" event={"ID":"ceda3eee-1d3b-4306-9943-04197e1833b5","Type":"ContainerDied","Data":"dc42b3d0fe3b3f992ecdcb431fbea09ecfaebdd55ddf7e5783e40f9f3e37ad4f"}
Oct 30 01:15:04.601747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.601732    4425 scope.go:122] "RemoveContainer" containerID="dc42b3d0fe3b3f992ecdcb431fbea09ecfaebdd55ddf7e5783e40f9f3e37ad4f"
Oct 30 01:15:04.602193 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:04.602138    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk_odh-ai-gateway-infra(ceda3eee-1d3b-4306-9943-04197e1833b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk" podUID="ceda3eee-1d3b-4306-9943-04197e1833b5"
Oct 30 01:15:04.610238 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.610211    4425 scope.go:122] "RemoveContainer" containerID="be56f968c780d23a6e06a221fefff2a4cfd425e7c05be79f1dacb73de39f4937"
Oct 30 01:15:04.627113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.627088    4425 scope.go:122] "RemoveContainer" containerID="fe3ff1ea009623b0a9afe895854a68c299aff01e6694fb9691af4add63c83f7f"
Oct 30 01:15:04.654226 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.654147    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"
Oct 30 01:15:04.663848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.662825    4425 scope.go:122] "RemoveContainer" containerID="e09d789322280c06e140e52d1ab83f8c16bed3ed6a5cba3eca3e2a00f69fe710"
Oct 30 01:15:04.690131 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.690061    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"
Oct 30 01:15:04.747547 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.747499    4425 scope.go:122] "RemoveContainer" containerID="86c8c740f537c47a60dc30e7c246c448e85613612542282c8f098833c2920cf4"
Oct 30 01:15:04.804193 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:04.804148    4425 scope.go:122] "RemoveContainer" containerID="343f97dd787b31e62197f7cbeda819e3bbad464d53b9afcf879b679b25708511"
Oct 30 01:15:05.021597 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.021564    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"]
Oct 30 01:15:05.022422 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:15:05.022396    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44e61e3_669d_4ac3_8686_68f72e4df64c.slice/crio-34c2a34a0998775a407a1e293e9c6702480f89db8ca8d916de05e407fb73f666 WatchSource:0}: Error finding container 34c2a34a0998775a407a1e293e9c6702480f89db8ca8d916de05e407fb73f666: Status 404 returned error can't find the container with id 34c2a34a0998775a407a1e293e9c6702480f89db8ca8d916de05e407fb73f666
Oct 30 01:15:05.102705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.102670    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"]
Oct 30 01:15:05.108893 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:15:05.108860    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c5e5c8_a1f3_41b1_b8a3_8fecf93e1b9e.slice/crio-fbf28af0e6ba601a39c5a1f6a9c1116645aa44898fb88b1087ae463a743c439c WatchSource:0}: Error finding container fbf28af0e6ba601a39c5a1f6a9c1116645aa44898fb88b1087ae463a743c439c: Status 404 returned error can't find the container with id fbf28af0e6ba601a39c5a1f6a9c1116645aa44898fb88b1087ae463a743c439c
Oct 30 01:15:05.272308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.272264    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"]
Oct 30 01:15:05.277447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.277418    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"
Oct 30 01:15:05.294676 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.294642    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-bdf6ca\""
Oct 30 01:15:05.298904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.298865    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"]
Oct 30 01:15:05.328047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.328003    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"]
Oct 30 01:15:05.332843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.332769    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"
Oct 30 01:15:05.356217 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.349976    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-bdf6ca-dockercfg-rcslh\""
Oct 30 01:15:05.359221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.357519    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"]
Oct 30 01:15:05.385003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.384967    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j827d\" (UniqueName: \"kubernetes.io/projected/60123056-76c2-4e39-a284-d0a9ae34e1e8-kube-api-access-j827d\") pod \"payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm\" (UID: \"60123056-76c2-4e39-a284-d0a9ae34e1e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"
Oct 30 01:15:05.385226 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.385023    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6003b30d-107b-4075-aa52-8b00fd6f3d2c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp\" (UID: \"6003b30d-107b-4075-aa52-8b00fd6f3d2c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"
Oct 30 01:15:05.385226 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.385128    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7szdv\" (UniqueName: \"kubernetes.io/projected/6003b30d-107b-4075-aa52-8b00fd6f3d2c-kube-api-access-7szdv\") pod \"payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp\" (UID: \"6003b30d-107b-4075-aa52-8b00fd6f3d2c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"
Oct 30 01:15:05.385314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.385298    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60123056-76c2-4e39-a284-d0a9ae34e1e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm\" (UID: \"60123056-76c2-4e39-a284-d0a9ae34e1e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"
Oct 30 01:15:05.486810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.486768    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60123056-76c2-4e39-a284-d0a9ae34e1e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm\" (UID: \"60123056-76c2-4e39-a284-d0a9ae34e1e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"
Oct 30 01:15:05.487063 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.486946    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j827d\" (UniqueName: \"kubernetes.io/projected/60123056-76c2-4e39-a284-d0a9ae34e1e8-kube-api-access-j827d\") pod \"payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm\" (UID: \"60123056-76c2-4e39-a284-d0a9ae34e1e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"
Oct 30 01:15:05.487143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.487086    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6003b30d-107b-4075-aa52-8b00fd6f3d2c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp\" (UID: \"6003b30d-107b-4075-aa52-8b00fd6f3d2c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"
Oct 30 01:15:05.487222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.487147    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7szdv\" (UniqueName: \"kubernetes.io/projected/6003b30d-107b-4075-aa52-8b00fd6f3d2c-kube-api-access-7szdv\") pod \"payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp\" (UID: \"6003b30d-107b-4075-aa52-8b00fd6f3d2c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"
Oct 30 01:15:05.487647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.487623    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60123056-76c2-4e39-a284-d0a9ae34e1e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm\" (UID: \"60123056-76c2-4e39-a284-d0a9ae34e1e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"
Oct 30 01:15:05.488053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.488031    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6003b30d-107b-4075-aa52-8b00fd6f3d2c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp\" (UID: \"6003b30d-107b-4075-aa52-8b00fd6f3d2c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"
Oct 30 01:15:05.509835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.509795    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7szdv\" (UniqueName: \"kubernetes.io/projected/6003b30d-107b-4075-aa52-8b00fd6f3d2c-kube-api-access-7szdv\") pod \"payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp\" (UID: \"6003b30d-107b-4075-aa52-8b00fd6f3d2c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"
Oct 30 01:15:05.509987 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.509841    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j827d\" (UniqueName: \"kubernetes.io/projected/60123056-76c2-4e39-a284-d0a9ae34e1e8-kube-api-access-j827d\") pod \"payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm\" (UID: \"60123056-76c2-4e39-a284-d0a9ae34e1e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"
Oct 30 01:15:05.593469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.593363    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"
Oct 30 01:15:05.614774 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.614742    4425 generic.go:356] "Generic (PLEG): container finished" podID="365c7093-230d-487b-a582-1feacf89487d" containerID="96c4ae46f7081688b8f440a12916e1fe96962f6e8af58de780362f7e1e8f0456" exitCode=60
Oct 30 01:15:05.614927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.614817    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr" event={"ID":"365c7093-230d-487b-a582-1feacf89487d","Type":"ContainerDied","Data":"96c4ae46f7081688b8f440a12916e1fe96962f6e8af58de780362f7e1e8f0456"}
Oct 30 01:15:05.614927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.614892    4425 scope.go:122] "RemoveContainer" containerID="3a2090c4b6d4fb157bd205ca2d7f402b6894b31aa2c78451d65e547edd1ae0c7"
Oct 30 01:15:05.615261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.615243    4425 scope.go:122] "RemoveContainer" containerID="96c4ae46f7081688b8f440a12916e1fe96962f6e8af58de780362f7e1e8f0456"
Oct 30 01:15:05.615647 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:05.615624    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr_odh-ai-gateway-infra(365c7093-230d-487b-a582-1feacf89487d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr" podUID="365c7093-230d-487b-a582-1feacf89487d"
Oct 30 01:15:05.618183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.618146    4425 scope.go:122] "RemoveContainer" containerID="09f5696e2e7541b2d493751f30b24947e45654557b115aae86c6210324756932"
Oct 30 01:15:05.618761 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:05.618731    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4_odh-ai-gateway-infra(315812a7-031f-48c3-a854-9aca8e8c63f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4" podUID="315812a7-031f-48c3-a854-9aca8e8c63f8"
Oct 30 01:15:05.622946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.622913    4425 scope.go:122] "RemoveContainer" containerID="3b0b906e9e5db26d017d88df21446f8bfbb7089a8359f3b187aef16b07510acf"
Oct 30 01:15:05.623422 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:05.623391    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw_odh-ai-gateway-infra(052185d0-cc6f-4f21-acb3-c634271726a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw" podUID="052185d0-cc6f-4f21-acb3-c634271726a6"
Oct 30 01:15:05.625852 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.625827    4425 scope.go:122] "RemoveContainer" containerID="204218ffc644c4466bc44e059a22273fb1c5ebeeaac1c7ad47177e3b551ff856"
Oct 30 01:15:05.626384 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:05.626349    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2_odh-ai-gateway-infra(d696b9da-4e58-4b10-8880-78a0cb2d4e86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2" podUID="d696b9da-4e58-4b10-8880-78a0cb2d4e86"
Oct 30 01:15:05.631244 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.631217    4425 scope.go:122] "RemoveContainer" containerID="84b4ee85b99012f117d4aaa93f9d280dfb15f36eead1228e1896df6b8f5183b1"
Oct 30 01:15:05.631639 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:05.631609    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5_odh-ai-gateway-infra(b65a9b4b-56e6-4a60-b6b5-44994c564061)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5" podUID="b65a9b4b-56e6-4a60-b6b5-44994c564061"
Oct 30 01:15:05.633765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.633738    4425 scope.go:122] "RemoveContainer" containerID="027b7ef09627a8fe77b16fa859301f8aebdf7b676acb921120c884731f1e85d2"
Oct 30 01:15:05.634168 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:05.634126    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb_odh-ai-gateway-infra(08c3da1c-0e85-47da-a3dd-ec211c8faff8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb" podUID="08c3da1c-0e85-47da-a3dd-ec211c8faff8"
Oct 30 01:15:05.636310 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.636291    4425 scope.go:122] "RemoveContainer" containerID="dc42b3d0fe3b3f992ecdcb431fbea09ecfaebdd55ddf7e5783e40f9f3e37ad4f"
Oct 30 01:15:05.636663 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:05.636641    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk_odh-ai-gateway-infra(ceda3eee-1d3b-4306-9943-04197e1833b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk" podUID="ceda3eee-1d3b-4306-9943-04197e1833b5"
Oct 30 01:15:05.637892 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.637868    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj" event={"ID":"c44e61e3-669d-4ac3-8686-68f72e4df64c","Type":"ContainerStarted","Data":"5bfc0d0c7e3a421906c18d71ebf1342fceaa91c7253d76364e5a07c7732d27fe"}
Oct 30 01:15:05.637995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.637924    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj" event={"ID":"c44e61e3-669d-4ac3-8686-68f72e4df64c","Type":"ContainerStarted","Data":"34c2a34a0998775a407a1e293e9c6702480f89db8ca8d916de05e407fb73f666"}
Oct 30 01:15:05.637995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.637959    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"
Oct 30 01:15:05.640125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.640100    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx" event={"ID":"60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e","Type":"ContainerStarted","Data":"03642b2e4f65bd9fe504e826e029fff1f74202dc3ba90f4942137a0726d18f9f"}
Oct 30 01:15:05.640286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.640194    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx" event={"ID":"60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e","Type":"ContainerStarted","Data":"fbf28af0e6ba601a39c5a1f6a9c1116645aa44898fb88b1087ae463a743c439c"}
Oct 30 01:15:05.640286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.640231    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"
Oct 30 01:15:05.665230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.665136    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"
Oct 30 01:15:05.750290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.749986    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj" podStartSLOduration=1.749960287 podStartE2EDuration="1.749960287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:05.744409744 +0000 UTC m=+2014.494065053" watchObservedRunningTime="2026-10-30 01:15:05.749960287 +0000 UTC m=+2014.499615599"
Oct 30 01:15:05.789725 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.787737    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx" podStartSLOduration=1.7877124100000001 podStartE2EDuration="1.78771241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:05.78349543 +0000 UTC m=+2014.533150739" watchObservedRunningTime="2026-10-30 01:15:05.78771241 +0000 UTC m=+2014.537367720"
Oct 30 01:15:05.875723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.875684    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 01:15:05.875965 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.875751    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg" podUID="be82dda6-937b-4ab8-bf48-1b3471845c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 01:15:05.922977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:05.922940    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"]
Oct 30 01:15:05.923610 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:15:05.923579    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6003b30d_107b_4075_aa52_8b00fd6f3d2c.slice/crio-3117e7ee54fdf7e13100ce768d9876abfcc4ce0345b5ca5c128de09659c0a82a WatchSource:0}: Error finding container 3117e7ee54fdf7e13100ce768d9876abfcc4ce0345b5ca5c128de09659c0a82a: Status 404 returned error can't find the container with id 3117e7ee54fdf7e13100ce768d9876abfcc4ce0345b5ca5c128de09659c0a82a
Oct 30 01:15:06.018610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.018560    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"]
Oct 30 01:15:06.031324 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:15:06.031291    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60123056_76c2_4e39_a284_d0a9ae34e1e8.slice/crio-3a4e6471a917cba0bba95720160cea6600b8811a784d3deefca34581271e658d WatchSource:0}: Error finding container 3a4e6471a917cba0bba95720160cea6600b8811a784d3deefca34581271e658d: Status 404 returned error can't find the container with id 3a4e6471a917cba0bba95720160cea6600b8811a784d3deefca34581271e658d
Oct 30 01:15:06.051122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.051003    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 01:15:06.116554 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.116445    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-token\" (UniqueName: \"kubernetes.io/projected/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-token\") pod \"be82dda6-937b-4ab8-bf48-1b3471845c9a\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") "
Oct 30 01:15:06.116554 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.116518    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-credential-socket\") pod \"be82dda6-937b-4ab8-bf48-1b3471845c9a\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") "
Oct 30 01:15:06.116809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.116615    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be82dda6-937b-4ab8-bf48-1b3471845c9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be82dda6-937b-4ab8-bf48-1b3471845c9a-istiod-ca-cert\") pod \"be82dda6-937b-4ab8-bf48-1b3471845c9a\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") "
Oct 30 01:15:06.116809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.116675    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-workload-socket\") pod \"be82dda6-937b-4ab8-bf48-1b3471845c9a\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") "
Oct 30 01:15:06.116809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.116776    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-data\") pod \"be82dda6-937b-4ab8-bf48-1b3471845c9a\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") "
Oct 30 01:15:06.116977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.116864    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-envoy\") pod \"be82dda6-937b-4ab8-bf48-1b3471845c9a\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") "
Oct 30 01:15:06.116977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.116952    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-podinfo\") pod \"be82dda6-937b-4ab8-bf48-1b3471845c9a\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") "
Oct 30 01:15:06.117087 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.117009    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-workload-certs\") pod \"be82dda6-937b-4ab8-bf48-1b3471845c9a\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") "
Oct 30 01:15:06.117087 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.117053    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be82dda6-937b-4ab8-bf48-1b3471845c9a-kube-api-access-zq684\" (UniqueName: \"kubernetes.io/projected/be82dda6-937b-4ab8-bf48-1b3471845c9a-kube-api-access-zq684\") pod \"be82dda6-937b-4ab8-bf48-1b3471845c9a\" (UID: \"be82dda6-937b-4ab8-bf48-1b3471845c9a\") "
Oct 30 01:15:06.118394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.118362    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-workload-socket" pod "be82dda6-937b-4ab8-bf48-1b3471845c9a" (UID: "be82dda6-937b-4ab8-bf48-1b3471845c9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:06.118856 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.118827    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-credential-socket" pod "be82dda6-937b-4ab8-bf48-1b3471845c9a" (UID: "be82dda6-937b-4ab8-bf48-1b3471845c9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:06.119336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.119298    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-workload-certs" pod "be82dda6-937b-4ab8-bf48-1b3471845c9a" (UID: "be82dda6-937b-4ab8-bf48-1b3471845c9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:06.119449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.119381    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-data" pod "be82dda6-937b-4ab8-bf48-1b3471845c9a" (UID: "be82dda6-937b-4ab8-bf48-1b3471845c9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:06.119865 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.119830    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be82dda6-937b-4ab8-bf48-1b3471845c9a-istiod-ca-cert" pod "be82dda6-937b-4ab8-bf48-1b3471845c9a" (UID: "be82dda6-937b-4ab8-bf48-1b3471845c9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:06.120441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.120409    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be82dda6-937b-4ab8-bf48-1b3471845c9a-kube-api-access-zq684" pod "be82dda6-937b-4ab8-bf48-1b3471845c9a" (UID: "be82dda6-937b-4ab8-bf48-1b3471845c9a"). InnerVolumeSpecName "kube-api-access-zq684". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:06.121007 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.120974    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-token" pod "be82dda6-937b-4ab8-bf48-1b3471845c9a" (UID: "be82dda6-937b-4ab8-bf48-1b3471845c9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:06.121120 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.121063    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-podinfo" pod "be82dda6-937b-4ab8-bf48-1b3471845c9a" (UID: "be82dda6-937b-4ab8-bf48-1b3471845c9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:15:06.132036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.131907    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-envoy" pod "be82dda6-937b-4ab8-bf48-1b3471845c9a" (UID: "be82dda6-937b-4ab8-bf48-1b3471845c9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:06.220325 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.220286    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.220573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.220543    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq684\" (UniqueName: \"kubernetes.io/projected/be82dda6-937b-4ab8-bf48-1b3471845c9a-kube-api-access-zq684\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.220573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.220571    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.220752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.220587    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.220752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.220600    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be82dda6-937b-4ab8-bf48-1b3471845c9a-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.220752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.220616    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.220752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.220627    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.220752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.220640    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.220752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.220652    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be82dda6-937b-4ab8-bf48-1b3471845c9a-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.659715 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.659682    4425 scope.go:122] "RemoveContainer" containerID="96c4ae46f7081688b8f440a12916e1fe96962f6e8af58de780362f7e1e8f0456"
Oct 30 01:15:06.660299 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:06.660078    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr_odh-ai-gateway-infra(365c7093-230d-487b-a582-1feacf89487d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr" podUID="365c7093-230d-487b-a582-1feacf89487d"
Oct 30 01:15:06.661609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.661574    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp" event={"ID":"6003b30d-107b-4075-aa52-8b00fd6f3d2c","Type":"ContainerStarted","Data":"a4d6078b3c08d80f0225529396916cfb9b5104f10a9f7f4ffa86231c00237221"}
Oct 30 01:15:06.661734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.661639    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp" event={"ID":"6003b30d-107b-4075-aa52-8b00fd6f3d2c","Type":"ContainerStarted","Data":"3117e7ee54fdf7e13100ce768d9876abfcc4ce0345b5ca5c128de09659c0a82a"}
Oct 30 01:15:06.661734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.661702    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"
Oct 30 01:15:06.664061 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.664026    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm" event={"ID":"60123056-76c2-4e39-a284-d0a9ae34e1e8","Type":"ContainerStarted","Data":"217d289d19ec55b868645ca6a88f16dfb6043a62b23a650abead30a3420fb881"}
Oct 30 01:15:06.664249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.664093    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"
Oct 30 01:15:06.664249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.664119    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm" event={"ID":"60123056-76c2-4e39-a284-d0a9ae34e1e8","Type":"ContainerStarted","Data":"3a4e6471a917cba0bba95720160cea6600b8811a784d3deefca34581271e658d"}
Oct 30 01:15:06.665913 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.665883    4425 generic.go:356] "Generic (PLEG): container finished" podID="be82dda6-937b-4ab8-bf48-1b3471845c9a" containerID="96da4152bcc19c5aa137686d2562a071a72f0b4e362d65cc7cb06dae5102dab8" exitCode=0
Oct 30 01:15:06.666034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.665931    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"
Oct 30 01:15:06.666034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.665966    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg" event={"ID":"be82dda6-937b-4ab8-bf48-1b3471845c9a","Type":"ContainerDied","Data":"96da4152bcc19c5aa137686d2562a071a72f0b4e362d65cc7cb06dae5102dab8"}
Oct 30 01:15:06.666034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.666026    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg" event={"ID":"be82dda6-937b-4ab8-bf48-1b3471845c9a","Type":"ContainerDied","Data":"48c466f0d72f07295fb27ec121f2db568719ad35d22561ef52a4544f0762a3a5"}
Oct 30 01:15:06.666239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.666060    4425 scope.go:122] "RemoveContainer" containerID="96da4152bcc19c5aa137686d2562a071a72f0b4e362d65cc7cb06dae5102dab8"
Oct 30 01:15:06.681422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.681394    4425 scope.go:122] "RemoveContainer" containerID="96da4152bcc19c5aa137686d2562a071a72f0b4e362d65cc7cb06dae5102dab8"
Oct 30 01:15:06.681901 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:06.681871    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96da4152bcc19c5aa137686d2562a071a72f0b4e362d65cc7cb06dae5102dab8\": container with ID starting with 96da4152bcc19c5aa137686d2562a071a72f0b4e362d65cc7cb06dae5102dab8 not found: ID does not exist" containerID="96da4152bcc19c5aa137686d2562a071a72f0b4e362d65cc7cb06dae5102dab8"
Oct 30 01:15:06.682013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.681913    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96da4152bcc19c5aa137686d2562a071a72f0b4e362d65cc7cb06dae5102dab8"} err="failed to get container status \"96da4152bcc19c5aa137686d2562a071a72f0b4e362d65cc7cb06dae5102dab8\": rpc error: code = NotFound desc = could not find container \"96da4152bcc19c5aa137686d2562a071a72f0b4e362d65cc7cb06dae5102dab8\": container with ID starting with 96da4152bcc19c5aa137686d2562a071a72f0b4e362d65cc7cb06dae5102dab8 not found: ID does not exist"
Oct 30 01:15:06.730857 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.730790    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm" podStartSLOduration=1.730774146 podStartE2EDuration="1.730774146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:06.729787409 +0000 UTC m=+2015.479442717" watchObservedRunningTime="2026-10-30 01:15:06.730774146 +0000 UTC m=+2015.480429455"
Oct 30 01:15:06.769901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.769842    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp" podStartSLOduration=1.769824184 podStartE2EDuration="1.769824184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:06.764961864 +0000 UTC m=+2015.514617172" watchObservedRunningTime="2026-10-30 01:15:06.769824184 +0000 UTC m=+2015.519479492"
Oct 30 01:15:06.813002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.812965    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"]
Oct 30 01:15:06.821877 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:06.821842    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-nc5zg"]
Oct 30 01:15:07.424854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:07.424815    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be82dda6-937b-4ab8-bf48-1b3471845c9a" path="/var/lib/kubelet/pods/be82dda6-937b-4ab8-bf48-1b3471845c9a/volumes"
Oct 30 01:15:07.558365 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:07.558325    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:15:12.331498 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.331452    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"]
Oct 30 01:15:12.332690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.332664    4425 cpu_manager.go:414] "containerMap: removing container" podUID="be82dda6-937b-4ab8-bf48-1b3471845c9a" containerName="istio-proxy"
Oct 30 01:15:12.332750 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.332695    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be82dda6-937b-4ab8-bf48-1b3471845c9a" containerName="istio-proxy"
Oct 30 01:15:12.332915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.332900    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="be82dda6-937b-4ab8-bf48-1b3471845c9a" containerName="istio-proxy"
Oct 30 01:15:12.345211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.345136    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"
Oct 30 01:15:12.365052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.363506    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:15:12.366886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.366085    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"]
Oct 30 01:15:12.414327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.414285    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11af40fe-8b0c-4d7a-a1a9-46a491e85424-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a520e2b9-pprxc\" (UID: \"11af40fe-8b0c-4d7a-a1a9-46a491e85424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"
Oct 30 01:15:12.414532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.414451    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjnz8\" (UniqueName: \"kubernetes.io/projected/11af40fe-8b0c-4d7a-a1a9-46a491e85424-kube-api-access-tjnz8\") pod \"maas-api-revoke-sub-a520e2b9-pprxc\" (UID: \"11af40fe-8b0c-4d7a-a1a9-46a491e85424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"
Oct 30 01:15:12.414532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.414481    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11af40fe-8b0c-4d7a-a1a9-46a491e85424-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a520e2b9-pprxc\" (UID: \"11af40fe-8b0c-4d7a-a1a9-46a491e85424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"
Oct 30 01:15:12.516572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.516526    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11af40fe-8b0c-4d7a-a1a9-46a491e85424-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a520e2b9-pprxc\" (UID: \"11af40fe-8b0c-4d7a-a1a9-46a491e85424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"
Oct 30 01:15:12.516825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.516799    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjnz8\" (UniqueName: \"kubernetes.io/projected/11af40fe-8b0c-4d7a-a1a9-46a491e85424-kube-api-access-tjnz8\") pod \"maas-api-revoke-sub-a520e2b9-pprxc\" (UID: \"11af40fe-8b0c-4d7a-a1a9-46a491e85424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"
Oct 30 01:15:12.516902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.516848    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11af40fe-8b0c-4d7a-a1a9-46a491e85424-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a520e2b9-pprxc\" (UID: \"11af40fe-8b0c-4d7a-a1a9-46a491e85424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"
Oct 30 01:15:12.517452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.517423    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11af40fe-8b0c-4d7a-a1a9-46a491e85424-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a520e2b9-pprxc\" (UID: \"11af40fe-8b0c-4d7a-a1a9-46a491e85424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"
Oct 30 01:15:12.519531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.519506    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11af40fe-8b0c-4d7a-a1a9-46a491e85424-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a520e2b9-pprxc\" (UID: \"11af40fe-8b0c-4d7a-a1a9-46a491e85424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"
Oct 30 01:15:12.539003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.538956    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjnz8\" (UniqueName: \"kubernetes.io/projected/11af40fe-8b0c-4d7a-a1a9-46a491e85424-kube-api-access-tjnz8\") pod \"maas-api-revoke-sub-a520e2b9-pprxc\" (UID: \"11af40fe-8b0c-4d7a-a1a9-46a491e85424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"
Oct 30 01:15:12.675296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.675112    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"
Oct 30 01:15:12.994613 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:15:12.994579    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11af40fe_8b0c_4d7a_a1a9_46a491e85424.slice/crio-e2fa65819c0a4d0a9cfba164d1317f3c04d7381e8a3999616ea954e34e4c867e WatchSource:0}: Error finding container e2fa65819c0a4d0a9cfba164d1317f3c04d7381e8a3999616ea954e34e4c867e: Status 404 returned error can't find the container with id e2fa65819c0a4d0a9cfba164d1317f3c04d7381e8a3999616ea954e34e4c867e
Oct 30 01:15:12.995088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:12.995057    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"]
Oct 30 01:15:13.708847 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:13.708744    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:15:13.740971 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:13.740914    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc" event={"ID":"11af40fe-8b0c-4d7a-a1a9-46a491e85424","Type":"ContainerStarted","Data":"9ea2a9d6aab015f4dfc678d3e3e35f0be399e9d74c5f7929a2e2d321a7bb94eb"}
Oct 30 01:15:13.741260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:13.741000    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc" event={"ID":"11af40fe-8b0c-4d7a-a1a9-46a491e85424","Type":"ContainerStarted","Data":"e2fa65819c0a4d0a9cfba164d1317f3c04d7381e8a3999616ea954e34e4c867e"}
Oct 30 01:15:13.780006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:13.779945    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc" podStartSLOduration=1.7799266550000001 podStartE2EDuration="1.779926655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:13.776794003 +0000 UTC m=+2022.526449312" watchObservedRunningTime="2026-10-30 01:15:13.779926655 +0000 UTC m=+2022.529581965"
Oct 30 01:15:14.415754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:14.415670    4425 scope.go:122] "RemoveContainer" containerID="84b4ee85b99012f117d4aaa93f9d280dfb15f36eead1228e1896df6b8f5183b1"
Oct 30 01:15:14.754065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:14.753983    4425 generic.go:356] "Generic (PLEG): container finished" podID="11af40fe-8b0c-4d7a-a1a9-46a491e85424" containerID="9ea2a9d6aab015f4dfc678d3e3e35f0be399e9d74c5f7929a2e2d321a7bb94eb" exitCode=0
Oct 30 01:15:14.754444 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:14.754121    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc" event={"ID":"11af40fe-8b0c-4d7a-a1a9-46a491e85424","Type":"ContainerDied","Data":"9ea2a9d6aab015f4dfc678d3e3e35f0be399e9d74c5f7929a2e2d321a7bb94eb"}
Oct 30 01:15:14.827862 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:14.827817    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"]
Oct 30 01:15:14.834472 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:14.834441    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"
Oct 30 01:15:14.856986 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:14.856951    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-9da49f\""
Oct 30 01:15:14.865864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:14.865813    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"]
Oct 30 01:15:14.969029 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:14.968991    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxhdn\" (UniqueName: \"kubernetes.io/projected/50a5c7e3-26ea-42d6-b760-2ac74b5bdda1-kube-api-access-fxhdn\") pod \"payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s\" (UID: \"50a5c7e3-26ea-42d6-b760-2ac74b5bdda1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"
Oct 30 01:15:14.969280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:14.969113    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a5c7e3-26ea-42d6-b760-2ac74b5bdda1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s\" (UID: \"50a5c7e3-26ea-42d6-b760-2ac74b5bdda1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"
Oct 30 01:15:15.071189 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:15.071123    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxhdn\" (UniqueName: \"kubernetes.io/projected/50a5c7e3-26ea-42d6-b760-2ac74b5bdda1-kube-api-access-fxhdn\") pod \"payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s\" (UID: \"50a5c7e3-26ea-42d6-b760-2ac74b5bdda1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"
Oct 30 01:15:15.071402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:15.071274    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a5c7e3-26ea-42d6-b760-2ac74b5bdda1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s\" (UID: \"50a5c7e3-26ea-42d6-b760-2ac74b5bdda1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"
Oct 30 01:15:15.072325 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:15.072297    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a5c7e3-26ea-42d6-b760-2ac74b5bdda1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s\" (UID: \"50a5c7e3-26ea-42d6-b760-2ac74b5bdda1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"
Oct 30 01:15:15.093338 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:15.093301    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxhdn\" (UniqueName: \"kubernetes.io/projected/50a5c7e3-26ea-42d6-b760-2ac74b5bdda1-kube-api-access-fxhdn\") pod \"payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s\" (UID: \"50a5c7e3-26ea-42d6-b760-2ac74b5bdda1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"
Oct 30 01:15:15.152613 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:15.152580    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"
Oct 30 01:15:15.415720 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:15.415632    4425 scope.go:122] "RemoveContainer" containerID="3b0b906e9e5db26d017d88df21446f8bfbb7089a8359f3b187aef16b07510acf"
Oct 30 01:15:15.415720 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:15.415713    4425 scope.go:122] "RemoveContainer" containerID="96c4ae46f7081688b8f440a12916e1fe96962f6e8af58de780362f7e1e8f0456"
Oct 30 01:15:15.468969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:15.468937    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"]
Oct 30 01:15:15.510439 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:15:15.510403    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a5c7e3_26ea_42d6_b760_2ac74b5bdda1.slice/crio-c2cd6bfd36a8b426507f4f897a80185d2b6fb04a7b5269b6564043e029c8f97c WatchSource:0}: Error finding container c2cd6bfd36a8b426507f4f897a80185d2b6fb04a7b5269b6564043e029c8f97c: Status 404 returned error can't find the container with id c2cd6bfd36a8b426507f4f897a80185d2b6fb04a7b5269b6564043e029c8f97c
Oct 30 01:15:15.767712 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:15.767677    4425 generic.go:356] "Generic (PLEG): container finished" podID="b65a9b4b-56e6-4a60-b6b5-44994c564061" containerID="f89ad25bc52947e845446dffb67bf78da4053d087d585c60546668faf6ed45d2" exitCode=60
Oct 30 01:15:15.768206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:15.767757    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5" event={"ID":"b65a9b4b-56e6-4a60-b6b5-44994c564061","Type":"ContainerDied","Data":"f89ad25bc52947e845446dffb67bf78da4053d087d585c60546668faf6ed45d2"}
Oct 30 01:15:15.768206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:15.767838    4425 scope.go:122] "RemoveContainer" containerID="84b4ee85b99012f117d4aaa93f9d280dfb15f36eead1228e1896df6b8f5183b1"
Oct 30 01:15:15.768356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:15.768336    4425 scope.go:122] "RemoveContainer" containerID="f89ad25bc52947e845446dffb67bf78da4053d087d585c60546668faf6ed45d2"
Oct 30 01:15:15.768748 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:15.768723    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5_odh-ai-gateway-infra(b65a9b4b-56e6-4a60-b6b5-44994c564061)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5" podUID="b65a9b4b-56e6-4a60-b6b5-44994c564061"
Oct 30 01:15:15.770116 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:15.770083    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s" event={"ID":"50a5c7e3-26ea-42d6-b760-2ac74b5bdda1","Type":"ContainerStarted","Data":"c2c3e91f9224de206a584ce7e15d5b0de34b92d221f300c56f353dd98d17541c"}
Oct 30 01:15:15.770239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:15.770139    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s" event={"ID":"50a5c7e3-26ea-42d6-b760-2ac74b5bdda1","Type":"ContainerStarted","Data":"c2cd6bfd36a8b426507f4f897a80185d2b6fb04a7b5269b6564043e029c8f97c"}
Oct 30 01:15:15.770348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:15.770329    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"
Oct 30 01:15:15.842441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:15.842293    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s" podStartSLOduration=1.842271904 podStartE2EDuration="1.842271904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:15.840685544 +0000 UTC m=+2024.590340853" watchObservedRunningTime="2026-10-30 01:15:15.842271904 +0000 UTC m=+2024.591927211"
Oct 30 01:15:15.943356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:15.943329    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"
Oct 30 01:15:16.106807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.106769    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11af40fe-8b0c-4d7a-a1a9-46a491e85424-kube-api-access-tjnz8\" (UniqueName: \"kubernetes.io/projected/11af40fe-8b0c-4d7a-a1a9-46a491e85424-kube-api-access-tjnz8\") pod \"11af40fe-8b0c-4d7a-a1a9-46a491e85424\" (UID: \"11af40fe-8b0c-4d7a-a1a9-46a491e85424\") "
Oct 30 01:15:16.106983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.106964    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11af40fe-8b0c-4d7a-a1a9-46a491e85424-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11af40fe-8b0c-4d7a-a1a9-46a491e85424-maas-api-openshift-ca\") pod \"11af40fe-8b0c-4d7a-a1a9-46a491e85424\" (UID: \"11af40fe-8b0c-4d7a-a1a9-46a491e85424\") "
Oct 30 01:15:16.107140 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.107124    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11af40fe-8b0c-4d7a-a1a9-46a491e85424-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11af40fe-8b0c-4d7a-a1a9-46a491e85424-maas-api-xks-ca\") pod \"11af40fe-8b0c-4d7a-a1a9-46a491e85424\" (UID: \"11af40fe-8b0c-4d7a-a1a9-46a491e85424\") "
Oct 30 01:15:16.107453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.107384    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11af40fe-8b0c-4d7a-a1a9-46a491e85424-maas-api-openshift-ca" pod "11af40fe-8b0c-4d7a-a1a9-46a491e85424" (UID: "11af40fe-8b0c-4d7a-a1a9-46a491e85424"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:16.107648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.107626    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11af40fe-8b0c-4d7a-a1a9-46a491e85424-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:16.109242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.109217    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11af40fe-8b0c-4d7a-a1a9-46a491e85424-kube-api-access-tjnz8" pod "11af40fe-8b0c-4d7a-a1a9-46a491e85424" (UID: "11af40fe-8b0c-4d7a-a1a9-46a491e85424"). InnerVolumeSpecName "kube-api-access-tjnz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:16.109344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.109239    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11af40fe-8b0c-4d7a-a1a9-46a491e85424-maas-api-xks-ca" pod "11af40fe-8b0c-4d7a-a1a9-46a491e85424" (UID: "11af40fe-8b0c-4d7a-a1a9-46a491e85424"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:15:16.208801 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.208711    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11af40fe-8b0c-4d7a-a1a9-46a491e85424-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:16.208801 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.208748    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjnz8\" (UniqueName: \"kubernetes.io/projected/11af40fe-8b0c-4d7a-a1a9-46a491e85424-kube-api-access-tjnz8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:16.670496 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.670463    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"
Oct 30 01:15:16.670651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.670537    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"
Oct 30 01:15:16.784769 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.784725    4425 generic.go:356] "Generic (PLEG): container finished" podID="365c7093-230d-487b-a582-1feacf89487d" containerID="c02de1fff933d46d3b671b1d4d9833fd588b23ad63eb3029760453255f2314ee" exitCode=60
Oct 30 01:15:16.785276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.784790    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr" event={"ID":"365c7093-230d-487b-a582-1feacf89487d","Type":"ContainerDied","Data":"c02de1fff933d46d3b671b1d4d9833fd588b23ad63eb3029760453255f2314ee"}
Oct 30 01:15:16.785276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.784854    4425 scope.go:122] "RemoveContainer" containerID="96c4ae46f7081688b8f440a12916e1fe96962f6e8af58de780362f7e1e8f0456"
Oct 30 01:15:16.785416 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.785352    4425 scope.go:122] "RemoveContainer" containerID="c02de1fff933d46d3b671b1d4d9833fd588b23ad63eb3029760453255f2314ee"
Oct 30 01:15:16.785735 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:16.785710    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr_odh-ai-gateway-infra(365c7093-230d-487b-a582-1feacf89487d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr" podUID="365c7093-230d-487b-a582-1feacf89487d"
Oct 30 01:15:16.787038 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.787017    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"
Oct 30 01:15:16.787256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.787049    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc" event={"ID":"11af40fe-8b0c-4d7a-a1a9-46a491e85424","Type":"ContainerDied","Data":"e2fa65819c0a4d0a9cfba164d1317f3c04d7381e8a3999616ea954e34e4c867e"}
Oct 30 01:15:16.787344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.787294    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2fa65819c0a4d0a9cfba164d1317f3c04d7381e8a3999616ea954e34e4c867e"
Oct 30 01:15:16.789466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.789432    4425 generic.go:356] "Generic (PLEG): container finished" podID="052185d0-cc6f-4f21-acb3-c634271726a6" containerID="e437f563899241494a5830e6be5b3254c946b9939f6779592bd01096838ebea8" exitCode=60
Oct 30 01:15:16.789581 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.789506    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw" event={"ID":"052185d0-cc6f-4f21-acb3-c634271726a6","Type":"ContainerDied","Data":"e437f563899241494a5830e6be5b3254c946b9939f6779592bd01096838ebea8"}
Oct 30 01:15:16.790049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.790032    4425 scope.go:122] "RemoveContainer" containerID="e437f563899241494a5830e6be5b3254c946b9939f6779592bd01096838ebea8"
Oct 30 01:15:16.790565 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:16.790483    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw_odh-ai-gateway-infra(052185d0-cc6f-4f21-acb3-c634271726a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw" podUID="052185d0-cc6f-4f21-acb3-c634271726a6"
Oct 30 01:15:16.800884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.800856    4425 scope.go:122] "RemoveContainer" containerID="3b0b906e9e5db26d017d88df21446f8bfbb7089a8359f3b187aef16b07510acf"
Oct 30 01:15:16.819693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.819653    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5"]
Oct 30 01:15:16.975649 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:16.975618    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5"
Oct 30 01:15:17.019493 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.019457    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b65a9b4b-56e6-4a60-b6b5-44994c564061-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65a9b4b-56e6-4a60-b6b5-44994c564061-maas-api-service-ca\") pod \"b65a9b4b-56e6-4a60-b6b5-44994c564061\" (UID: \"b65a9b4b-56e6-4a60-b6b5-44994c564061\") "
Oct 30 01:15:17.019714 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.019578    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b65a9b4b-56e6-4a60-b6b5-44994c564061-kube-api-access-9qkhf\" (UniqueName: \"kubernetes.io/projected/b65a9b4b-56e6-4a60-b6b5-44994c564061-kube-api-access-9qkhf\") pod \"b65a9b4b-56e6-4a60-b6b5-44994c564061\" (UID: \"b65a9b4b-56e6-4a60-b6b5-44994c564061\") "
Oct 30 01:15:17.020054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.019910    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65a9b4b-56e6-4a60-b6b5-44994c564061-maas-api-service-ca" pod "b65a9b4b-56e6-4a60-b6b5-44994c564061" (UID: "b65a9b4b-56e6-4a60-b6b5-44994c564061"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:17.023208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.020226    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b65a9b4b-56e6-4a60-b6b5-44994c564061-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:17.023208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.022598    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65a9b4b-56e6-4a60-b6b5-44994c564061-kube-api-access-9qkhf" pod "b65a9b4b-56e6-4a60-b6b5-44994c564061" (UID: "b65a9b4b-56e6-4a60-b6b5-44994c564061"). InnerVolumeSpecName "kube-api-access-9qkhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:17.121567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.121526    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qkhf\" (UniqueName: \"kubernetes.io/projected/b65a9b4b-56e6-4a60-b6b5-44994c564061-kube-api-access-9qkhf\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:17.415285 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.415196    4425 scope.go:122] "RemoveContainer" containerID="dc42b3d0fe3b3f992ecdcb431fbea09ecfaebdd55ddf7e5783e40f9f3e37ad4f"
Oct 30 01:15:17.415448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.415293    4425 scope.go:122] "RemoveContainer" containerID="09f5696e2e7541b2d493751f30b24947e45654557b115aae86c6210324756932"
Oct 30 01:15:17.415512 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.415481    4425 scope.go:122] "RemoveContainer" containerID="204218ffc644c4466bc44e059a22273fb1c5ebeeaac1c7ad47177e3b551ff856"
Oct 30 01:15:17.415753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.415731    4425 scope.go:122] "RemoveContainer" containerID="027b7ef09627a8fe77b16fa859301f8aebdf7b676acb921120c884731f1e85d2"
Oct 30 01:15:17.680517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.680486    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"
Oct 30 01:15:17.680696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.680569    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"
Oct 30 01:15:17.801306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.801276    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5"
Oct 30 01:15:17.801306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.801289    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5" event={"ID":"b65a9b4b-56e6-4a60-b6b5-44994c564061","Type":"ContainerDied","Data":"88b6804d1d2308bc23098098cc7acba25432292c16b0ee0a6c3eb51a927e80cd"}
Oct 30 01:15:17.801825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.801363    4425 scope.go:122] "RemoveContainer" containerID="f89ad25bc52947e845446dffb67bf78da4053d087d585c60546668faf6ed45d2"
Oct 30 01:15:17.837880 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.837838    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr"]
Oct 30 01:15:17.853974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.853920    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5"]
Oct 30 01:15:17.861579 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.861529    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888715-pzkf5"]
Oct 30 01:15:17.886554 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:17.886513    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw"]
Oct 30 01:15:18.218934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.218853    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr"
Oct 30 01:15:18.224590 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.224529    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw"
Oct 30 01:15:18.246179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.246123    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365c7093-230d-487b-a582-1feacf89487d-kube-api-access-zb5mg\" (UniqueName: \"kubernetes.io/projected/365c7093-230d-487b-a582-1feacf89487d-kube-api-access-zb5mg\") pod \"365c7093-230d-487b-a582-1feacf89487d\" (UID: \"365c7093-230d-487b-a582-1feacf89487d\") "
Oct 30 01:15:18.246375 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.246281    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/052185d0-cc6f-4f21-acb3-c634271726a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052185d0-cc6f-4f21-acb3-c634271726a6-maas-api-service-ca\") pod \"052185d0-cc6f-4f21-acb3-c634271726a6\" (UID: \"052185d0-cc6f-4f21-acb3-c634271726a6\") "
Oct 30 01:15:18.246465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.246443    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365c7093-230d-487b-a582-1feacf89487d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/365c7093-230d-487b-a582-1feacf89487d-maas-api-service-ca\") pod \"365c7093-230d-487b-a582-1feacf89487d\" (UID: \"365c7093-230d-487b-a582-1feacf89487d\") "
Oct 30 01:15:18.246687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.246661    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052185d0-cc6f-4f21-acb3-c634271726a6-kube-api-access-nfjcl\" (UniqueName: \"kubernetes.io/projected/052185d0-cc6f-4f21-acb3-c634271726a6-kube-api-access-nfjcl\") pod \"052185d0-cc6f-4f21-acb3-c634271726a6\" (UID: \"052185d0-cc6f-4f21-acb3-c634271726a6\") "
Oct 30 01:15:18.247151 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.246866    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365c7093-230d-487b-a582-1feacf89487d-maas-api-service-ca" pod "365c7093-230d-487b-a582-1feacf89487d" (UID: "365c7093-230d-487b-a582-1feacf89487d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:18.247151 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.246865    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052185d0-cc6f-4f21-acb3-c634271726a6-maas-api-service-ca" pod "052185d0-cc6f-4f21-acb3-c634271726a6" (UID: "052185d0-cc6f-4f21-acb3-c634271726a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:18.247405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.247384    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/052185d0-cc6f-4f21-acb3-c634271726a6-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:18.247490 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.247408    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/365c7093-230d-487b-a582-1feacf89487d-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:18.249113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.249086    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365c7093-230d-487b-a582-1feacf89487d-kube-api-access-zb5mg" pod "365c7093-230d-487b-a582-1feacf89487d" (UID: "365c7093-230d-487b-a582-1feacf89487d"). InnerVolumeSpecName "kube-api-access-zb5mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:18.249598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.249570    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052185d0-cc6f-4f21-acb3-c634271726a6-kube-api-access-nfjcl" pod "052185d0-cc6f-4f21-acb3-c634271726a6" (UID: "052185d0-cc6f-4f21-acb3-c634271726a6"). InnerVolumeSpecName "kube-api-access-nfjcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:18.348976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.348941    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfjcl\" (UniqueName: \"kubernetes.io/projected/052185d0-cc6f-4f21-acb3-c634271726a6-kube-api-access-nfjcl\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:18.349260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.349240    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb5mg\" (UniqueName: \"kubernetes.io/projected/365c7093-230d-487b-a582-1feacf89487d-kube-api-access-zb5mg\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:18.818361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.818262    4425 generic.go:356] "Generic (PLEG): container finished" podID="d696b9da-4e58-4b10-8880-78a0cb2d4e86" containerID="2e9073ac027d5ad1243d3e50386dc21442825689f259cf2e67cce627dd7c83b4" exitCode=60
Oct 30 01:15:18.818361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.818339    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2" event={"ID":"d696b9da-4e58-4b10-8880-78a0cb2d4e86","Type":"ContainerDied","Data":"2e9073ac027d5ad1243d3e50386dc21442825689f259cf2e67cce627dd7c83b4"}
Oct 30 01:15:18.818917 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.818403    4425 scope.go:122] "RemoveContainer" containerID="204218ffc644c4466bc44e059a22273fb1c5ebeeaac1c7ad47177e3b551ff856"
Oct 30 01:15:18.818917 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.818880    4425 scope.go:122] "RemoveContainer" containerID="2e9073ac027d5ad1243d3e50386dc21442825689f259cf2e67cce627dd7c83b4"
Oct 30 01:15:18.819339 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:18.819308    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2_odh-ai-gateway-infra(d696b9da-4e58-4b10-8880-78a0cb2d4e86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2" podUID="d696b9da-4e58-4b10-8880-78a0cb2d4e86"
Oct 30 01:15:18.821952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.821918    4425 generic.go:356] "Generic (PLEG): container finished" podID="08c3da1c-0e85-47da-a3dd-ec211c8faff8" containerID="348fe6cc28df059ae19e92d879127211bdbd56a1ddcf2dd90a7db64cf91f693b" exitCode=60
Oct 30 01:15:18.822079 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.821984    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb" event={"ID":"08c3da1c-0e85-47da-a3dd-ec211c8faff8","Type":"ContainerDied","Data":"348fe6cc28df059ae19e92d879127211bdbd56a1ddcf2dd90a7db64cf91f693b"}
Oct 30 01:15:18.822438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.822417    4425 scope.go:122] "RemoveContainer" containerID="348fe6cc28df059ae19e92d879127211bdbd56a1ddcf2dd90a7db64cf91f693b"
Oct 30 01:15:18.822803 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:18.822779    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb_odh-ai-gateway-infra(08c3da1c-0e85-47da-a3dd-ec211c8faff8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb" podUID="08c3da1c-0e85-47da-a3dd-ec211c8faff8"
Oct 30 01:15:18.824514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.824487    4425 generic.go:356] "Generic (PLEG): container finished" podID="ceda3eee-1d3b-4306-9943-04197e1833b5" containerID="9ffc2e77830b32e42099ae64af14e7844fd9836b8b85fd611a82fda5330406f9" exitCode=60
Oct 30 01:15:18.824619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.824553    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk" event={"ID":"ceda3eee-1d3b-4306-9943-04197e1833b5","Type":"ContainerDied","Data":"9ffc2e77830b32e42099ae64af14e7844fd9836b8b85fd611a82fda5330406f9"}
Oct 30 01:15:18.824990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.824971    4425 scope.go:122] "RemoveContainer" containerID="9ffc2e77830b32e42099ae64af14e7844fd9836b8b85fd611a82fda5330406f9"
Oct 30 01:15:18.825352 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:18.825331    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk_odh-ai-gateway-infra(ceda3eee-1d3b-4306-9943-04197e1833b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk" podUID="ceda3eee-1d3b-4306-9943-04197e1833b5"
Oct 30 01:15:18.826298 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.826279    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr"
Oct 30 01:15:18.826412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.826301    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr" event={"ID":"365c7093-230d-487b-a582-1feacf89487d","Type":"ContainerDied","Data":"8f8dd99c9150f02eb26c8b47015dcc3e9ed226c4c444f24531e0fe936a55506d"}
Oct 30 01:15:18.828710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.828690    4425 generic.go:356] "Generic (PLEG): container finished" podID="315812a7-031f-48c3-a854-9aca8e8c63f8" containerID="7d0d418110ecf111ef0bd848ab2e2c1be428c4b63747c49e4d126b48860773be" exitCode=60
Oct 30 01:15:18.828813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.828758    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4" event={"ID":"315812a7-031f-48c3-a854-9aca8e8c63f8","Type":"ContainerDied","Data":"7d0d418110ecf111ef0bd848ab2e2c1be428c4b63747c49e4d126b48860773be"}
Oct 30 01:15:18.829257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.829232    4425 scope.go:122] "RemoveContainer" containerID="7d0d418110ecf111ef0bd848ab2e2c1be428c4b63747c49e4d126b48860773be"
Oct 30 01:15:18.829599 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:18.829573    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4_odh-ai-gateway-infra(315812a7-031f-48c3-a854-9aca8e8c63f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4" podUID="315812a7-031f-48c3-a854-9aca8e8c63f8"
Oct 30 01:15:18.830482 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.830466    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw"
Oct 30 01:15:18.830578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.830484    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw" event={"ID":"052185d0-cc6f-4f21-acb3-c634271726a6","Type":"ContainerDied","Data":"801a33f1c6102bccb0723f45a33ee45d7e726c84cb63eef2ecf5ad8b02c06915"}
Oct 30 01:15:18.834833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.834815    4425 scope.go:122] "RemoveContainer" containerID="027b7ef09627a8fe77b16fa859301f8aebdf7b676acb921120c884731f1e85d2"
Oct 30 01:15:18.850002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.849979    4425 scope.go:122] "RemoveContainer" containerID="dc42b3d0fe3b3f992ecdcb431fbea09ecfaebdd55ddf7e5783e40f9f3e37ad4f"
Oct 30 01:15:18.865670 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.865645    4425 scope.go:122] "RemoveContainer" containerID="c02de1fff933d46d3b671b1d4d9833fd588b23ad63eb3029760453255f2314ee"
Oct 30 01:15:18.877452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.877423    4425 scope.go:122] "RemoveContainer" containerID="09f5696e2e7541b2d493751f30b24947e45654557b115aae86c6210324756932"
Oct 30 01:15:18.893696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.893673    4425 scope.go:122] "RemoveContainer" containerID="e437f563899241494a5830e6be5b3254c946b9939f6779592bd01096838ebea8"
Oct 30 01:15:18.928023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.927963    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr"]
Oct 30 01:15:18.932714 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.932682    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-39b241-29888715-9cdkr"]
Oct 30 01:15:18.995054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.995021    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw"]
Oct 30 01:15:18.999765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:18.999733    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888715-2n8nw"]
Oct 30 01:15:19.432593 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:19.432484    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="052185d0-cc6f-4f21-acb3-c634271726a6" path="/var/lib/kubelet/pods/052185d0-cc6f-4f21-acb3-c634271726a6/volumes"
Oct 30 01:15:19.433356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:19.433327    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365c7093-230d-487b-a582-1feacf89487d" path="/var/lib/kubelet/pods/365c7093-230d-487b-a582-1feacf89487d/volumes"
Oct 30 01:15:19.434174 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:19.434133    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b65a9b4b-56e6-4a60-b6b5-44994c564061" path="/var/lib/kubelet/pods/b65a9b4b-56e6-4a60-b6b5-44994c564061/volumes"
Oct 30 01:15:19.867028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:19.866986    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2"]
Oct 30 01:15:19.905955 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:19.905918    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk"]
Oct 30 01:15:19.979236 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:19.979189    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb"]
Oct 30 01:15:20.050413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.050381    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4"]
Oct 30 01:15:20.085000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.083723    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2"
Oct 30 01:15:20.184234 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.181606    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d696b9da-4e58-4b10-8880-78a0cb2d4e86-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d696b9da-4e58-4b10-8880-78a0cb2d4e86-maas-api-service-ca\") pod \"d696b9da-4e58-4b10-8880-78a0cb2d4e86\" (UID: \"d696b9da-4e58-4b10-8880-78a0cb2d4e86\") "
Oct 30 01:15:20.184234 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.181686    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d696b9da-4e58-4b10-8880-78a0cb2d4e86-kube-api-access-9pxhj\" (UniqueName: \"kubernetes.io/projected/d696b9da-4e58-4b10-8880-78a0cb2d4e86-kube-api-access-9pxhj\") pod \"d696b9da-4e58-4b10-8880-78a0cb2d4e86\" (UID: \"d696b9da-4e58-4b10-8880-78a0cb2d4e86\") "
Oct 30 01:15:20.184234 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.183071    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d696b9da-4e58-4b10-8880-78a0cb2d4e86-maas-api-service-ca" pod "d696b9da-4e58-4b10-8880-78a0cb2d4e86" (UID: "d696b9da-4e58-4b10-8880-78a0cb2d4e86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:20.190356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.190314    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d696b9da-4e58-4b10-8880-78a0cb2d4e86-kube-api-access-9pxhj" pod "d696b9da-4e58-4b10-8880-78a0cb2d4e86" (UID: "d696b9da-4e58-4b10-8880-78a0cb2d4e86"). InnerVolumeSpecName "kube-api-access-9pxhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:20.224102 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.224065    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk"
Oct 30 01:15:20.272537 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.272508    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb"
Oct 30 01:15:20.275909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.275880    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4"
Oct 30 01:15:20.283203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.283139    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceda3eee-1d3b-4306-9943-04197e1833b5-kube-api-access-42gmc\" (UniqueName: \"kubernetes.io/projected/ceda3eee-1d3b-4306-9943-04197e1833b5-kube-api-access-42gmc\") pod \"ceda3eee-1d3b-4306-9943-04197e1833b5\" (UID: \"ceda3eee-1d3b-4306-9943-04197e1833b5\") "
Oct 30 01:15:20.283413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.283255    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceda3eee-1d3b-4306-9943-04197e1833b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceda3eee-1d3b-4306-9943-04197e1833b5-maas-api-service-ca\") pod \"ceda3eee-1d3b-4306-9943-04197e1833b5\" (UID: \"ceda3eee-1d3b-4306-9943-04197e1833b5\") "
Oct 30 01:15:20.283644 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.283616    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceda3eee-1d3b-4306-9943-04197e1833b5-maas-api-service-ca" pod "ceda3eee-1d3b-4306-9943-04197e1833b5" (UID: "ceda3eee-1d3b-4306-9943-04197e1833b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:20.284092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.284067    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d696b9da-4e58-4b10-8880-78a0cb2d4e86-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.284191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.284099    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceda3eee-1d3b-4306-9943-04197e1833b5-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.284191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.284114    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pxhj\" (UniqueName: \"kubernetes.io/projected/d696b9da-4e58-4b10-8880-78a0cb2d4e86-kube-api-access-9pxhj\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.285719 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.285685    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceda3eee-1d3b-4306-9943-04197e1833b5-kube-api-access-42gmc" pod "ceda3eee-1d3b-4306-9943-04197e1833b5" (UID: "ceda3eee-1d3b-4306-9943-04197e1833b5"). InnerVolumeSpecName "kube-api-access-42gmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:20.384993 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.384954    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c3da1c-0e85-47da-a3dd-ec211c8faff8-kube-api-access-8b88s\" (UniqueName: \"kubernetes.io/projected/08c3da1c-0e85-47da-a3dd-ec211c8faff8-kube-api-access-8b88s\") pod \"08c3da1c-0e85-47da-a3dd-ec211c8faff8\" (UID: \"08c3da1c-0e85-47da-a3dd-ec211c8faff8\") "
Oct 30 01:15:20.385246 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.385098    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c3da1c-0e85-47da-a3dd-ec211c8faff8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c3da1c-0e85-47da-a3dd-ec211c8faff8-maas-api-service-ca\") pod \"08c3da1c-0e85-47da-a3dd-ec211c8faff8\" (UID: \"08c3da1c-0e85-47da-a3dd-ec211c8faff8\") "
Oct 30 01:15:20.385246 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.385130    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/315812a7-031f-48c3-a854-9aca8e8c63f8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/315812a7-031f-48c3-a854-9aca8e8c63f8-maas-api-service-ca\") pod \"315812a7-031f-48c3-a854-9aca8e8c63f8\" (UID: \"315812a7-031f-48c3-a854-9aca8e8c63f8\") "
Oct 30 01:15:20.385246 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.385225    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/315812a7-031f-48c3-a854-9aca8e8c63f8-kube-api-access-9hxgn\" (UniqueName: \"kubernetes.io/projected/315812a7-031f-48c3-a854-9aca8e8c63f8-kube-api-access-9hxgn\") pod \"315812a7-031f-48c3-a854-9aca8e8c63f8\" (UID: \"315812a7-031f-48c3-a854-9aca8e8c63f8\") "
Oct 30 01:15:20.385509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.385484    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c3da1c-0e85-47da-a3dd-ec211c8faff8-maas-api-service-ca" pod "08c3da1c-0e85-47da-a3dd-ec211c8faff8" (UID: "08c3da1c-0e85-47da-a3dd-ec211c8faff8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:20.385586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.385558    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/315812a7-031f-48c3-a854-9aca8e8c63f8-maas-api-service-ca" pod "315812a7-031f-48c3-a854-9aca8e8c63f8" (UID: "315812a7-031f-48c3-a854-9aca8e8c63f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:20.385786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.385770    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c3da1c-0e85-47da-a3dd-ec211c8faff8-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.385851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.385792    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/315812a7-031f-48c3-a854-9aca8e8c63f8-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.385851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.385807    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42gmc\" (UniqueName: \"kubernetes.io/projected/ceda3eee-1d3b-4306-9943-04197e1833b5-kube-api-access-42gmc\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.387439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.387409    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/315812a7-031f-48c3-a854-9aca8e8c63f8-kube-api-access-9hxgn" pod "315812a7-031f-48c3-a854-9aca8e8c63f8" (UID: "315812a7-031f-48c3-a854-9aca8e8c63f8"). InnerVolumeSpecName "kube-api-access-9hxgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:20.387581 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.387455    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c3da1c-0e85-47da-a3dd-ec211c8faff8-kube-api-access-8b88s" pod "08c3da1c-0e85-47da-a3dd-ec211c8faff8" (UID: "08c3da1c-0e85-47da-a3dd-ec211c8faff8"). InnerVolumeSpecName "kube-api-access-8b88s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:20.487580 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.487484    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b88s\" (UniqueName: \"kubernetes.io/projected/08c3da1c-0e85-47da-a3dd-ec211c8faff8-kube-api-access-8b88s\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.487580 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.487522    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hxgn\" (UniqueName: \"kubernetes.io/projected/315812a7-031f-48c3-a854-9aca8e8c63f8-kube-api-access-9hxgn\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:15:20.858437 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.858400    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4"
Oct 30 01:15:20.858437 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.858425    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4" event={"ID":"315812a7-031f-48c3-a854-9aca8e8c63f8","Type":"ContainerDied","Data":"0f28ac014c729ddf93db163656809d0165165c803b7384162fd1de546f3d4491"}
Oct 30 01:15:20.858714 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.858496    4425 scope.go:122] "RemoveContainer" containerID="7d0d418110ecf111ef0bd848ab2e2c1be428c4b63747c49e4d126b48860773be"
Oct 30 01:15:20.860402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.860287    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2"
Oct 30 01:15:20.860402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.860331    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2" event={"ID":"d696b9da-4e58-4b10-8880-78a0cb2d4e86","Type":"ContainerDied","Data":"574c62ee4f259c3c6021c27d13d69e5ab3944b068e5d252d427985678b69b9e3"}
Oct 30 01:15:20.862225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.862190    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb" event={"ID":"08c3da1c-0e85-47da-a3dd-ec211c8faff8","Type":"ContainerDied","Data":"aace3236b5b6784e9651e04f56aa166140a05ea586df4aa3c382bb6469d6d5c0"}
Oct 30 01:15:20.862380 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.862200    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb"
Oct 30 01:15:20.864031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.863937    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk"
Oct 30 01:15:20.864031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.863971    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk" event={"ID":"ceda3eee-1d3b-4306-9943-04197e1833b5","Type":"ContainerDied","Data":"67cd96a22c799038a322b8a644ce9774ea47da0b2d07eb0f71a5b64f80961900"}
Oct 30 01:15:20.874232 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.873907    4425 scope.go:122] "RemoveContainer" containerID="2e9073ac027d5ad1243d3e50386dc21442825689f259cf2e67cce627dd7c83b4"
Oct 30 01:15:20.886804 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.886775    4425 scope.go:122] "RemoveContainer" containerID="348fe6cc28df059ae19e92d879127211bdbd56a1ddcf2dd90a7db64cf91f693b"
Oct 30 01:15:20.897931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.897905    4425 scope.go:122] "RemoveContainer" containerID="9ffc2e77830b32e42099ae64af14e7844fd9836b8b85fd611a82fda5330406f9"
Oct 30 01:15:20.947020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.946979    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2"]
Oct 30 01:15:20.956567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.956525    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-df4db1-29888715-hp8j2"]
Oct 30 01:15:20.982394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.982354    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb"]
Oct 30 01:15:20.992908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:20.992869    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888715-cxbcb"]
Oct 30 01:15:21.022219 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:21.022173    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4"]
Oct 30 01:15:21.033014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:21.032974    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888715-r5lz4"]
Oct 30 01:15:21.067207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:21.067144    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk"]
Oct 30 01:15:21.071752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:21.071711    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888715-72khk"]
Oct 30 01:15:21.424891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:21.424788    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c3da1c-0e85-47da-a3dd-ec211c8faff8" path="/var/lib/kubelet/pods/08c3da1c-0e85-47da-a3dd-ec211c8faff8/volumes"
Oct 30 01:15:21.425530 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:21.425504    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="315812a7-031f-48c3-a854-9aca8e8c63f8" path="/var/lib/kubelet/pods/315812a7-031f-48c3-a854-9aca8e8c63f8/volumes"
Oct 30 01:15:21.426003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:21.425983    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceda3eee-1d3b-4306-9943-04197e1833b5" path="/var/lib/kubelet/pods/ceda3eee-1d3b-4306-9943-04197e1833b5/volumes"
Oct 30 01:15:21.426523 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:21.426502    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d696b9da-4e58-4b10-8880-78a0cb2d4e86" path="/var/lib/kubelet/pods/d696b9da-4e58-4b10-8880-78a0cb2d4e86/volumes"
Oct 30 01:15:26.793384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:26.793351    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"
Oct 30 01:15:33.478541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:33.478504    4425 scope.go:122] "RemoveContainer" containerID="935be13dec14ae8de7adc8d08a48a9400464f2bc5f780437faa3dbf6f632f09b"
Oct 30 01:15:33.490003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:33.489966    4425 scope.go:122] "RemoveContainer" containerID="68da545cc1431df306b1d65c3d0e754d07f225cf81776cf8d9f072ee0584fc06"
Oct 30 01:15:33.500670 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:33.500643    4425 scope.go:122] "RemoveContainer" containerID="598e1939137d46ce745f705fcade7fffe29b83ea1f39d97a141e1159fa37831b"
Oct 30 01:15:37.558305 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:15:37.558259    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:15:38.969463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.969422    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"]
Oct 30 01:15:38.970121 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970103    4425 cpu_manager.go:414] "containerMap: removing container" podUID="315812a7-031f-48c3-a854-9aca8e8c63f8" containerName="cleanup"
Oct 30 01:15:38.970214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970122    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="315812a7-031f-48c3-a854-9aca8e8c63f8" containerName="cleanup"
Oct 30 01:15:38.970214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970135    4425 cpu_manager.go:414] "containerMap: removing container" podUID="052185d0-cc6f-4f21-acb3-c634271726a6" containerName="cleanup"
Oct 30 01:15:38.970214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970142    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052185d0-cc6f-4f21-acb3-c634271726a6" containerName="cleanup"
Oct 30 01:15:38.970214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970148    4425 cpu_manager.go:414] "containerMap: removing container" podUID="052185d0-cc6f-4f21-acb3-c634271726a6" containerName="cleanup"
Oct 30 01:15:38.970214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970166    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052185d0-cc6f-4f21-acb3-c634271726a6" containerName="cleanup"
Oct 30 01:15:38.970214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970180    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ceda3eee-1d3b-4306-9943-04197e1833b5" containerName="cleanup"
Oct 30 01:15:38.970214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970185    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceda3eee-1d3b-4306-9943-04197e1833b5" containerName="cleanup"
Oct 30 01:15:38.970214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970191    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b65a9b4b-56e6-4a60-b6b5-44994c564061" containerName="cleanup"
Oct 30 01:15:38.970214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970195    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65a9b4b-56e6-4a60-b6b5-44994c564061" containerName="cleanup"
Oct 30 01:15:38.970214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970202    4425 cpu_manager.go:414] "containerMap: removing container" podUID="315812a7-031f-48c3-a854-9aca8e8c63f8" containerName="cleanup"
Oct 30 01:15:38.970214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970205    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="315812a7-031f-48c3-a854-9aca8e8c63f8" containerName="cleanup"
Oct 30 01:15:38.970214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970213    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d696b9da-4e58-4b10-8880-78a0cb2d4e86" containerName="cleanup"
Oct 30 01:15:38.970214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970217    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d696b9da-4e58-4b10-8880-78a0cb2d4e86" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970237    4425 cpu_manager.go:414] "containerMap: removing container" podUID="08c3da1c-0e85-47da-a3dd-ec211c8faff8" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970242    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c3da1c-0e85-47da-a3dd-ec211c8faff8" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970255    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d696b9da-4e58-4b10-8880-78a0cb2d4e86" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970262    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d696b9da-4e58-4b10-8880-78a0cb2d4e86" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970272    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b65a9b4b-56e6-4a60-b6b5-44994c564061" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970279    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65a9b4b-56e6-4a60-b6b5-44994c564061" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970293    4425 cpu_manager.go:414] "containerMap: removing container" podUID="08c3da1c-0e85-47da-a3dd-ec211c8faff8" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970298    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c3da1c-0e85-47da-a3dd-ec211c8faff8" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970304    4425 cpu_manager.go:414] "containerMap: removing container" podUID="365c7093-230d-487b-a582-1feacf89487d" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970308    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365c7093-230d-487b-a582-1feacf89487d" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970315    4425 cpu_manager.go:414] "containerMap: removing container" podUID="052185d0-cc6f-4f21-acb3-c634271726a6" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970318    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052185d0-cc6f-4f21-acb3-c634271726a6" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970327    4425 cpu_manager.go:414] "containerMap: removing container" podUID="11af40fe-8b0c-4d7a-a1a9-46a491e85424" containerName="revoke-keys"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970354    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11af40fe-8b0c-4d7a-a1a9-46a491e85424" containerName="revoke-keys"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970363    4425 cpu_manager.go:414] "containerMap: removing container" podUID="365c7093-230d-487b-a582-1feacf89487d" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970371    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365c7093-230d-487b-a582-1feacf89487d" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970387    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b65a9b4b-56e6-4a60-b6b5-44994c564061" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970396    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65a9b4b-56e6-4a60-b6b5-44994c564061" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970406    4425 cpu_manager.go:414] "containerMap: removing container" podUID="365c7093-230d-487b-a582-1feacf89487d" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970415    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365c7093-230d-487b-a582-1feacf89487d" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970432    4425 cpu_manager.go:414] "containerMap: removing container" podUID="08c3da1c-0e85-47da-a3dd-ec211c8faff8" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970440    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c3da1c-0e85-47da-a3dd-ec211c8faff8" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970450    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ceda3eee-1d3b-4306-9943-04197e1833b5" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970455    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceda3eee-1d3b-4306-9943-04197e1833b5" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970592    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="315812a7-031f-48c3-a854-9aca8e8c63f8" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970602    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="052185d0-cc6f-4f21-acb3-c634271726a6" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970608    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="052185d0-cc6f-4f21-acb3-c634271726a6" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970616    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65a9b4b-56e6-4a60-b6b5-44994c564061" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970621    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceda3eee-1d3b-4306-9943-04197e1833b5" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970627    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="315812a7-031f-48c3-a854-9aca8e8c63f8" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970633    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d696b9da-4e58-4b10-8880-78a0cb2d4e86" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970639    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceda3eee-1d3b-4306-9943-04197e1833b5" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970648    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c3da1c-0e85-47da-a3dd-ec211c8faff8" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970654    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65a9b4b-56e6-4a60-b6b5-44994c564061" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970662    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="365c7093-230d-487b-a582-1feacf89487d" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970668    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="052185d0-cc6f-4f21-acb3-c634271726a6" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970674    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="11af40fe-8b0c-4d7a-a1a9-46a491e85424" containerName="revoke-keys"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970680    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="365c7093-230d-487b-a582-1feacf89487d" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970684    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d696b9da-4e58-4b10-8880-78a0cb2d4e86" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970690    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65a9b4b-56e6-4a60-b6b5-44994c564061" containerName="cleanup"
Oct 30 01:15:38.970697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.970698    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c3da1c-0e85-47da-a3dd-ec211c8faff8" containerName="cleanup"
Oct 30 01:15:38.978027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.977985    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"
Oct 30 01:15:38.992525 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.992481    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-bbb402\""
Oct 30 01:15:38.995562 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:38.995529    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"]
Oct 30 01:15:39.047112 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.047063    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"]
Oct 30 01:15:39.048053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.048030    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d696b9da-4e58-4b10-8880-78a0cb2d4e86" containerName="cleanup"
Oct 30 01:15:39.048105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.048056    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d696b9da-4e58-4b10-8880-78a0cb2d4e86" containerName="cleanup"
Oct 30 01:15:39.048191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.048122    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ceda3eee-1d3b-4306-9943-04197e1833b5" containerName="cleanup"
Oct 30 01:15:39.048191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.048130    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceda3eee-1d3b-4306-9943-04197e1833b5" containerName="cleanup"
Oct 30 01:15:39.048191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.048139    4425 cpu_manager.go:414] "containerMap: removing container" podUID="315812a7-031f-48c3-a854-9aca8e8c63f8" containerName="cleanup"
Oct 30 01:15:39.048191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.048147    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="315812a7-031f-48c3-a854-9aca8e8c63f8" containerName="cleanup"
Oct 30 01:15:39.048336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.048326    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceda3eee-1d3b-4306-9943-04197e1833b5" containerName="cleanup"
Oct 30 01:15:39.048373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.048340    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c3da1c-0e85-47da-a3dd-ec211c8faff8" containerName="cleanup"
Oct 30 01:15:39.048373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.048354    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="365c7093-230d-487b-a582-1feacf89487d" containerName="cleanup"
Oct 30 01:15:39.048373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.048363    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d696b9da-4e58-4b10-8880-78a0cb2d4e86" containerName="cleanup"
Oct 30 01:15:39.048514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.048379    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="315812a7-031f-48c3-a854-9aca8e8c63f8" containerName="cleanup"
Oct 30 01:15:39.052433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.052410    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"
Oct 30 01:15:39.070397 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.070362    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-bbb402-dockercfg-8ctbp\""
Oct 30 01:15:39.073842 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.073807    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"]
Oct 30 01:15:39.077473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.077436    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d4143b-72d6-4da0-88dc-f726562e4c30-plugins-config-volume\") pod \"payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx\" (UID: \"46d4143b-72d6-4da0-88dc-f726562e4c30\") " pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"
Oct 30 01:15:39.077652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.077569    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z5mv\" (UniqueName: \"kubernetes.io/projected/d069dc1c-bd34-44a1-a01c-694812788c8f-kube-api-access-8z5mv\") pod \"payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk\" (UID: \"d069dc1c-bd34-44a1-a01c-694812788c8f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"
Oct 30 01:15:39.077723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.077653    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d069dc1c-bd34-44a1-a01c-694812788c8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk\" (UID: \"d069dc1c-bd34-44a1-a01c-694812788c8f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"
Oct 30 01:15:39.077805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.077783    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfpzs\" (UniqueName: \"kubernetes.io/projected/46d4143b-72d6-4da0-88dc-f726562e4c30-kube-api-access-wfpzs\") pod \"payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx\" (UID: \"46d4143b-72d6-4da0-88dc-f726562e4c30\") " pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"
Oct 30 01:15:39.182095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.182043    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d069dc1c-bd34-44a1-a01c-694812788c8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk\" (UID: \"d069dc1c-bd34-44a1-a01c-694812788c8f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"
Oct 30 01:15:39.182400 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.182137    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfpzs\" (UniqueName: \"kubernetes.io/projected/46d4143b-72d6-4da0-88dc-f726562e4c30-kube-api-access-wfpzs\") pod \"payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx\" (UID: \"46d4143b-72d6-4da0-88dc-f726562e4c30\") " pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"
Oct 30 01:15:39.182476 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.182452    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d4143b-72d6-4da0-88dc-f726562e4c30-plugins-config-volume\") pod \"payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx\" (UID: \"46d4143b-72d6-4da0-88dc-f726562e4c30\") " pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"
Oct 30 01:15:39.182538 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.182523    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z5mv\" (UniqueName: \"kubernetes.io/projected/d069dc1c-bd34-44a1-a01c-694812788c8f-kube-api-access-8z5mv\") pod \"payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk\" (UID: \"d069dc1c-bd34-44a1-a01c-694812788c8f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"
Oct 30 01:15:39.184450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.184416    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d4143b-72d6-4da0-88dc-f726562e4c30-plugins-config-volume\") pod \"payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx\" (UID: \"46d4143b-72d6-4da0-88dc-f726562e4c30\") " pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"
Oct 30 01:15:39.185107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.185082    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d069dc1c-bd34-44a1-a01c-694812788c8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk\" (UID: \"d069dc1c-bd34-44a1-a01c-694812788c8f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"
Oct 30 01:15:39.204190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.204144    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfpzs\" (UniqueName: \"kubernetes.io/projected/46d4143b-72d6-4da0-88dc-f726562e4c30-kube-api-access-wfpzs\") pod \"payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx\" (UID: \"46d4143b-72d6-4da0-88dc-f726562e4c30\") " pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"
Oct 30 01:15:39.204389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.204208    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z5mv\" (UniqueName: \"kubernetes.io/projected/d069dc1c-bd34-44a1-a01c-694812788c8f-kube-api-access-8z5mv\") pod \"payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk\" (UID: \"d069dc1c-bd34-44a1-a01c-694812788c8f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"
Oct 30 01:15:39.298295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.298257    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"
Oct 30 01:15:39.367263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.367230    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"
Oct 30 01:15:39.615410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.615371    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"]
Oct 30 01:15:39.615600 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:15:39.615572    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d4143b_72d6_4da0_88dc_f726562e4c30.slice/crio-84b4075609c5c5b122db6549b8b844241bc8aabfa6ab2f2d0db905643f23ff02 WatchSource:0}: Error finding container 84b4075609c5c5b122db6549b8b844241bc8aabfa6ab2f2d0db905643f23ff02: Status 404 returned error can't find the container with id 84b4075609c5c5b122db6549b8b844241bc8aabfa6ab2f2d0db905643f23ff02
Oct 30 01:15:39.706943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:39.706788    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"]
Oct 30 01:15:39.713845 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:15:39.713812    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd069dc1c_bd34_44a1_a01c_694812788c8f.slice/crio-f16e369ceef6514b71064a142d967bf879086d1b1a5c7ce6d25868f8819f5658 WatchSource:0}: Error finding container f16e369ceef6514b71064a142d967bf879086d1b1a5c7ce6d25868f8819f5658: Status 404 returned error can't find the container with id f16e369ceef6514b71064a142d967bf879086d1b1a5c7ce6d25868f8819f5658
Oct 30 01:15:40.051265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:40.051213    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk" event={"ID":"d069dc1c-bd34-44a1-a01c-694812788c8f","Type":"ContainerStarted","Data":"ee5f027fcd6a6be50e1465d2f931ba8fe2112de2f1b342a2b12333185b733801"}
Oct 30 01:15:40.051829 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:40.051296    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk" event={"ID":"d069dc1c-bd34-44a1-a01c-694812788c8f","Type":"ContainerStarted","Data":"f16e369ceef6514b71064a142d967bf879086d1b1a5c7ce6d25868f8819f5658"}
Oct 30 01:15:40.051829 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:40.051368    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"
Oct 30 01:15:40.056187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:40.054713    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx" event={"ID":"46d4143b-72d6-4da0-88dc-f726562e4c30","Type":"ContainerStarted","Data":"2d2dd0c2c926339b065eb19eb57e2e293948d244e52fc176353203874cd867ca"}
Oct 30 01:15:40.056187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:40.054783    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx" event={"ID":"46d4143b-72d6-4da0-88dc-f726562e4c30","Type":"ContainerStarted","Data":"84b4075609c5c5b122db6549b8b844241bc8aabfa6ab2f2d0db905643f23ff02"}
Oct 30 01:15:40.056187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:40.054874    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"
Oct 30 01:15:40.094098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:40.094041    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk" podStartSLOduration=1.094025903 podStartE2EDuration="1.094025903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:40.086982575 +0000 UTC m=+2048.836637884" watchObservedRunningTime="2026-10-30 01:15:40.094025903 +0000 UTC m=+2048.843681212"
Oct 30 01:15:40.125527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:40.125452    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx" podStartSLOduration=2.125433322 podStartE2EDuration="2.125433322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:40.123692794 +0000 UTC m=+2048.873348097" watchObservedRunningTime="2026-10-30 01:15:40.125433322 +0000 UTC m=+2048.875088634"
Oct 30 01:15:41.903043 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:41.902991    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"]
Oct 30 01:15:41.912457 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:41.912418    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:41.930646 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:41.930615    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-0e1a98\"/\"kube-root-ca.crt\""
Oct 30 01:15:41.930847 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:41.930705    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-0e1a98\"/\"default-dockercfg-qhcq2\""
Oct 30 01:15:41.931078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:41.931049    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"]
Oct 30 01:15:41.935947 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:41.935916    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-0e1a98\"/\"openshift-service-ca.crt\""
Oct 30 01:15:41.936111 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:41.936066    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-0e1a98\"/\"auth-test-model-0e1a98-kserve-self-signed-certs\""
Oct 30 01:15:42.029550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.029500    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-home\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.029725 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.029568    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-model-cache\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.029725 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.029672    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3477fee-0b54-47ee-9a25-b1c67c27e698-tls-certs\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.029842 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.029815    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-dshm\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.029928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.029912    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-tmp-dir\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.030116 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.030099    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26djw\" (UniqueName: \"kubernetes.io/projected/e3477fee-0b54-47ee-9a25-b1c67c27e698-kube-api-access-26djw\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.131384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.131330    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26djw\" (UniqueName: \"kubernetes.io/projected/e3477fee-0b54-47ee-9a25-b1c67c27e698-kube-api-access-26djw\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.131607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.131423    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-home\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.131607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.131458    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-model-cache\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.131607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.131574    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3477fee-0b54-47ee-9a25-b1c67c27e698-tls-certs\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.131792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.131686    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-dshm\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.131792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.131755    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-tmp-dir\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.131893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.131865    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-home\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.131893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.131872    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-model-cache\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.132251 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.132226    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-tmp-dir\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.134052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.134031    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-dshm\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.134392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.134370    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3477fee-0b54-47ee-9a25-b1c67c27e698-tls-certs\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.157032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.156867    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26djw\" (UniqueName: \"kubernetes.io/projected/e3477fee-0b54-47ee-9a25-b1c67c27e698-kube-api-access-26djw\") pod \"auth-test-model-0e1a98-kserve-865fbdc49-79g6c\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.240631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.240595    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:42.540431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.540390    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:15:42.607681 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:42.607649    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"]
Oct 30 01:15:42.608352 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:15:42.608319    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3477fee_0b54_47ee_9a25_b1c67c27e698.slice/crio-a7b20ed968526bac4936161ec7dfcea252bb826adaae2b133b12c88eea889a06 WatchSource:0}: Error finding container a7b20ed968526bac4936161ec7dfcea252bb826adaae2b133b12c88eea889a06: Status 404 returned error can't find the container with id a7b20ed968526bac4936161ec7dfcea252bb826adaae2b133b12c88eea889a06
Oct 30 01:15:43.082425 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:43.082384    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c" event={"ID":"e3477fee-0b54-47ee-9a25-b1c67c27e698","Type":"ContainerStarted","Data":"b1c809ee3967409ab4a441fa70ee170caea41ea5aff3b9b3d0081c80b8115a22"}
Oct 30 01:15:43.082910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:43.082436    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c" event={"ID":"e3477fee-0b54-47ee-9a25-b1c67c27e698","Type":"ContainerStarted","Data":"a7b20ed968526bac4936161ec7dfcea252bb826adaae2b133b12c88eea889a06"}
Oct 30 01:15:43.120549 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:43.120488    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c" podStartSLOduration=2.120471177 podStartE2EDuration="2.120471177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:43.119377506 +0000 UTC m=+2051.869032816" watchObservedRunningTime="2026-10-30 01:15:43.120471177 +0000 UTC m=+2051.870126530"
Oct 30 01:15:43.241822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:43.241776    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:45.856297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:45.856255    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"]
Oct 30 01:15:45.862035 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:45.861980    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:45.874999 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:45.874963    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d5b2b8-openshift-default-dockercfg-br5n4\""
Oct 30 01:15:45.890292 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:45.890250    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"]
Oct 30 01:15:45.988897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:45.988857    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:45.989069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:45.988947    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:45.989069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:45.988972    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:45.989252 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:45.989112    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:45.989252 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:45.989173    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:45.989381 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:45.989289    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:45.989381 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:45.989344    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:45.989490 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:45.989434    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxn52\" (UniqueName: \"kubernetes.io/projected/01641f6d-e95b-4aeb-88db-8b5154e79bfe-kube-api-access-lxn52\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:45.989490 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:45.989480    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.090660 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.090621    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.090825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.090680    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.090825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.090715    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.090825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.090759    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.090825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.090804    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxn52\" (UniqueName: \"kubernetes.io/projected/01641f6d-e95b-4aeb-88db-8b5154e79bfe-kube-api-access-lxn52\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.091080 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.090849    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.091080 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.090888    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.091080 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.090995    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.091080 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.091024    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.091308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.091081    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-workload-certs\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.091308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.091116    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-credential-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.091308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.091185    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-data\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.091611 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.091539    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-workload-socket\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.091898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.091879    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istiod-ca-cert\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.093405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.093379    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-podinfo\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.093519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.093427    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-envoy\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.113027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.112946    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-token\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.113552 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.113529    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxn52\" (UniqueName: \"kubernetes.io/projected/01641f6d-e95b-4aeb-88db-8b5154e79bfe-kube-api-access-lxn52\") pod \"e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") " pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.179218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.179176    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:46.509931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.509892    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"]
Oct 30 01:15:46.511029 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:15:46.510994    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01641f6d_e95b_4aeb_88db_8b5154e79bfe.slice/crio-84159016c4af34cecf3be1885c5a14546b17ee1d53b67a86efda54c11e620e4a WatchSource:0}: Error finding container 84159016c4af34cecf3be1885c5a14546b17ee1d53b67a86efda54c11e620e4a: Status 404 returned error can't find the container with id 84159016c4af34cecf3be1885c5a14546b17ee1d53b67a86efda54c11e620e4a
Oct 30 01:15:46.514102 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.514057    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:15:46.514216 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.514139    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:15:46.514270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:46.514215    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:15:47.127145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:47.127103    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp" event={"ID":"01641f6d-e95b-4aeb-88db-8b5154e79bfe","Type":"ContainerStarted","Data":"cda91935b4a1719fc24d08266d90b4a84eeac9b713e9908b8ca10607c00286a9"}
Oct 30 01:15:47.127145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:47.127184    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp" event={"ID":"01641f6d-e95b-4aeb-88db-8b5154e79bfe","Type":"ContainerStarted","Data":"84159016c4af34cecf3be1885c5a14546b17ee1d53b67a86efda54c11e620e4a"}
Oct 30 01:15:47.183177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:47.182456    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:47.185184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:47.185039    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:15:47.185184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:47.185093    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp" podUID="01641f6d-e95b-4aeb-88db-8b5154e79bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 01:15:48.180325 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:48.180282    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:15:48.180759 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:48.180380    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp" podUID="01641f6d-e95b-4aeb-88db-8b5154e79bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 01:15:49.180512 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:49.180469    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:15:49.181040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:49.180580    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp" podUID="01641f6d-e95b-4aeb-88db-8b5154e79bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 01:15:50.180971 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:50.180932    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:15:50.181594 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:50.180996    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp" podUID="01641f6d-e95b-4aeb-88db-8b5154e79bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 01:15:51.075235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:51.075202    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"
Oct 30 01:15:51.075564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:51.075372    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"
Oct 30 01:15:51.115764 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:51.115702    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp" podStartSLOduration=6.115685941 podStartE2EDuration="6.115685941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:47.187540533 +0000 UTC m=+2055.937195843" watchObservedRunningTime="2026-10-30 01:15:51.115685941 +0000 UTC m=+2059.865341250"
Oct 30 01:15:51.181713 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:51.181667    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:15:51.182258 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:51.181771    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp" podUID="01641f6d-e95b-4aeb-88db-8b5154e79bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 01:15:52.180222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:52.180179    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:15:52.180460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:52.180246    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp" podUID="01641f6d-e95b-4aeb-88db-8b5154e79bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 01:15:52.242176 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:52.241813    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:52.292796 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:52.292752    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:53.180386 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:53.180339    4425 patch_prober.go:29] interesting pod/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 01:15:53.180634 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:53.180418    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp" podUID="01641f6d-e95b-4aeb-88db-8b5154e79bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 01:15:53.222990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:53.222956    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:15:54.187662 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:54.187626    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:54.188211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:54.188150    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:15:54.189007 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:15:54.188987    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:16:06.796947 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.796905    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"]
Oct 30 01:16:06.802243 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.802217    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.825286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.825249    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-8ae41e\"/\"openshift-service-ca.crt\""
Oct 30 01:16:06.825286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.825262    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-8ae41e\"/\"kube-root-ca.crt\""
Oct 30 01:16:06.825550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.825424    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-8ae41e\"/\"default-dockercfg-vbqhs\""
Oct 30 01:16:06.825550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.825516    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-8ae41e\"/\"auth-test-model-8ae41e-kserve-self-signed-certs\""
Oct 30 01:16:06.835682 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.835099    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"]
Oct 30 01:16:06.862605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.862552    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-dshm\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.862787 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.862681    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-model-cache\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.862787 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.862720    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-tmp-dir\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.862787 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.862748    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9j8f\" (UniqueName: \"kubernetes.io/projected/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-kube-api-access-f9j8f\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.862903 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.862845    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-tls-certs\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.863002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.862976    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-home\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.964993 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.964942    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-dshm\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.965227 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.965050    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-model-cache\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.965227 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.965105    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-tmp-dir\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.965227 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.965138    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9j8f\" (UniqueName: \"kubernetes.io/projected/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-kube-api-access-f9j8f\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.965389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.965241    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-tls-certs\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.965389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.965357    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-home\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.965540 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.965511    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-tmp-dir\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.965683 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.965661    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-model-cache\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.965804 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.965780    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-home\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.968254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.968226    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-dshm\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.968380 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.968309    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-tls-certs\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:06.986963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:06.986932    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9j8f\" (UniqueName: \"kubernetes.io/projected/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-kube-api-access-f9j8f\") pod \"auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") " pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:07.123841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:07.123740    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:07.482320 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:07.482281    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"]
Oct 30 01:16:07.483337 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:16:07.483308    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cc6610f_e2c9_4802_bba2_e3df8f40c9e0.slice/crio-1f103f5503f4e553219fabeefcf56e0fd3100f1d4b2c2fc6bf5a1532d71eff18 WatchSource:0}: Error finding container 1f103f5503f4e553219fabeefcf56e0fd3100f1d4b2c2fc6bf5a1532d71eff18: Status 404 returned error can't find the container with id 1f103f5503f4e553219fabeefcf56e0fd3100f1d4b2c2fc6bf5a1532d71eff18
Oct 30 01:16:07.559428 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:16:07.559390    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:16:08.359436 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:08.359392    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv" event={"ID":"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0","Type":"ContainerStarted","Data":"a56f8d29e4a1e58e5a4896247d50d595b5c5f2df36d83d76c1a29f74a642d07f"}
Oct 30 01:16:08.359883 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:08.359448    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv" event={"ID":"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0","Type":"ContainerStarted","Data":"1f103f5503f4e553219fabeefcf56e0fd3100f1d4b2c2fc6bf5a1532d71eff18"}
Oct 30 01:16:08.400511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:08.400454    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv" podStartSLOduration=2.400437266 podStartE2EDuration="2.400437266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:08.395416062 +0000 UTC m=+2077.145071382" watchObservedRunningTime="2026-10-30 01:16:08.400437266 +0000 UTC m=+2077.150092575"
Oct 30 01:16:09.029844 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:09.029811    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"]
Oct 30 01:16:09.039523 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:09.039476    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be79cd53-nkbs5"]
Oct 30 01:16:09.125639 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:09.125608    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:09.425427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:09.425281    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e225afe-9707-464a-8d42-20c33b7c5390" path="/var/lib/kubelet/pods/2e225afe-9707-464a-8d42-20c33b7c5390/volumes"
Oct 30 01:16:17.098529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.098491    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"]
Oct 30 01:16:17.104343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.104320    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.110188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.110122    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-dshm\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.110352 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.110241    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxx47\" (UniqueName: \"kubernetes.io/projected/0f250df5-aab6-45c8-afa8-b911491be2e3-kube-api-access-gxx47\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.110352 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.110280    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-model-cache\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.110486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.110377    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-tmp-dir\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.110486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.110422    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f250df5-aab6-45c8-afa8-b911491be2e3-tls-certs\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.110622 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.110606    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-home\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.123566 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.123528    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-df4db1\"/\"kube-root-ca.crt\""
Oct 30 01:16:17.123768 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.123617    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-df4db1\"/\"openshift-service-ca.crt\""
Oct 30 01:16:17.125947 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.125918    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:17.136175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.135577    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-df4db1\"/\"default-dockercfg-kbfs9\""
Oct 30 01:16:17.136175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.135642    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-df4db1\"/\"auth-test-model-df4db1-kserve-self-signed-certs\""
Oct 30 01:16:17.140999 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.140965    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"]
Oct 30 01:16:17.144270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.144236    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:17.211839 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.211799    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f250df5-aab6-45c8-afa8-b911491be2e3-tls-certs\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.212066 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.212008    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-home\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.212150 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.212075    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-dshm\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.212150 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.212170    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxx47\" (UniqueName: \"kubernetes.io/projected/0f250df5-aab6-45c8-afa8-b911491be2e3-kube-api-access-gxx47\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.212327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.212204    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-model-cache\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.212327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.212288    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-tmp-dir\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.212506 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.212426    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-home\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.212885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.212842    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-model-cache\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.213139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.213104    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-tmp-dir\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.214943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.214911    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-dshm\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.215207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.215183    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f250df5-aab6-45c8-afa8-b911491be2e3-tls-certs\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.248826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.248788    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxx47\" (UniqueName: \"kubernetes.io/projected/0f250df5-aab6-45c8-afa8-b911491be2e3-kube-api-access-gxx47\") pod \"auth-test-model-df4db1-kserve-69bbc8dc77-t894d\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") " pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.426265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.425196    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:17.489052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.488737    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:16:17.764020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:17.763988    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"]
Oct 30 01:16:17.764949 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:16:17.764921    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f250df5_aab6_45c8_afa8_b911491be2e3.slice/crio-515e840a8335613b9e5f621da9628aeb7c70f55a0300af0c01da741075165bd1 WatchSource:0}: Error finding container 515e840a8335613b9e5f621da9628aeb7c70f55a0300af0c01da741075165bd1: Status 404 returned error can't find the container with id 515e840a8335613b9e5f621da9628aeb7c70f55a0300af0c01da741075165bd1
Oct 30 01:16:18.470000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:18.469946    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d" event={"ID":"0f250df5-aab6-45c8-afa8-b911491be2e3","Type":"ContainerStarted","Data":"4d6d7ffa777bfac0b10bd720582db2b71b94b2c86ee32625c6306bba0f2a47d1"}
Oct 30 01:16:18.470000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:18.470007    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d" event={"ID":"0f250df5-aab6-45c8-afa8-b911491be2e3","Type":"ContainerStarted","Data":"515e840a8335613b9e5f621da9628aeb7c70f55a0300af0c01da741075165bd1"}
Oct 30 01:16:18.509338 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:18.509270    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d" podStartSLOduration=1.509245597 podStartE2EDuration="1.509245597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:18.50662495 +0000 UTC m=+2087.256280259" watchObservedRunningTime="2026-10-30 01:16:18.509245597 +0000 UTC m=+2087.258900907"
Oct 30 01:16:19.425415 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:19.425383    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:20.971272 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:20.971235    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:16:26.033018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:26.032978    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"]
Oct 30 01:16:26.042411 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:26.042372    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-006f7362-kl77s"]
Oct 30 01:16:27.425768 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:27.425730    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0907da7b-464b-4ad4-949c-d956330e6477" path="/var/lib/kubelet/pods/0907da7b-464b-4ad4-949c-d956330e6477/volumes"
Oct 30 01:16:27.426248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:27.426119    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:27.439836 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:27.439806    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:27.572853 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:27.572821    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:16:28.446735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.446698    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"]
Oct 30 01:16:28.451735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.451701    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.465853 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.465818    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-35042a\"/\"default-dockercfg-fpwtx\""
Oct 30 01:16:28.475969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.475929    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-35042a\"/\"kube-root-ca.crt\""
Oct 30 01:16:28.475969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.475944    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-35042a\"/\"auth-test-model-35042a-kserve-self-signed-certs\""
Oct 30 01:16:28.478364 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.478321    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-35042a\"/\"openshift-service-ca.crt\""
Oct 30 01:16:28.484614 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.484577    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"]
Oct 30 01:16:28.576388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.576356    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-dshm\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.576388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.576399    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-model-cache\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.576692 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.576544    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-home\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.576692 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.576603    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-tmp-dir\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.576692 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.576630    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6vpl\" (UniqueName: \"kubernetes.io/projected/0ea9e224-aceb-4b04-b85b-b8730d32fb39-kube-api-access-q6vpl\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.576872 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.576697    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea9e224-aceb-4b04-b85b-b8730d32fb39-tls-certs\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.681690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.681642    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea9e224-aceb-4b04-b85b-b8730d32fb39-tls-certs\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.681947 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.681925    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-dshm\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.682020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.681987    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-model-cache\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.682121 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.682104    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-home\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.682259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.682241    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-tmp-dir\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.682337 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.682294    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6vpl\" (UniqueName: \"kubernetes.io/projected/0ea9e224-aceb-4b04-b85b-b8730d32fb39-kube-api-access-q6vpl\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.688207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.684295    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-tmp-dir\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.688207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.684582    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-model-cache\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.733656 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.733545    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-dshm\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.738179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.734549    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-home\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.761194 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.760982    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea9e224-aceb-4b04-b85b-b8730d32fb39-tls-certs\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:28.795212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:28.795131    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6vpl\" (UniqueName: \"kubernetes.io/projected/0ea9e224-aceb-4b04-b85b-b8730d32fb39-kube-api-access-q6vpl\") pod \"auth-test-model-35042a-kserve-59bd7f6c69-6w72r\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") " pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:29.070848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:29.070814    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:29.387544 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:29.387510    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"]
Oct 30 01:16:29.387937 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:16:29.387906    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ea9e224_aceb_4b04_b85b_b8730d32fb39.slice/crio-865f93af1b0189bc8e17d4b3d43fa50982e714c2ba8fb85f75f0b38a38860f10 WatchSource:0}: Error finding container 865f93af1b0189bc8e17d4b3d43fa50982e714c2ba8fb85f75f0b38a38860f10: Status 404 returned error can't find the container with id 865f93af1b0189bc8e17d4b3d43fa50982e714c2ba8fb85f75f0b38a38860f10
Oct 30 01:16:29.583036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:29.582998    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r" event={"ID":"0ea9e224-aceb-4b04-b85b-b8730d32fb39","Type":"ContainerStarted","Data":"a981ab1b67efd20e4ffcb210f03e0553aaf02c8a597cbc1aee3f445d2425da4f"}
Oct 30 01:16:29.583507 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:29.583048    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r" event={"ID":"0ea9e224-aceb-4b04-b85b-b8730d32fb39","Type":"ContainerStarted","Data":"865f93af1b0189bc8e17d4b3d43fa50982e714c2ba8fb85f75f0b38a38860f10"}
Oct 30 01:16:29.620658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:29.620594    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r" podStartSLOduration=1.620577182 podStartE2EDuration="1.620577182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:29.619530852 +0000 UTC m=+2098.369186163" watchObservedRunningTime="2026-10-30 01:16:29.620577182 +0000 UTC m=+2098.370232501"
Oct 30 01:16:30.071299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:30.071256    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:33.613795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:33.613756    4425 scope.go:122] "RemoveContainer" containerID="b41603746ddf0d0d141fa4a00a1a47fe37a403e82851ccdc64ed5fdac2dbdf46"
Oct 30 01:16:33.625134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:33.625110    4425 scope.go:122] "RemoveContainer" containerID="05508247032e1b507b8d3bb29015e8f5499d18b11074838c994ef803ba77e3f3"
Oct 30 01:16:37.557555 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:16:37.557509    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:16:39.031456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:39.031414    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"]
Oct 30 01:16:39.041143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:39.041097    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0398835f-jjgsn"]
Oct 30 01:16:39.073879 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:39.073842    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:39.087147 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:39.087117    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:39.425248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:39.425116    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c71f4c8-8e3d-4ba7-8307-b56719826937" path="/var/lib/kubelet/pods/8c71f4c8-8e3d-4ba7-8307-b56719826937/volumes"
Oct 30 01:16:39.682404 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:39.682318    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:16:52.039420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:52.039131    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"]
Oct 30 01:16:52.052219 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:52.052148    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16f1cb73-hg76h"]
Oct 30 01:16:53.425397 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:53.425364    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f6d78a2-1478-43fa-89e5-fd12a889f62f" path="/var/lib/kubelet/pods/0f6d78a2-1478-43fa-89e5-fd12a889f62f/volumes"
Oct 30 01:16:57.497588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.497544    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"]
Oct 30 01:16:57.502717 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.502688    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.515138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.515096    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-1854c4\"/\"openshift-service-ca.crt\""
Oct 30 01:16:57.525028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.524993    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-1854c4\"/\"kube-root-ca.crt\""
Oct 30 01:16:57.525264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.525064    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-1854c4\"/\"default-dockercfg-nsxdz\""
Oct 30 01:16:57.525264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.525076    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-1854c4\"/\"auth-test-model-1854c4-kserve-self-signed-certs\""
Oct 30 01:16:57.530431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.530392    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"]
Oct 30 01:16:57.622868 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.622810    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-tmp-dir\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.623078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.623001    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lrfm\" (UniqueName: \"kubernetes.io/projected/3dbbc68a-083b-4a93-9f03-55c2660151aa-kube-api-access-4lrfm\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.623078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.623039    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-dshm\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.623199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.623188    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-home\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.623323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.623305    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-model-cache\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.623380 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.623367    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dbbc68a-083b-4a93-9f03-55c2660151aa-tls-certs\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.725131 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.725092    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-home\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.725384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.725239    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-model-cache\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.725384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.725290    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dbbc68a-083b-4a93-9f03-55c2660151aa-tls-certs\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.725384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.725352    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-tmp-dir\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.725585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.725422    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrfm\" (UniqueName: \"kubernetes.io/projected/3dbbc68a-083b-4a93-9f03-55c2660151aa-kube-api-access-4lrfm\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.725585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.725443    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-dshm\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.725690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.725590    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-home\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.725690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.725647    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-model-cache\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.725768 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.725754    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-tmp-dir\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.727854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.727821    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-dshm\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.728304 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.728260    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dbbc68a-083b-4a93-9f03-55c2660151aa-tls-certs\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.747044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.747004    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lrfm\" (UniqueName: \"kubernetes.io/projected/3dbbc68a-083b-4a93-9f03-55c2660151aa-kube-api-access-4lrfm\") pod \"auth-test-model-1854c4-kserve-7955bdbb97-jdnxk\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:57.829197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:57.821511    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:16:58.220862 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:16:58.220822    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dbbc68a_083b_4a93_9f03_55c2660151aa.slice/crio-d3f0e6b8d28b781a8f8fba4c7aba44936275671a3ab0504fb45b09a36b6ecde1 WatchSource:0}: Error finding container d3f0e6b8d28b781a8f8fba4c7aba44936275671a3ab0504fb45b09a36b6ecde1: Status 404 returned error can't find the container with id d3f0e6b8d28b781a8f8fba4c7aba44936275671a3ab0504fb45b09a36b6ecde1
Oct 30 01:16:58.222096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:58.222068    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"]
Oct 30 01:16:58.908788 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:58.908750    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk" event={"ID":"3dbbc68a-083b-4a93-9f03-55c2660151aa","Type":"ContainerStarted","Data":"ea94241c38d56f6fb716a8ddc25dced28851c06eb0086f885035674a4f7a6802"}
Oct 30 01:16:58.908788 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:58.908801    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk" event={"ID":"3dbbc68a-083b-4a93-9f03-55c2660151aa","Type":"ContainerStarted","Data":"d3f0e6b8d28b781a8f8fba4c7aba44936275671a3ab0504fb45b09a36b6ecde1"}
Oct 30 01:16:58.947591 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:58.947511    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk" podStartSLOduration=1.947279392 podStartE2EDuration="1.947279392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:58.945399864 +0000 UTC m=+2127.695055174" watchObservedRunningTime="2026-10-30 01:16:58.947279392 +0000 UTC m=+2127.696934791"
Oct 30 01:16:59.823805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:16:59.823762    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:17:03.031463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:03.031422    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"]
Oct 30 01:17:03.040886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:03.040854    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-045742de-4rqcn"]
Oct 30 01:17:03.426320 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:03.426229    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f370d8c-e615-445e-a16c-b0931d6c174a" path="/var/lib/kubelet/pods/5f370d8c-e615-445e-a16c-b0931d6c174a/volumes"
Oct 30 01:17:07.569245 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:17:07.569195    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:17:07.823606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:07.823494    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:17:07.836794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:07.836764    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:17:08.005812 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:08.005781    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:17:09.646600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:09.646555    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:17:15.029073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:15.029026    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"]
Oct 30 01:17:15.039193 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:15.039119    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85202810-d4nn6"]
Oct 30 01:17:15.424473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:15.424377    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad28d88-d05f-4e08-9d34-ba8d61a42c5c" path="/var/lib/kubelet/pods/dad28d88-d05f-4e08-9d34-ba8d61a42c5c/volumes"
Oct 30 01:17:23.411124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:23.411070    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"]
Oct 30 01:17:23.419696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:23.419662    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"
Oct 30 01:17:23.443636 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:23.443594    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"]
Oct 30 01:17:23.444972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:23.444944    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:17:23.448574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:23.445287    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:17:23.481605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:23.481567    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:17:23.481888 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:23.481605    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:17:23.537968 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:23.537916    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abda8f2-faa6-47af-b64e-01590b31a3f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7\" (UID: \"9abda8f2-faa6-47af-b64e-01590b31a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"
Oct 30 01:17:23.538224 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:23.538057    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv4xp\" (UniqueName: \"kubernetes.io/projected/9abda8f2-faa6-47af-b64e-01590b31a3f6-kube-api-access-kv4xp\") pod \"maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7\" (UID: \"9abda8f2-faa6-47af-b64e-01590b31a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"
Oct 30 01:17:23.538362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:23.538341    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abda8f2-faa6-47af-b64e-01590b31a3f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7\" (UID: \"9abda8f2-faa6-47af-b64e-01590b31a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"
Oct 30 01:17:23.639462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:23.639416    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abda8f2-faa6-47af-b64e-01590b31a3f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7\" (UID: \"9abda8f2-faa6-47af-b64e-01590b31a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"
Oct 30 01:17:23.639698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:23.639592    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abda8f2-faa6-47af-b64e-01590b31a3f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7\" (UID: \"9abda8f2-faa6-47af-b64e-01590b31a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"
Oct 30 01:17:23.639698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:23.639637    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4xp\" (UniqueName: \"kubernetes.io/projected/9abda8f2-faa6-47af-b64e-01590b31a3f6-kube-api-access-kv4xp\") pod \"maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7\" (UID: \"9abda8f2-faa6-47af-b64e-01590b31a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"
Oct 30 01:17:23.640366 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:23.640330    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abda8f2-faa6-47af-b64e-01590b31a3f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7\" (UID: \"9abda8f2-faa6-47af-b64e-01590b31a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"
Oct 30 01:17:23.642096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:23.642068    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abda8f2-faa6-47af-b64e-01590b31a3f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7\" (UID: \"9abda8f2-faa6-47af-b64e-01590b31a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"
Oct 30 01:17:23.663231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:23.663129    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4xp\" (UniqueName: \"kubernetes.io/projected/9abda8f2-faa6-47af-b64e-01590b31a3f6-kube-api-access-kv4xp\") pod \"maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7\" (UID: \"9abda8f2-faa6-47af-b64e-01590b31a3f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"
Oct 30 01:17:23.763356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:23.763315    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"
Oct 30 01:17:24.090606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:24.090563    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"]
Oct 30 01:17:24.093935 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:17:24.093902    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abda8f2_faa6_47af_b64e_01590b31a3f6.slice/crio-acd1907fd460d462d20eb4cff73ef11bc16a2cae300090460ec33de98b1598de WatchSource:0}: Error finding container acd1907fd460d462d20eb4cff73ef11bc16a2cae300090460ec33de98b1598de: Status 404 returned error can't find the container with id acd1907fd460d462d20eb4cff73ef11bc16a2cae300090460ec33de98b1598de
Oct 30 01:17:24.145982 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:24.145932    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7" event={"ID":"9abda8f2-faa6-47af-b64e-01590b31a3f6","Type":"ContainerStarted","Data":"acd1907fd460d462d20eb4cff73ef11bc16a2cae300090460ec33de98b1598de"}
Oct 30 01:17:25.156600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:25.156564    4425 generic.go:356] "Generic (PLEG): container finished" podID="9abda8f2-faa6-47af-b64e-01590b31a3f6" containerID="d6baa504d8cb4c23fffde8daeeba42f4a5e32c54194e251be4e090411a65da08" exitCode=0
Oct 30 01:17:25.157006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:25.156645    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7" event={"ID":"9abda8f2-faa6-47af-b64e-01590b31a3f6","Type":"ContainerDied","Data":"d6baa504d8cb4c23fffde8daeeba42f4a5e32c54194e251be4e090411a65da08"}
Oct 30 01:17:26.365232 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:26.365199    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"
Oct 30 01:17:26.488836 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:26.488719    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9abda8f2-faa6-47af-b64e-01590b31a3f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abda8f2-faa6-47af-b64e-01590b31a3f6-maas-api-xks-ca\") pod \"9abda8f2-faa6-47af-b64e-01590b31a3f6\" (UID: \"9abda8f2-faa6-47af-b64e-01590b31a3f6\") "
Oct 30 01:17:26.489020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:26.488952    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abda8f2-faa6-47af-b64e-01590b31a3f6-kube-api-access-kv4xp\" (UniqueName: \"kubernetes.io/projected/9abda8f2-faa6-47af-b64e-01590b31a3f6-kube-api-access-kv4xp\") pod \"9abda8f2-faa6-47af-b64e-01590b31a3f6\" (UID: \"9abda8f2-faa6-47af-b64e-01590b31a3f6\") "
Oct 30 01:17:26.489020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:26.488983    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9abda8f2-faa6-47af-b64e-01590b31a3f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abda8f2-faa6-47af-b64e-01590b31a3f6-maas-api-openshift-ca\") pod \"9abda8f2-faa6-47af-b64e-01590b31a3f6\" (UID: \"9abda8f2-faa6-47af-b64e-01590b31a3f6\") "
Oct 30 01:17:26.489447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:26.489410    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abda8f2-faa6-47af-b64e-01590b31a3f6-maas-api-openshift-ca" pod "9abda8f2-faa6-47af-b64e-01590b31a3f6" (UID: "9abda8f2-faa6-47af-b64e-01590b31a3f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:26.490969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:26.490940    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abda8f2-faa6-47af-b64e-01590b31a3f6-maas-api-xks-ca" pod "9abda8f2-faa6-47af-b64e-01590b31a3f6" (UID: "9abda8f2-faa6-47af-b64e-01590b31a3f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:26.491166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:26.491131    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abda8f2-faa6-47af-b64e-01590b31a3f6-kube-api-access-kv4xp" pod "9abda8f2-faa6-47af-b64e-01590b31a3f6" (UID: "9abda8f2-faa6-47af-b64e-01590b31a3f6"). InnerVolumeSpecName "kube-api-access-kv4xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:26.591134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:26.591089    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abda8f2-faa6-47af-b64e-01590b31a3f6-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:17:26.591134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:26.591127    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv4xp\" (UniqueName: \"kubernetes.io/projected/9abda8f2-faa6-47af-b64e-01590b31a3f6-kube-api-access-kv4xp\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:17:26.591134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:26.591139    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abda8f2-faa6-47af-b64e-01590b31a3f6-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:17:27.173516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.173476    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"]
Oct 30 01:17:27.174277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.174245    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9abda8f2-faa6-47af-b64e-01590b31a3f6" containerName="revoke-keys"
Oct 30 01:17:27.174277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.174269    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9abda8f2-faa6-47af-b64e-01590b31a3f6" containerName="revoke-keys"
Oct 30 01:17:27.174495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.174465    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9abda8f2-faa6-47af-b64e-01590b31a3f6" containerName="revoke-keys"
Oct 30 01:17:27.176606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.176579    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"
Oct 30 01:17:27.184128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.179253    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7" event={"ID":"9abda8f2-faa6-47af-b64e-01590b31a3f6","Type":"ContainerDied","Data":"acd1907fd460d462d20eb4cff73ef11bc16a2cae300090460ec33de98b1598de"}
Oct 30 01:17:27.184128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.179377    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.184128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.179384    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acd1907fd460d462d20eb4cff73ef11bc16a2cae300090460ec33de98b1598de"
Oct 30 01:17:27.199589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.199547    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"]
Oct 30 01:17:27.203901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.203192    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bdf6ca\"/\"default-dockercfg-8cktf\""
Oct 30 01:17:27.203901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.203284    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bdf6ca\"/\"openshift-service-ca.crt\""
Oct 30 01:17:27.206854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.206822    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bdf6ca\"/\"kube-root-ca.crt\""
Oct 30 01:17:27.207050 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.206823    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bdf6ca\"/\"auth-test-model-bdf6ca-kserve-self-signed-certs\""
Oct 30 01:17:27.302149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.302095    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-home\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.302149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.302170    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-model-cache\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.302443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.302423    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6706d150-ebe5-4bdd-ba5f-34126ade3970-tls-certs\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.302492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.302475    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbxc2\" (UniqueName: \"kubernetes.io/projected/6706d150-ebe5-4bdd-ba5f-34126ade3970-kube-api-access-gbxc2\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.302577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.302563    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-tmp-dir\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.302729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.302703    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-dshm\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.404694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.404647    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-dshm\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.405276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.404736    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-home\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.405276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.404763    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-model-cache\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.405276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.404948    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6706d150-ebe5-4bdd-ba5f-34126ade3970-tls-certs\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.405276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.404990    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxc2\" (UniqueName: \"kubernetes.io/projected/6706d150-ebe5-4bdd-ba5f-34126ade3970-kube-api-access-gbxc2\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.405276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.405040    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-tmp-dir\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.405602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.405490    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-tmp-dir\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.405664 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.405604    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-home\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.405756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.405718    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-model-cache\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.407859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.407825    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-dshm\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.408195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.408146    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6706d150-ebe5-4bdd-ba5f-34126ade3970-tls-certs\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.427885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.427782    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxc2\" (UniqueName: \"kubernetes.io/projected/6706d150-ebe5-4bdd-ba5f-34126ade3970-kube-api-access-gbxc2\") pod \"auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") " pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.500193 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.499230    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:27.845455 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:17:27.845416    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6706d150_ebe5_4bdd_ba5f_34126ade3970.slice/crio-ba9bb1e25bd4a38973a42f258c5546ca2f89a5e4245da3f7e881271b8cad00f8 WatchSource:0}: Error finding container ba9bb1e25bd4a38973a42f258c5546ca2f89a5e4245da3f7e881271b8cad00f8: Status 404 returned error can't find the container with id ba9bb1e25bd4a38973a42f258c5546ca2f89a5e4245da3f7e881271b8cad00f8
Oct 30 01:17:27.845649 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:27.845628    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"]
Oct 30 01:17:28.030412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:28.030370    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"]
Oct 30 01:17:28.034875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:28.034842    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c51e9993-lnv9p"]
Oct 30 01:17:28.187980 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:28.187890    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4" event={"ID":"6706d150-ebe5-4bdd-ba5f-34126ade3970","Type":"ContainerStarted","Data":"ab9a6c76edebcd389bef4a41d1dc27b8b5df73977780b720c34fe4c5f96b68ae"}
Oct 30 01:17:28.187980 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:28.187935    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4" event={"ID":"6706d150-ebe5-4bdd-ba5f-34126ade3970","Type":"ContainerStarted","Data":"ba9bb1e25bd4a38973a42f258c5546ca2f89a5e4245da3f7e881271b8cad00f8"}
Oct 30 01:17:28.225191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:28.225090    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4" podStartSLOduration=1.225071844 podStartE2EDuration="1.225071844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:28.223741675 +0000 UTC m=+2156.973397193" watchObservedRunningTime="2026-10-30 01:17:28.225071844 +0000 UTC m=+2156.974727236"
Oct 30 01:17:28.500739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:28.500638    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:29.433416 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:29.433378    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="115c494a-00fd-4c37-a258-e9675dd36c02" path="/var/lib/kubelet/pods/115c494a-00fd-4c37-a258-e9675dd36c02/volumes"
Oct 30 01:17:32.030092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:32.030056    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"]
Oct 30 01:17:32.039459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:32.039420    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6a9a4c7-68nbp"]
Oct 30 01:17:33.424853 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:33.424815    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21" path="/var/lib/kubelet/pods/db662dd3-c1cb-4cd8-8d8e-e6cd2428cc21/volumes"
Oct 30 01:17:33.665300 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:33.665264    4425 scope.go:122] "RemoveContainer" containerID="e5dc512815f4224c42093b6ea0585333a32f6210e1926f253973498e0757ac11"
Oct 30 01:17:33.677383 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:33.677357    4425 scope.go:122] "RemoveContainer" containerID="47aac729f02a73dcfddfdf26db7461742338f79879bd03b325d9d8eb3243623e"
Oct 30 01:17:33.688601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:33.688580    4425 scope.go:122] "RemoveContainer" containerID="32987fa66b3f1a3bf300f14cf9472d3d2c465fc7ac8106cb494136f488ac5c81"
Oct 30 01:17:33.703528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:33.703423    4425 scope.go:122] "RemoveContainer" containerID="c17fc3babd856b59729fd9a0973647464e38252ef83fa011ec09e21cb348eb33"
Oct 30 01:17:33.830140 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:33.830093    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"]
Oct 30 01:17:33.852873 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:33.852831    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"]
Oct 30 01:17:37.500669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:37.500627    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:37.515647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:37.515612    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:37.559459 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:17:37.559414    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:17:38.031788 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:38.031753    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"]
Oct 30 01:17:38.041316 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:38.041273    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24e8b543-84r2f"]
Oct 30 01:17:38.287267 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:38.287170    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:17:38.830936 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:38.830847    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx" podUID="46d4143b-72d6-4da0-88dc-f726562e4c30" containerName="payload-processing" containerID="cri-o://2d2dd0c2c926339b065eb19eb57e2e293948d244e52fc176353203874cd867ca" gracePeriod=25
Oct 30 01:17:38.853484 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:38.853440    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk" podUID="d069dc1c-bd34-44a1-a01c-694812788c8f" containerName="payload-pre-processing" containerID="cri-o://ee5f027fcd6a6be50e1465d2f931ba8fe2112de2f1b342a2b12333185b733801" gracePeriod=25
Oct 30 01:17:39.205731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.204562    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"
Oct 30 01:17:39.262172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.262056    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"
Oct 30 01:17:39.288131 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.288092    4425 generic.go:356] "Generic (PLEG): container finished" podID="46d4143b-72d6-4da0-88dc-f726562e4c30" containerID="2d2dd0c2c926339b065eb19eb57e2e293948d244e52fc176353203874cd867ca" exitCode=0
Oct 30 01:17:39.288380 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.288202    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"
Oct 30 01:17:39.288462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.288379    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx" event={"ID":"46d4143b-72d6-4da0-88dc-f726562e4c30","Type":"ContainerDied","Data":"2d2dd0c2c926339b065eb19eb57e2e293948d244e52fc176353203874cd867ca"}
Oct 30 01:17:39.288462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.288444    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx" event={"ID":"46d4143b-72d6-4da0-88dc-f726562e4c30","Type":"ContainerDied","Data":"84b4075609c5c5b122db6549b8b844241bc8aabfa6ab2f2d0db905643f23ff02"}
Oct 30 01:17:39.288569 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.288477    4425 scope.go:122] "RemoveContainer" containerID="2d2dd0c2c926339b065eb19eb57e2e293948d244e52fc176353203874cd867ca"
Oct 30 01:17:39.291215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.291138    4425 generic.go:356] "Generic (PLEG): container finished" podID="d069dc1c-bd34-44a1-a01c-694812788c8f" containerID="ee5f027fcd6a6be50e1465d2f931ba8fe2112de2f1b342a2b12333185b733801" exitCode=0
Oct 30 01:17:39.291477 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.291214    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk" event={"ID":"d069dc1c-bd34-44a1-a01c-694812788c8f","Type":"ContainerDied","Data":"ee5f027fcd6a6be50e1465d2f931ba8fe2112de2f1b342a2b12333185b733801"}
Oct 30 01:17:39.291578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.291293    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"
Oct 30 01:17:39.291731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.291528    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk" event={"ID":"d069dc1c-bd34-44a1-a01c-694812788c8f","Type":"ContainerDied","Data":"f16e369ceef6514b71064a142d967bf879086d1b1a5c7ce6d25868f8819f5658"}
Oct 30 01:17:39.300134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.300098    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d069dc1c-bd34-44a1-a01c-694812788c8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d069dc1c-bd34-44a1-a01c-694812788c8f-plugins-config-volume\") pod \"d069dc1c-bd34-44a1-a01c-694812788c8f\" (UID: \"d069dc1c-bd34-44a1-a01c-694812788c8f\") "
Oct 30 01:17:39.300411 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.300178    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d4143b-72d6-4da0-88dc-f726562e4c30-kube-api-access-wfpzs\" (UniqueName: \"kubernetes.io/projected/46d4143b-72d6-4da0-88dc-f726562e4c30-kube-api-access-wfpzs\") pod \"46d4143b-72d6-4da0-88dc-f726562e4c30\" (UID: \"46d4143b-72d6-4da0-88dc-f726562e4c30\") "
Oct 30 01:17:39.301044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.300852    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d4143b-72d6-4da0-88dc-f726562e4c30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d4143b-72d6-4da0-88dc-f726562e4c30-plugins-config-volume\") pod \"46d4143b-72d6-4da0-88dc-f726562e4c30\" (UID: \"46d4143b-72d6-4da0-88dc-f726562e4c30\") "
Oct 30 01:17:39.301562 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.301093    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d069dc1c-bd34-44a1-a01c-694812788c8f-plugins-config-volume" pod "d069dc1c-bd34-44a1-a01c-694812788c8f" (UID: "d069dc1c-bd34-44a1-a01c-694812788c8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:39.301562 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.301407    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d4143b-72d6-4da0-88dc-f726562e4c30-plugins-config-volume" pod "46d4143b-72d6-4da0-88dc-f726562e4c30" (UID: "46d4143b-72d6-4da0-88dc-f726562e4c30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:39.301973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.301935    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d069dc1c-bd34-44a1-a01c-694812788c8f-kube-api-access-8z5mv\" (UniqueName: \"kubernetes.io/projected/d069dc1c-bd34-44a1-a01c-694812788c8f-kube-api-access-8z5mv\") pod \"d069dc1c-bd34-44a1-a01c-694812788c8f\" (UID: \"d069dc1c-bd34-44a1-a01c-694812788c8f\") "
Oct 30 01:17:39.303609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.302769    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d4143b-72d6-4da0-88dc-f726562e4c30-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.303609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.302796    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d069dc1c-bd34-44a1-a01c-694812788c8f-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.313938 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.313910    4425 scope.go:122] "RemoveContainer" containerID="2d2dd0c2c926339b065eb19eb57e2e293948d244e52fc176353203874cd867ca"
Oct 30 01:17:39.314614 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.314586    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d4143b-72d6-4da0-88dc-f726562e4c30-kube-api-access-wfpzs" pod "46d4143b-72d6-4da0-88dc-f726562e4c30" (UID: "46d4143b-72d6-4da0-88dc-f726562e4c30"). InnerVolumeSpecName "kube-api-access-wfpzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:39.314729 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:17:39.314635    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2dd0c2c926339b065eb19eb57e2e293948d244e52fc176353203874cd867ca\": container with ID starting with 2d2dd0c2c926339b065eb19eb57e2e293948d244e52fc176353203874cd867ca not found: ID does not exist" containerID="2d2dd0c2c926339b065eb19eb57e2e293948d244e52fc176353203874cd867ca"
Oct 30 01:17:39.314729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.314679    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2dd0c2c926339b065eb19eb57e2e293948d244e52fc176353203874cd867ca"} err="failed to get container status \"2d2dd0c2c926339b065eb19eb57e2e293948d244e52fc176353203874cd867ca\": rpc error: code = NotFound desc = could not find container \"2d2dd0c2c926339b065eb19eb57e2e293948d244e52fc176353203874cd867ca\": container with ID starting with 2d2dd0c2c926339b065eb19eb57e2e293948d244e52fc176353203874cd867ca not found: ID does not exist"
Oct 30 01:17:39.314729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.314705    4425 scope.go:122] "RemoveContainer" containerID="ee5f027fcd6a6be50e1465d2f931ba8fe2112de2f1b342a2b12333185b733801"
Oct 30 01:17:39.317127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.317078    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d069dc1c-bd34-44a1-a01c-694812788c8f-kube-api-access-8z5mv" pod "d069dc1c-bd34-44a1-a01c-694812788c8f" (UID: "d069dc1c-bd34-44a1-a01c-694812788c8f"). InnerVolumeSpecName "kube-api-access-8z5mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:39.331601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.331572    4425 scope.go:122] "RemoveContainer" containerID="ee5f027fcd6a6be50e1465d2f931ba8fe2112de2f1b342a2b12333185b733801"
Oct 30 01:17:39.332029 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:17:39.332001    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee5f027fcd6a6be50e1465d2f931ba8fe2112de2f1b342a2b12333185b733801\": container with ID starting with ee5f027fcd6a6be50e1465d2f931ba8fe2112de2f1b342a2b12333185b733801 not found: ID does not exist" containerID="ee5f027fcd6a6be50e1465d2f931ba8fe2112de2f1b342a2b12333185b733801"
Oct 30 01:17:39.332143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.332038    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee5f027fcd6a6be50e1465d2f931ba8fe2112de2f1b342a2b12333185b733801"} err="failed to get container status \"ee5f027fcd6a6be50e1465d2f931ba8fe2112de2f1b342a2b12333185b733801\": rpc error: code = NotFound desc = could not find container \"ee5f027fcd6a6be50e1465d2f931ba8fe2112de2f1b342a2b12333185b733801\": container with ID starting with ee5f027fcd6a6be50e1465d2f931ba8fe2112de2f1b342a2b12333185b733801 not found: ID does not exist"
Oct 30 01:17:39.403890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.403779    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z5mv\" (UniqueName: \"kubernetes.io/projected/d069dc1c-bd34-44a1-a01c-694812788c8f-kube-api-access-8z5mv\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.403890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.403818    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfpzs\" (UniqueName: \"kubernetes.io/projected/46d4143b-72d6-4da0-88dc-f726562e4c30-kube-api-access-wfpzs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.424963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.424923    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed932198-ee76-4ff5-a777-34e0363f290f" path="/var/lib/kubelet/pods/ed932198-ee76-4ff5-a777-34e0363f290f/volumes"
Oct 30 01:17:39.627818 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.627774    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"]
Oct 30 01:17:39.637290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.637255    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-bbb402-75458b4c86-hf8qx"]
Oct 30 01:17:39.665806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.665707    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"]
Oct 30 01:17:39.670216 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:39.670180    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-bbb402-85c866768c-f99kk"]
Oct 30 01:17:41.425179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:41.425131    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d4143b-72d6-4da0-88dc-f726562e4c30" path="/var/lib/kubelet/pods/46d4143b-72d6-4da0-88dc-f726562e4c30/volumes"
Oct 30 01:17:41.425807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:41.425788    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d069dc1c-bd34-44a1-a01c-694812788c8f" path="/var/lib/kubelet/pods/d069dc1c-bd34-44a1-a01c-694812788c8f/volumes"
Oct 30 01:17:47.032682 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:47.032638    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"]
Oct 30 01:17:47.041073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:47.041032    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebbb688e-dhj2c"]
Oct 30 01:17:47.428119 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:47.427957    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b48dd4d4-9bcb-443c-87f0-1f3b68d4849f" path="/var/lib/kubelet/pods/b48dd4d4-9bcb-443c-87f0-1f3b68d4849f/volumes"
Oct 30 01:17:51.432988 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:51.432953    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"]
Oct 30 01:17:51.442197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:51.442131    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bbb402-xq7r7"]
Oct 30 01:17:53.426540 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:53.426506    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9abda8f2-faa6-47af-b64e-01590b31a3f6" path="/var/lib/kubelet/pods/9abda8f2-faa6-47af-b64e-01590b31a3f6/volumes"
Oct 30 01:17:56.037455 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:56.037404    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"]
Oct 30 01:17:56.043318 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:56.043278    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fc7c87-q5vxc"]
Oct 30 01:17:57.424627 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:17:57.424594    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77aca47d-c426-4970-a1f3-44a22295b2dc" path="/var/lib/kubelet/pods/77aca47d-c426-4970-a1f3-44a22295b2dc/volumes"
Oct 30 01:18:04.029653 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:04.029614    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"]
Oct 30 01:18:04.039475 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:04.039437    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3b68163-89j6s"]
Oct 30 01:18:05.425024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:05.424981    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="304cd043-6da9-4144-9aca-dc55f326a17a" path="/var/lib/kubelet/pods/304cd043-6da9-4144-9aca-dc55f326a17a/volumes"
Oct 30 01:18:07.571434 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:18:07.571394    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:18:10.028748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:10.028704    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"]
Oct 30 01:18:10.037937 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:10.037903    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3827c40-kckhw"]
Oct 30 01:18:11.423858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:11.423823    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76f3c083-0082-47eb-b638-acb89ee0f865" path="/var/lib/kubelet/pods/76f3c083-0082-47eb-b638-acb89ee0f865/volumes"
Oct 30 01:18:12.032602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:12.032567    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"]
Oct 30 01:18:12.036556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:12.036526    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2127eacf-p8d77"]
Oct 30 01:18:13.424981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:13.424943    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a352527e-bda1-46c6-96c9-e6d63cb0fa0d" path="/var/lib/kubelet/pods/a352527e-bda1-46c6-96c9-e6d63cb0fa0d/volumes"
Oct 30 01:18:15.459901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:15.459817    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:18:24.030411 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:24.030370    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"]
Oct 30 01:18:24.038130 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:24.038093    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b6bb0f-2j97k"]
Oct 30 01:18:25.423977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:25.423940    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e" path="/var/lib/kubelet/pods/48a5a9d7-8c0d-46ac-9db9-1ab3af8f728e/volumes"
Oct 30 01:18:26.028264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:26.028219    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"]
Oct 30 01:18:26.038245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:26.038207    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11ad4bf4-9zclv"]
Oct 30 01:18:27.424726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:27.424683    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41cc0607-960c-4a25-bd81-9c3352304048" path="/var/lib/kubelet/pods/41cc0607-960c-4a25-bd81-9c3352304048/volumes"
Oct 30 01:18:33.755799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:33.755749    4425 scope.go:122] "RemoveContainer" containerID="64dee34f57a29b2e80e449c6ab8fc25448026322fa21597771cbeeb5bd185e73"
Oct 30 01:18:33.767600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:33.767572    4425 scope.go:122] "RemoveContainer" containerID="b07d35f99516d4cecd41625c52309035095733153a957266a73c0c70f3fd36e7"
Oct 30 01:18:33.779888 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:33.779856    4425 scope.go:122] "RemoveContainer" containerID="e5d28b34a3f6585f9e8b7b2836b254aed4b9b6c48a19b61cfd377c61af7275ef"
Oct 30 01:18:33.791746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:33.791712    4425 scope.go:122] "RemoveContainer" containerID="8fe564defd38f9e5957414429a7fe23f48e3add8858e1982dabbc40e30f054f8"
Oct 30 01:18:33.804507 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:33.804462    4425 scope.go:122] "RemoveContainer" containerID="505be1d96779e56e4d19002c2ec74eb00bf336d6379795a00faf77b42a33240e"
Oct 30 01:18:33.816633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:33.816601    4425 scope.go:122] "RemoveContainer" containerID="20ed2b33855a36d004c0691ddb49b7ace4a98a6bdfa3e487680b8750bbc20589"
Oct 30 01:18:33.828292 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:33.828265    4425 scope.go:122] "RemoveContainer" containerID="6f1c91677138fae3af43f7eb528d30995db717240ef9c9816adc8e93d7be2367"
Oct 30 01:18:33.839705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:33.839676    4425 scope.go:122] "RemoveContainer" containerID="510dec8e1beeb9bd9f6789530324863381d3eb6b3c6108267895af9fca2ec44a"
Oct 30 01:18:33.851903 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:33.851879    4425 scope.go:122] "RemoveContainer" containerID="2fefc02d9d273169e59171067592ea881e7a7962290295507852fad27044062d"
Oct 30 01:18:36.029625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:36.029588    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"]
Oct 30 01:18:36.038950 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:36.038913    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43631f79-7z9s8"]
Oct 30 01:18:37.424571 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:37.424537    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8184aa82-c3ff-48b6-975f-0293efcd9f20" path="/var/lib/kubelet/pods/8184aa82-c3ff-48b6-975f-0293efcd9f20/volumes"
Oct 30 01:18:37.560684 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:18:37.560640    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:18:39.034738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:39.034701    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"]
Oct 30 01:18:39.044123 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:39.044086    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ce417c4-ppxzl"]
Oct 30 01:18:39.425511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:39.425404    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b4cb78-804f-406f-a7d3-2a869c09282c" path="/var/lib/kubelet/pods/b1b4cb78-804f-406f-a7d3-2a869c09282c/volumes"
Oct 30 01:18:46.029370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:46.029330    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"]
Oct 30 01:18:46.039372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:46.039337    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3bf48ae-8nsqd"]
Oct 30 01:18:47.424546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:47.424511    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ece684-decd-4047-98f5-c925eb2c2004" path="/var/lib/kubelet/pods/a5ece684-decd-4047-98f5-c925eb2c2004/volumes"
Oct 30 01:18:51.028805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:51.028762    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"]
Oct 30 01:18:51.038411 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:51.038368    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2185bf49-vz9mm"]
Oct 30 01:18:51.425531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:51.425445    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b381e5c3-4609-4569-805e-349e19a53cce" path="/var/lib/kubelet/pods/b381e5c3-4609-4569-805e-349e19a53cce/volumes"
Oct 30 01:18:52.029134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:52.029091    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"]
Oct 30 01:18:52.038637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:52.038604    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d775c9-79v8p"]
Oct 30 01:18:53.424884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:18:53.424851    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cbd41ec-10c0-4677-b9bf-6134b50df573" path="/var/lib/kubelet/pods/0cbd41ec-10c0-4677-b9bf-6134b50df573/volumes"
Oct 30 01:19:03.033613 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:03.033573    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"]
Oct 30 01:19:03.040480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:03.040448    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"]
Oct 30 01:19:03.044300 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:03.044258    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-422de6cd-d5f5w"]
Oct 30 01:19:03.047835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:03.047802    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466885d2-5rgwq"]
Oct 30 01:19:03.424978 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:03.424877    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d2b4db3-0cca-4726-ab42-a62d7337034a" path="/var/lib/kubelet/pods/5d2b4db3-0cca-4726-ab42-a62d7337034a/volumes"
Oct 30 01:19:03.425564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:03.425537    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ac87d8-e7ed-4f22-98c8-798328b4cb1e" path="/var/lib/kubelet/pods/a8ac87d8-e7ed-4f22-98c8-798328b4cb1e/volumes"
Oct 30 01:19:07.560347 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:19:07.560310    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:19:10.528546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:10.528493    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"]
Oct 30 01:19:10.529103 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:10.528839    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj" podUID="c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" containerName="istio-proxy" containerID="cri-o://a0d57118a0ed30f19382259ddd53b57a17d750e2800b387ca728c49775a9c388" gracePeriod=30
Oct 30 01:19:15.809790 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.809717    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:19:15.852837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.852775    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-credential-socket\") pod \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") "
Oct 30 01:19:15.853059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.852869    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-workload-certs\") pod \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") "
Oct 30 01:19:15.853059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.852922    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-data\") pod \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") "
Oct 30 01:19:15.853059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.853009    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-podinfo\") pod \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") "
Oct 30 01:19:15.853059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.853056    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istiod-ca-cert\") pod \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") "
Oct 30 01:19:15.853348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.853101    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-workload-socket\") pod \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") "
Oct 30 01:19:15.853348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.853201    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-envoy\") pod \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") "
Oct 30 01:19:15.853348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.853252    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-workload-certs" pod "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" (UID: "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:15.853532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.853368    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-kube-api-access-sg949\" (UniqueName: \"kubernetes.io/projected/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-kube-api-access-sg949\") pod \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") "
Oct 30 01:19:15.853532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.853366    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-data" pod "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" (UID: "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:15.853532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.853380    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-workload-socket" pod "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" (UID: "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:15.853532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.853458    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-credential-socket" pod "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" (UID: "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:15.853756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.853537    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-token\" (UniqueName: \"kubernetes.io/projected/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-token\") pod \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\" (UID: \"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e\") "
Oct 30 01:19:15.853756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.853577    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istiod-ca-cert" pod "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" (UID: "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:15.854332 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.854309    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:19:15.854412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.854339    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:19:15.854412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.854355    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:19:15.854412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.854368    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:19:15.854412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.854381    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:19:15.855565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.855542    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-envoy" pod "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" (UID: "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:15.855656 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.855637    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-podinfo" pod "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" (UID: "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:15.855840 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.855825    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-kube-api-access-sg949" pod "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" (UID: "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e"). InnerVolumeSpecName "kube-api-access-sg949". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:15.855926 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.855911    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-token" pod "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" (UID: "c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:15.955632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.955576    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:19:15.955632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.955625    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:19:15.955632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.955634    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:19:15.955632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:15.955643    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg949\" (UniqueName: \"kubernetes.io/projected/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e-kube-api-access-sg949\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:19:16.162777 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:16.162678    4425 generic.go:356] "Generic (PLEG): container finished" podID="c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" containerID="a0d57118a0ed30f19382259ddd53b57a17d750e2800b387ca728c49775a9c388" exitCode=0
Oct 30 01:19:16.162777 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:16.162727    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"
Oct 30 01:19:16.162777 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:16.162765    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj" event={"ID":"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e","Type":"ContainerDied","Data":"a0d57118a0ed30f19382259ddd53b57a17d750e2800b387ca728c49775a9c388"}
Oct 30 01:19:16.163046 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:16.162830    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj" event={"ID":"c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e","Type":"ContainerDied","Data":"b41f113eca674e7adb277648bd17a66d1dbbf801c19b0c42d07f7c1e4c220e7f"}
Oct 30 01:19:16.163046 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:16.162854    4425 scope.go:122] "RemoveContainer" containerID="a0d57118a0ed30f19382259ddd53b57a17d750e2800b387ca728c49775a9c388"
Oct 30 01:19:16.175790 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:16.175766    4425 scope.go:122] "RemoveContainer" containerID="a0d57118a0ed30f19382259ddd53b57a17d750e2800b387ca728c49775a9c388"
Oct 30 01:19:16.176184 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:19:16.176143    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0d57118a0ed30f19382259ddd53b57a17d750e2800b387ca728c49775a9c388\": container with ID starting with a0d57118a0ed30f19382259ddd53b57a17d750e2800b387ca728c49775a9c388 not found: ID does not exist" containerID="a0d57118a0ed30f19382259ddd53b57a17d750e2800b387ca728c49775a9c388"
Oct 30 01:19:16.176294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:16.176190    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d57118a0ed30f19382259ddd53b57a17d750e2800b387ca728c49775a9c388"} err="failed to get container status \"a0d57118a0ed30f19382259ddd53b57a17d750e2800b387ca728c49775a9c388\": rpc error: code = NotFound desc = could not find container \"a0d57118a0ed30f19382259ddd53b57a17d750e2800b387ca728c49775a9c388\": container with ID starting with a0d57118a0ed30f19382259ddd53b57a17d750e2800b387ca728c49775a9c388 not found: ID does not exist"
Oct 30 01:19:16.209623 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:16.209587    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"]
Oct 30 01:19:16.213348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:16.213315    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-cdtsj"]
Oct 30 01:19:17.165536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:17.165497    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:19:17.427469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:17.427386    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" path="/var/lib/kubelet/pods/c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e/volumes"
Oct 30 01:19:33.953774 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:33.953739    4425 scope.go:122] "RemoveContainer" containerID="78e90bfae287c9df02872f0e29e44fa989f99aea3a83e9038a86c5608346b023"
Oct 30 01:19:33.964565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:33.964544    4425 scope.go:122] "RemoveContainer" containerID="dfd6c7d95dce5564807c7bd24bd432eb9149a2dc2763a70d3d43fd89a830ddba"
Oct 30 01:19:33.975256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:33.975228    4425 scope.go:122] "RemoveContainer" containerID="55dcbb5545602710e226600f64c16c98ab730b524c60641e5a408fd6db858438"
Oct 30 01:19:33.986272 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:33.986150    4425 scope.go:122] "RemoveContainer" containerID="c73443d582cfb251622783aca1bf815505dfd0f12ce2b503a9d78b852432da22"
Oct 30 01:19:33.997051 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:33.997019    4425 scope.go:122] "RemoveContainer" containerID="8a239bb36b7c0a3f2722838ceb3e716958fcbd1584fd00249c9b84dd6cd064bf"
Oct 30 01:19:34.007533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:34.007510    4425 scope.go:122] "RemoveContainer" containerID="18e129718b99df06a437197d2fa573c6a00e4e20a94df974af95b48a84cc9915"
Oct 30 01:19:34.018709 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:34.018684    4425 scope.go:122] "RemoveContainer" containerID="275f4388e89209e027d8d991653d212a329811e7571a1f57352afe9674b4be8d"
Oct 30 01:19:34.029883 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:34.029858    4425 scope.go:122] "RemoveContainer" containerID="c372065e8ec675ac1f0531e77aabc4b1e4ed499e3b36616d4e2df122381e2019"
Oct 30 01:19:34.040908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:34.040882    4425 scope.go:122] "RemoveContainer" containerID="5a457810aea6d92410308ec71e7a8a87533a1dd6a4c0bf12e439a177fb4691c6"
Oct 30 01:19:34.051694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:34.051664    4425 scope.go:122] "RemoveContainer" containerID="c43b09aa2dcf6f5b50167b7812b9c8478f8e34653ff4c3e73056797b93f6957c"
Oct 30 01:19:34.063323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:34.063298    4425 scope.go:122] "RemoveContainer" containerID="8ec53a13ead2b00c9662fba6d67d93a85103928bf7011d3abcf7832bbc826ef9"
Oct 30 01:19:37.557398 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:19:37.557355    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:19:54.381095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:54.381009    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"]
Oct 30 01:19:54.381569 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:54.381266    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4" podUID="2591e8d9-706f-4362-816a-14376d116652" containerName="istio-proxy" containerID="cri-o://54c7fb25f4e17215a36f6ce236ab883a75301bcf905c5c7f752695a9ff64917f" gracePeriod=30
Oct 30 01:19:59.525353 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.525314    4425 generic.go:356] "Generic (PLEG): container finished" podID="2591e8d9-706f-4362-816a-14376d116652" containerID="54c7fb25f4e17215a36f6ce236ab883a75301bcf905c5c7f752695a9ff64917f" exitCode=0
Oct 30 01:19:59.525760 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.525390    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4" event={"ID":"2591e8d9-706f-4362-816a-14376d116652","Type":"ContainerDied","Data":"54c7fb25f4e17215a36f6ce236ab883a75301bcf905c5c7f752695a9ff64917f"}
Oct 30 01:19:59.682468 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.682438    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:19:59.821485 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.821438    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2591e8d9-706f-4362-816a-14376d116652-istio-token\" (UniqueName: \"kubernetes.io/projected/2591e8d9-706f-4362-816a-14376d116652-istio-token\") pod \"2591e8d9-706f-4362-816a-14376d116652\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") "
Oct 30 01:19:59.821701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.821656    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2591e8d9-706f-4362-816a-14376d116652-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2591e8d9-706f-4362-816a-14376d116652-istiod-ca-cert\") pod \"2591e8d9-706f-4362-816a-14376d116652\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") "
Oct 30 01:19:59.821774 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.821743    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-istio-data\") pod \"2591e8d9-706f-4362-816a-14376d116652\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") "
Oct 30 01:19:59.821856 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.821839    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-workload-socket\") pod \"2591e8d9-706f-4362-816a-14376d116652\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") "
Oct 30 01:19:59.821916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.821882    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-istio-envoy\") pod \"2591e8d9-706f-4362-816a-14376d116652\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") "
Oct 30 01:19:59.821972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.821944    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2591e8d9-706f-4362-816a-14376d116652-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2591e8d9-706f-4362-816a-14376d116652-istio-podinfo\") pod \"2591e8d9-706f-4362-816a-14376d116652\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") "
Oct 30 01:19:59.822028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.822004    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-credential-socket\") pod \"2591e8d9-706f-4362-816a-14376d116652\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") "
Oct 30 01:19:59.822082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.822030    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-istio-data" pod "2591e8d9-706f-4362-816a-14376d116652" (UID: "2591e8d9-706f-4362-816a-14376d116652"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:59.822140 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.822089    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2591e8d9-706f-4362-816a-14376d116652-kube-api-access-45r8r\" (UniqueName: \"kubernetes.io/projected/2591e8d9-706f-4362-816a-14376d116652-kube-api-access-45r8r\") pod \"2591e8d9-706f-4362-816a-14376d116652\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") "
Oct 30 01:19:59.822140 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.822102    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-workload-socket" pod "2591e8d9-706f-4362-816a-14376d116652" (UID: "2591e8d9-706f-4362-816a-14376d116652"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:59.822273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.822173    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2591e8d9-706f-4362-816a-14376d116652-istiod-ca-cert" pod "2591e8d9-706f-4362-816a-14376d116652" (UID: "2591e8d9-706f-4362-816a-14376d116652"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:59.822273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.822146    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-workload-certs\") pod \"2591e8d9-706f-4362-816a-14376d116652\" (UID: \"2591e8d9-706f-4362-816a-14376d116652\") "
Oct 30 01:19:59.822372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.822290    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-credential-socket" pod "2591e8d9-706f-4362-816a-14376d116652" (UID: "2591e8d9-706f-4362-816a-14376d116652"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:59.822476 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.822452    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-workload-certs" pod "2591e8d9-706f-4362-816a-14376d116652" (UID: "2591e8d9-706f-4362-816a-14376d116652"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:59.822915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.822881    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.822915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.822907    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2591e8d9-706f-4362-816a-14376d116652-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.823105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.822920    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.823105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.822933    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.823105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.822946    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.823919 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.823898    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2591e8d9-706f-4362-816a-14376d116652-istio-token" pod "2591e8d9-706f-4362-816a-14376d116652" (UID: "2591e8d9-706f-4362-816a-14376d116652"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:59.824592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.824559    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2591e8d9-706f-4362-816a-14376d116652-istio-podinfo" pod "2591e8d9-706f-4362-816a-14376d116652" (UID: "2591e8d9-706f-4362-816a-14376d116652"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:19:59.824592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.824578    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2591e8d9-706f-4362-816a-14376d116652-kube-api-access-45r8r" pod "2591e8d9-706f-4362-816a-14376d116652" (UID: "2591e8d9-706f-4362-816a-14376d116652"). InnerVolumeSpecName "kube-api-access-45r8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:59.824731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.824592    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-istio-envoy" pod "2591e8d9-706f-4362-816a-14376d116652" (UID: "2591e8d9-706f-4362-816a-14376d116652"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:59.924280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.924189    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2591e8d9-706f-4362-816a-14376d116652-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.924280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.924219    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2591e8d9-706f-4362-816a-14376d116652-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.924280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.924229    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45r8r\" (UniqueName: \"kubernetes.io/projected/2591e8d9-706f-4362-816a-14376d116652-kube-api-access-45r8r\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.924280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:19:59.924241    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2591e8d9-706f-4362-816a-14376d116652-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:00.534048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:00.534016    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"
Oct 30 01:20:00.534593 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:00.534045    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4" event={"ID":"2591e8d9-706f-4362-816a-14376d116652","Type":"ContainerDied","Data":"bcdc0ef99c03b366e29ba7506e02477a64248882e8597ab2b14347a5ddc6e726"}
Oct 30 01:20:00.534593 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:00.534125    4425 scope.go:122] "RemoveContainer" containerID="54c7fb25f4e17215a36f6ce236ab883a75301bcf905c5c7f752695a9ff64917f"
Oct 30 01:20:00.592567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:00.592521    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"]
Oct 30 01:20:00.601812 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:00.601772    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-ch2l4"]
Oct 30 01:20:01.423764 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:01.423732    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2591e8d9-706f-4362-816a-14376d116652" path="/var/lib/kubelet/pods/2591e8d9-706f-4362-816a-14376d116652/volumes"
Oct 30 01:20:07.558702 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:20:07.558665    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:20:16.030788 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:16.030748    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"]
Oct 30 01:20:16.037929 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:16.037894    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a520e2b9-pprxc"]
Oct 30 01:20:16.265810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:16.265764    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:20:17.426348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:17.426311    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11af40fe-8b0c-4d7a-a1a9-46a491e85424" path="/var/lib/kubelet/pods/11af40fe-8b0c-4d7a-a1a9-46a491e85424/volumes"
Oct 30 01:20:19.741738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.741679    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"]
Oct 30 01:20:19.744764 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.744717    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d069dc1c-bd34-44a1-a01c-694812788c8f" containerName="payload-pre-processing"
Oct 30 01:20:19.745058 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.745024    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d069dc1c-bd34-44a1-a01c-694812788c8f" containerName="payload-pre-processing"
Oct 30 01:20:19.745306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.745291    4425 cpu_manager.go:414] "containerMap: removing container" podUID="2591e8d9-706f-4362-816a-14376d116652" containerName="istio-proxy"
Oct 30 01:20:19.746544 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.746519    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2591e8d9-706f-4362-816a-14376d116652" containerName="istio-proxy"
Oct 30 01:20:19.746818 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.746789    4425 cpu_manager.go:414] "containerMap: removing container" podUID="46d4143b-72d6-4da0-88dc-f726562e4c30" containerName="payload-processing"
Oct 30 01:20:19.746937 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.746924    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d4143b-72d6-4da0-88dc-f726562e4c30" containerName="payload-processing"
Oct 30 01:20:19.747074 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.747062    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" containerName="istio-proxy"
Oct 30 01:20:19.757360 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.757329    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" containerName="istio-proxy"
Oct 30 01:20:19.757919 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.757887    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d5a87c-3ce5-44dc-8e33-6d0bb8e1e78e" containerName="istio-proxy"
Oct 30 01:20:19.767941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.761243    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d069dc1c-bd34-44a1-a01c-694812788c8f" containerName="payload-pre-processing"
Oct 30 01:20:19.767941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.761300    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2591e8d9-706f-4362-816a-14376d116652" containerName="istio-proxy"
Oct 30 01:20:19.767941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.761332    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d4143b-72d6-4da0-88dc-f726562e4c30" containerName="payload-processing"
Oct 30 01:20:19.776481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.776451    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"
Oct 30 01:20:19.817255 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.789123    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"]
Oct 30 01:20:19.817573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.791330    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:20:19.817685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.791405    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:20:19.817766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.804666    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:20:19.817848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.809312    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:20:19.817923 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.812824    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bd9909-1499-4bc5-84f4-f46c4a9a722c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bff1fb5-q8fl6\" (UID: \"88bd9909-1499-4bc5-84f4-f46c4a9a722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"
Oct 30 01:20:19.818244 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.818201    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bd9909-1499-4bc5-84f4-f46c4a9a722c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bff1fb5-q8fl6\" (UID: \"88bd9909-1499-4bc5-84f4-f46c4a9a722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"
Oct 30 01:20:19.818568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.818541    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whmrx\" (UniqueName: \"kubernetes.io/projected/88bd9909-1499-4bc5-84f4-f46c4a9a722c-kube-api-access-whmrx\") pod \"maas-api-revoke-sub-2bff1fb5-q8fl6\" (UID: \"88bd9909-1499-4bc5-84f4-f46c4a9a722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"
Oct 30 01:20:19.919560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.919506    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bd9909-1499-4bc5-84f4-f46c4a9a722c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bff1fb5-q8fl6\" (UID: \"88bd9909-1499-4bc5-84f4-f46c4a9a722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"
Oct 30 01:20:19.919777 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.919680    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whmrx\" (UniqueName: \"kubernetes.io/projected/88bd9909-1499-4bc5-84f4-f46c4a9a722c-kube-api-access-whmrx\") pod \"maas-api-revoke-sub-2bff1fb5-q8fl6\" (UID: \"88bd9909-1499-4bc5-84f4-f46c4a9a722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"
Oct 30 01:20:19.919862 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.919810    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bd9909-1499-4bc5-84f4-f46c4a9a722c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bff1fb5-q8fl6\" (UID: \"88bd9909-1499-4bc5-84f4-f46c4a9a722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"
Oct 30 01:20:19.920603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.920574    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bd9909-1499-4bc5-84f4-f46c4a9a722c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bff1fb5-q8fl6\" (UID: \"88bd9909-1499-4bc5-84f4-f46c4a9a722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"
Oct 30 01:20:19.922234 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.922210    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bd9909-1499-4bc5-84f4-f46c4a9a722c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bff1fb5-q8fl6\" (UID: \"88bd9909-1499-4bc5-84f4-f46c4a9a722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"
Oct 30 01:20:19.941462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:19.941424    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whmrx\" (UniqueName: \"kubernetes.io/projected/88bd9909-1499-4bc5-84f4-f46c4a9a722c-kube-api-access-whmrx\") pod \"maas-api-revoke-sub-2bff1fb5-q8fl6\" (UID: \"88bd9909-1499-4bc5-84f4-f46c4a9a722c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"
Oct 30 01:20:20.110361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:20.110320    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"
Oct 30 01:20:20.448610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:20.448573    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"]
Oct 30 01:20:20.450495 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:20:20.450462    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88bd9909_1499_4bc5_84f4_f46c4a9a722c.slice/crio-d16e87462d2eec2e4b2df448e74264bf7172c3e1dae105221fe323d437871d47 WatchSource:0}: Error finding container d16e87462d2eec2e4b2df448e74264bf7172c3e1dae105221fe323d437871d47: Status 404 returned error can't find the container with id d16e87462d2eec2e4b2df448e74264bf7172c3e1dae105221fe323d437871d47
Oct 30 01:20:20.700649 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:20.700553    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6" event={"ID":"88bd9909-1499-4bc5-84f4-f46c4a9a722c","Type":"ContainerStarted","Data":"d16e87462d2eec2e4b2df448e74264bf7172c3e1dae105221fe323d437871d47"}
Oct 30 01:20:21.709217 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:21.709098    4425 generic.go:356] "Generic (PLEG): container finished" podID="88bd9909-1499-4bc5-84f4-f46c4a9a722c" containerID="f0fe3bcd9413c6be98a587aae745a6ce69b335d64a377c3267a6d898a7ae8434" exitCode=0
Oct 30 01:20:21.709217 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:21.709188    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6" event={"ID":"88bd9909-1499-4bc5-84f4-f46c4a9a722c","Type":"ContainerDied","Data":"f0fe3bcd9413c6be98a587aae745a6ce69b335d64a377c3267a6d898a7ae8434"}
Oct 30 01:20:22.896755 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:22.896721    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"
Oct 30 01:20:22.984147 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:22.984032    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88bd9909-1499-4bc5-84f4-f46c4a9a722c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bd9909-1499-4bc5-84f4-f46c4a9a722c-maas-api-openshift-ca\") pod \"88bd9909-1499-4bc5-84f4-f46c4a9a722c\" (UID: \"88bd9909-1499-4bc5-84f4-f46c4a9a722c\") "
Oct 30 01:20:22.984558 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:22.984186    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88bd9909-1499-4bc5-84f4-f46c4a9a722c-kube-api-access-whmrx\" (UniqueName: \"kubernetes.io/projected/88bd9909-1499-4bc5-84f4-f46c4a9a722c-kube-api-access-whmrx\") pod \"88bd9909-1499-4bc5-84f4-f46c4a9a722c\" (UID: \"88bd9909-1499-4bc5-84f4-f46c4a9a722c\") "
Oct 30 01:20:22.984558 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:22.984270    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88bd9909-1499-4bc5-84f4-f46c4a9a722c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bd9909-1499-4bc5-84f4-f46c4a9a722c-maas-api-xks-ca\") pod \"88bd9909-1499-4bc5-84f4-f46c4a9a722c\" (UID: \"88bd9909-1499-4bc5-84f4-f46c4a9a722c\") "
Oct 30 01:20:22.986005 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:22.985978    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88bd9909-1499-4bc5-84f4-f46c4a9a722c-maas-api-openshift-ca" pod "88bd9909-1499-4bc5-84f4-f46c4a9a722c" (UID: "88bd9909-1499-4bc5-84f4-f46c4a9a722c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:22.991280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:22.991242    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88bd9909-1499-4bc5-84f4-f46c4a9a722c-kube-api-access-whmrx" pod "88bd9909-1499-4bc5-84f4-f46c4a9a722c" (UID: "88bd9909-1499-4bc5-84f4-f46c4a9a722c"). InnerVolumeSpecName "kube-api-access-whmrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:22.995368 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:22.995330    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88bd9909-1499-4bc5-84f4-f46c4a9a722c-maas-api-xks-ca" pod "88bd9909-1499-4bc5-84f4-f46c4a9a722c" (UID: "88bd9909-1499-4bc5-84f4-f46c4a9a722c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:23.086843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:23.086803    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bd9909-1499-4bc5-84f4-f46c4a9a722c-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:23.086843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:23.086838    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whmrx\" (UniqueName: \"kubernetes.io/projected/88bd9909-1499-4bc5-84f4-f46c4a9a722c-kube-api-access-whmrx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:23.087069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:23.086868    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bd9909-1499-4bc5-84f4-f46c4a9a722c-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:23.727531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:23.727440    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6" event={"ID":"88bd9909-1499-4bc5-84f4-f46c4a9a722c","Type":"ContainerDied","Data":"d16e87462d2eec2e4b2df448e74264bf7172c3e1dae105221fe323d437871d47"}
Oct 30 01:20:23.727531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:23.727457    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"
Oct 30 01:20:23.727531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:23.727498    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d16e87462d2eec2e4b2df448e74264bf7172c3e1dae105221fe323d437871d47"
Oct 30 01:20:30.215310 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.215267    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"]
Oct 30 01:20:30.216293 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.216274    4425 cpu_manager.go:414] "containerMap: removing container" podUID="88bd9909-1499-4bc5-84f4-f46c4a9a722c" containerName="revoke-keys"
Oct 30 01:20:30.216349 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.216295    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88bd9909-1499-4bc5-84f4-f46c4a9a722c" containerName="revoke-keys"
Oct 30 01:20:30.216468 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.216457    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="88bd9909-1499-4bc5-84f4-f46c4a9a722c" containerName="revoke-keys"
Oct 30 01:20:30.220378 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.220356    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"
Oct 30 01:20:30.243428 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.243388    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:20:30.243428 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.243398    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:20:30.243652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.243436    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:20:30.243652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.243459    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:20:30.247805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.247763    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"]
Oct 30 01:20:30.322568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.322523    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x8zn\" (UniqueName: \"kubernetes.io/projected/8073178c-d929-45e4-8dc8-8fd7de27b041-kube-api-access-6x8zn\") pod \"maas-api-revoke-sub-6fa45cdd-rxkth\" (UID: \"8073178c-d929-45e4-8dc8-8fd7de27b041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"
Oct 30 01:20:30.322774 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.322751    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8073178c-d929-45e4-8dc8-8fd7de27b041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fa45cdd-rxkth\" (UID: \"8073178c-d929-45e4-8dc8-8fd7de27b041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"
Oct 30 01:20:30.322842 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.322789    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8073178c-d929-45e4-8dc8-8fd7de27b041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fa45cdd-rxkth\" (UID: \"8073178c-d929-45e4-8dc8-8fd7de27b041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"
Oct 30 01:20:30.424215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.424172    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8073178c-d929-45e4-8dc8-8fd7de27b041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fa45cdd-rxkth\" (UID: \"8073178c-d929-45e4-8dc8-8fd7de27b041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"
Oct 30 01:20:30.424215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.424217    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8073178c-d929-45e4-8dc8-8fd7de27b041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fa45cdd-rxkth\" (UID: \"8073178c-d929-45e4-8dc8-8fd7de27b041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"
Oct 30 01:20:30.424450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.424326    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x8zn\" (UniqueName: \"kubernetes.io/projected/8073178c-d929-45e4-8dc8-8fd7de27b041-kube-api-access-6x8zn\") pod \"maas-api-revoke-sub-6fa45cdd-rxkth\" (UID: \"8073178c-d929-45e4-8dc8-8fd7de27b041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"
Oct 30 01:20:30.424913 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.424884    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8073178c-d929-45e4-8dc8-8fd7de27b041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fa45cdd-rxkth\" (UID: \"8073178c-d929-45e4-8dc8-8fd7de27b041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"
Oct 30 01:20:30.426768 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.426744    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8073178c-d929-45e4-8dc8-8fd7de27b041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fa45cdd-rxkth\" (UID: \"8073178c-d929-45e4-8dc8-8fd7de27b041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"
Oct 30 01:20:30.445561 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.445525    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x8zn\" (UniqueName: \"kubernetes.io/projected/8073178c-d929-45e4-8dc8-8fd7de27b041-kube-api-access-6x8zn\") pod \"maas-api-revoke-sub-6fa45cdd-rxkth\" (UID: \"8073178c-d929-45e4-8dc8-8fd7de27b041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"
Oct 30 01:20:30.535784 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.535739    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"
Oct 30 01:20:30.847069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:30.847034    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"]
Oct 30 01:20:30.849519 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:20:30.849484    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8073178c_d929_45e4_8dc8_8fd7de27b041.slice/crio-ece0827df3dc74a6ff2396d269fc8ee22d7c6afdb7651e878ec5029ce6e9ea64 WatchSource:0}: Error finding container ece0827df3dc74a6ff2396d269fc8ee22d7c6afdb7651e878ec5029ce6e9ea64: Status 404 returned error can't find the container with id ece0827df3dc74a6ff2396d269fc8ee22d7c6afdb7651e878ec5029ce6e9ea64
Oct 30 01:20:31.865911 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:31.865870    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth" event={"ID":"8073178c-d929-45e4-8dc8-8fd7de27b041","Type":"ContainerStarted","Data":"8b5b252cd103391d8e5e2370d6ef0c884ca69c4f6b553e61445def06552e7b7f"}
Oct 30 01:20:31.866345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:31.865926    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth" event={"ID":"8073178c-d929-45e4-8dc8-8fd7de27b041","Type":"ContainerStarted","Data":"ece0827df3dc74a6ff2396d269fc8ee22d7c6afdb7651e878ec5029ce6e9ea64"}
Oct 30 01:20:31.903574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:31.903518    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth" podStartSLOduration=1.903502884 podStartE2EDuration="1.903502884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:31.90182163 +0000 UTC m=+2340.651476942" watchObservedRunningTime="2026-10-30 01:20:31.903502884 +0000 UTC m=+2340.653158192"
Oct 30 01:20:32.875448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:32.875406    4425 generic.go:356] "Generic (PLEG): container finished" podID="8073178c-d929-45e4-8dc8-8fd7de27b041" containerID="8b5b252cd103391d8e5e2370d6ef0c884ca69c4f6b553e61445def06552e7b7f" exitCode=0
Oct 30 01:20:32.875910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:32.875486    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth" event={"ID":"8073178c-d929-45e4-8dc8-8fd7de27b041","Type":"ContainerDied","Data":"8b5b252cd103391d8e5e2370d6ef0c884ca69c4f6b553e61445def06552e7b7f"}
Oct 30 01:20:32.930818 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:32.930780    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"]
Oct 30 01:20:32.931035 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:32.930993    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn" podUID="bfee5645-34a5-4aec-a646-1742da2db219" containerName="istio-proxy" containerID="cri-o://4d12da9a8d2fbf2aa0ee18105f287f3b094b01431211ba204519c50bd5249380" gracePeriod=30
Oct 30 01:20:34.078743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.078713    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"
Oct 30 01:20:34.162989 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.162896    4425 scope.go:122] "RemoveContainer" containerID="5ec95f3a138311130ba11975114b39f1133b71cb337c89ff08185d5c34270507"
Oct 30 01:20:34.179980 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.179945    4425 scope.go:122] "RemoveContainer" containerID="65edeaeadd91e7bc5b8f67e2db7467d2039cd43c71ecc8a07e814653dee9f1b1"
Oct 30 01:20:34.186464 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.186435    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8073178c-d929-45e4-8dc8-8fd7de27b041-kube-api-access-6x8zn\" (UniqueName: \"kubernetes.io/projected/8073178c-d929-45e4-8dc8-8fd7de27b041-kube-api-access-6x8zn\") pod \"8073178c-d929-45e4-8dc8-8fd7de27b041\" (UID: \"8073178c-d929-45e4-8dc8-8fd7de27b041\") "
Oct 30 01:20:34.186646 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.186554    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8073178c-d929-45e4-8dc8-8fd7de27b041-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8073178c-d929-45e4-8dc8-8fd7de27b041-maas-api-xks-ca\") pod \"8073178c-d929-45e4-8dc8-8fd7de27b041\" (UID: \"8073178c-d929-45e4-8dc8-8fd7de27b041\") "
Oct 30 01:20:34.186719 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.186659    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8073178c-d929-45e4-8dc8-8fd7de27b041-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8073178c-d929-45e4-8dc8-8fd7de27b041-maas-api-openshift-ca\") pod \"8073178c-d929-45e4-8dc8-8fd7de27b041\" (UID: \"8073178c-d929-45e4-8dc8-8fd7de27b041\") "
Oct 30 01:20:34.187254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.187181    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8073178c-d929-45e4-8dc8-8fd7de27b041-maas-api-openshift-ca" pod "8073178c-d929-45e4-8dc8-8fd7de27b041" (UID: "8073178c-d929-45e4-8dc8-8fd7de27b041"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:34.188979 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.188950    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8073178c-d929-45e4-8dc8-8fd7de27b041-kube-api-access-6x8zn" pod "8073178c-d929-45e4-8dc8-8fd7de27b041" (UID: "8073178c-d929-45e4-8dc8-8fd7de27b041"). InnerVolumeSpecName "kube-api-access-6x8zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:34.189134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.189111    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8073178c-d929-45e4-8dc8-8fd7de27b041-maas-api-xks-ca" pod "8073178c-d929-45e4-8dc8-8fd7de27b041" (UID: "8073178c-d929-45e4-8dc8-8fd7de27b041"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:34.194641 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.194612    4425 scope.go:122] "RemoveContainer" containerID="65333d10df1ecf7f4be6efefe5d49b68c2be587ae6299e1e0341f9900396729a"
Oct 30 01:20:34.221899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.221871    4425 scope.go:122] "RemoveContainer" containerID="17b9ec166c4e239d95822aada815ea8e443e52de522e1f7fc132776632e97d40"
Oct 30 01:20:34.233438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.233409    4425 scope.go:122] "RemoveContainer" containerID="ff82a86a964dadf2f8bb78de944a13847d6cace75069f8e1fca1d3ffe72dce1d"
Oct 30 01:20:34.244105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.244079    4425 scope.go:122] "RemoveContainer" containerID="6c08741e2d913ee219b74ef27004bdcb56cec68dbfb10c041f095fe95ef94de4"
Oct 30 01:20:34.256675 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.256640    4425 scope.go:122] "RemoveContainer" containerID="16bca5ddd362acd3f03f8791029df8f6ddf3e8e7cbafb4deb70b3bf71298c917"
Oct 30 01:20:34.269847 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.269821    4425 scope.go:122] "RemoveContainer" containerID="b099bd53bd0c97cb0081631658ef32c40ec80d7f2cec20c9b4ed4ba10838ea60"
Oct 30 01:20:34.288927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.288903    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8073178c-d929-45e4-8dc8-8fd7de27b041-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:34.288998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.288929    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8073178c-d929-45e4-8dc8-8fd7de27b041-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:34.288998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.288948    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x8zn\" (UniqueName: \"kubernetes.io/projected/8073178c-d929-45e4-8dc8-8fd7de27b041-kube-api-access-6x8zn\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:34.896237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.896203    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"
Oct 30 01:20:34.896237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.896229    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth" event={"ID":"8073178c-d929-45e4-8dc8-8fd7de27b041","Type":"ContainerDied","Data":"ece0827df3dc74a6ff2396d269fc8ee22d7c6afdb7651e878ec5029ce6e9ea64"}
Oct 30 01:20:34.896517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:34.896301    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ece0827df3dc74a6ff2396d269fc8ee22d7c6afdb7651e878ec5029ce6e9ea64"
Oct 30 01:20:37.557139 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:20:37.557099    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:20:38.236422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.236323    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 01:20:38.345552 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.345503    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfee5645-34a5-4aec-a646-1742da2db219-kube-api-access-pwpgg\" (UniqueName: \"kubernetes.io/projected/bfee5645-34a5-4aec-a646-1742da2db219-kube-api-access-pwpgg\") pod \"bfee5645-34a5-4aec-a646-1742da2db219\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") "
Oct 30 01:20:38.345773 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.345604    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfee5645-34a5-4aec-a646-1742da2db219-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfee5645-34a5-4aec-a646-1742da2db219-istio-podinfo\") pod \"bfee5645-34a5-4aec-a646-1742da2db219\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") "
Oct 30 01:20:38.345773 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.345637    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfee5645-34a5-4aec-a646-1742da2db219-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfee5645-34a5-4aec-a646-1742da2db219-istiod-ca-cert\") pod \"bfee5645-34a5-4aec-a646-1742da2db219\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") "
Oct 30 01:20:38.345773 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.345664    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-workload-socket\") pod \"bfee5645-34a5-4aec-a646-1742da2db219\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") "
Oct 30 01:20:38.345773 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.345694    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-istio-envoy\") pod \"bfee5645-34a5-4aec-a646-1742da2db219\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") "
Oct 30 01:20:38.345996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.345831    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-workload-certs\") pod \"bfee5645-34a5-4aec-a646-1742da2db219\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") "
Oct 30 01:20:38.345996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.345946    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-istio-data\") pod \"bfee5645-34a5-4aec-a646-1742da2db219\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") "
Oct 30 01:20:38.346098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.346027    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-credential-socket\") pod \"bfee5645-34a5-4aec-a646-1742da2db219\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") "
Oct 30 01:20:38.346148 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.346118    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfee5645-34a5-4aec-a646-1742da2db219-istio-token\" (UniqueName: \"kubernetes.io/projected/bfee5645-34a5-4aec-a646-1742da2db219-istio-token\") pod \"bfee5645-34a5-4aec-a646-1742da2db219\" (UID: \"bfee5645-34a5-4aec-a646-1742da2db219\") "
Oct 30 01:20:38.346303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.346213    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-workload-certs" pod "bfee5645-34a5-4aec-a646-1742da2db219" (UID: "bfee5645-34a5-4aec-a646-1742da2db219"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:38.347088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.346470    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-istio-data" pod "bfee5645-34a5-4aec-a646-1742da2db219" (UID: "bfee5645-34a5-4aec-a646-1742da2db219"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:38.347088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.346715    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:38.347088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.346737    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:38.347088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.347018    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-credential-socket" pod "bfee5645-34a5-4aec-a646-1742da2db219" (UID: "bfee5645-34a5-4aec-a646-1742da2db219"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:38.347734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.347508    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfee5645-34a5-4aec-a646-1742da2db219-istiod-ca-cert" pod "bfee5645-34a5-4aec-a646-1742da2db219" (UID: "bfee5645-34a5-4aec-a646-1742da2db219"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:38.348366 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.348330    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-workload-socket" pod "bfee5645-34a5-4aec-a646-1742da2db219" (UID: "bfee5645-34a5-4aec-a646-1742da2db219"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:38.348544 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.348450    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfee5645-34a5-4aec-a646-1742da2db219-istio-podinfo" pod "bfee5645-34a5-4aec-a646-1742da2db219" (UID: "bfee5645-34a5-4aec-a646-1742da2db219"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:20:38.348904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.348882    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfee5645-34a5-4aec-a646-1742da2db219-kube-api-access-pwpgg" pod "bfee5645-34a5-4aec-a646-1742da2db219" (UID: "bfee5645-34a5-4aec-a646-1742da2db219"). InnerVolumeSpecName "kube-api-access-pwpgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:38.349326 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.349305    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfee5645-34a5-4aec-a646-1742da2db219-istio-token" pod "bfee5645-34a5-4aec-a646-1742da2db219" (UID: "bfee5645-34a5-4aec-a646-1742da2db219"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:38.350010 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.349993    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-istio-envoy" pod "bfee5645-34a5-4aec-a646-1742da2db219" (UID: "bfee5645-34a5-4aec-a646-1742da2db219"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:38.447784 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.447748    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:38.447784 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.447783    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfee5645-34a5-4aec-a646-1742da2db219-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:38.448036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.447795    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwpgg\" (UniqueName: \"kubernetes.io/projected/bfee5645-34a5-4aec-a646-1742da2db219-kube-api-access-pwpgg\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:38.448036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.447804    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfee5645-34a5-4aec-a646-1742da2db219-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:38.448036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.447813    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfee5645-34a5-4aec-a646-1742da2db219-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:38.448036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.447822    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:38.448036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.447829    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfee5645-34a5-4aec-a646-1742da2db219-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:38.923696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.923664    4425 generic.go:356] "Generic (PLEG): container finished" podID="bfee5645-34a5-4aec-a646-1742da2db219" containerID="4d12da9a8d2fbf2aa0ee18105f287f3b094b01431211ba204519c50bd5249380" exitCode=0
Oct 30 01:20:38.924303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.923715    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"
Oct 30 01:20:38.924303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.923742    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn" event={"ID":"bfee5645-34a5-4aec-a646-1742da2db219","Type":"ContainerDied","Data":"4d12da9a8d2fbf2aa0ee18105f287f3b094b01431211ba204519c50bd5249380"}
Oct 30 01:20:38.924303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.923804    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn" event={"ID":"bfee5645-34a5-4aec-a646-1742da2db219","Type":"ContainerDied","Data":"dd53e649c690e53111d458452efcade54273561d76d4fb2abb1eb9a954621b09"}
Oct 30 01:20:38.924303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.923835    4425 scope.go:122] "RemoveContainer" containerID="4d12da9a8d2fbf2aa0ee18105f287f3b094b01431211ba204519c50bd5249380"
Oct 30 01:20:38.935907 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.935885    4425 scope.go:122] "RemoveContainer" containerID="4d12da9a8d2fbf2aa0ee18105f287f3b094b01431211ba204519c50bd5249380"
Oct 30 01:20:38.936369 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:20:38.936343    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d12da9a8d2fbf2aa0ee18105f287f3b094b01431211ba204519c50bd5249380\": container with ID starting with 4d12da9a8d2fbf2aa0ee18105f287f3b094b01431211ba204519c50bd5249380 not found: ID does not exist" containerID="4d12da9a8d2fbf2aa0ee18105f287f3b094b01431211ba204519c50bd5249380"
Oct 30 01:20:38.936443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.936380    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d12da9a8d2fbf2aa0ee18105f287f3b094b01431211ba204519c50bd5249380"} err="failed to get container status \"4d12da9a8d2fbf2aa0ee18105f287f3b094b01431211ba204519c50bd5249380\": rpc error: code = NotFound desc = could not find container \"4d12da9a8d2fbf2aa0ee18105f287f3b094b01431211ba204519c50bd5249380\": container with ID starting with 4d12da9a8d2fbf2aa0ee18105f287f3b094b01431211ba204519c50bd5249380 not found: ID does not exist"
Oct 30 01:20:38.980357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.980319    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"]
Oct 30 01:20:38.989764 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:38.989729    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8z8gn"]
Oct 30 01:20:39.424884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:39.424849    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfee5645-34a5-4aec-a646-1742da2db219" path="/var/lib/kubelet/pods/bfee5645-34a5-4aec-a646-1742da2db219/volumes"
Oct 30 01:20:42.300879 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.300839    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"]
Oct 30 01:20:42.301920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.301889    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8073178c-d929-45e4-8dc8-8fd7de27b041" containerName="revoke-keys"
Oct 30 01:20:42.301920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.301914    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8073178c-d929-45e4-8dc8-8fd7de27b041" containerName="revoke-keys"
Oct 30 01:20:42.302107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.301983    4425 cpu_manager.go:414] "containerMap: removing container" podUID="bfee5645-34a5-4aec-a646-1742da2db219" containerName="istio-proxy"
Oct 30 01:20:42.302107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.301992    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfee5645-34a5-4aec-a646-1742da2db219" containerName="istio-proxy"
Oct 30 01:20:42.302212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.302186    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8073178c-d929-45e4-8dc8-8fd7de27b041" containerName="revoke-keys"
Oct 30 01:20:42.302212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.302208    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfee5645-34a5-4aec-a646-1742da2db219" containerName="istio-proxy"
Oct 30 01:20:42.307703 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.307679    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"
Oct 30 01:20:42.324550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.324503    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"]
Oct 30 01:20:42.330535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.330502    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:20:42.330535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.330518    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:20:42.330770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.330542    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:20:42.330770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.330557    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:20:42.414089 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.414051    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb2gm\" (UniqueName: \"kubernetes.io/projected/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-kube-api-access-rb2gm\") pod \"maas-api-revoke-sub-1ccf6204-67hfx\" (UID: \"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"
Oct 30 01:20:42.414306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.414265    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ccf6204-67hfx\" (UID: \"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"
Oct 30 01:20:42.414361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.414323    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ccf6204-67hfx\" (UID: \"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"
Oct 30 01:20:42.515389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.515350    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ccf6204-67hfx\" (UID: \"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"
Oct 30 01:20:42.515597 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.515408    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ccf6204-67hfx\" (UID: \"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"
Oct 30 01:20:42.515597 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.515552    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb2gm\" (UniqueName: \"kubernetes.io/projected/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-kube-api-access-rb2gm\") pod \"maas-api-revoke-sub-1ccf6204-67hfx\" (UID: \"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"
Oct 30 01:20:42.516321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.516295    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ccf6204-67hfx\" (UID: \"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"
Oct 30 01:20:42.518167 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.518121    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ccf6204-67hfx\" (UID: \"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"
Oct 30 01:20:42.538071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.538030    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb2gm\" (UniqueName: \"kubernetes.io/projected/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-kube-api-access-rb2gm\") pod \"maas-api-revoke-sub-1ccf6204-67hfx\" (UID: \"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"
Oct 30 01:20:42.624142 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.624042    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"
Oct 30 01:20:42.939434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.939402    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"]
Oct 30 01:20:42.941038 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:20:42.941008    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0ebf75_c6a7_40e4_b749_1b0fda28a42b.slice/crio-327a0f33663b715a7e9818f167e09a7f987c289cad50433db38a738f25f667e6 WatchSource:0}: Error finding container 327a0f33663b715a7e9818f167e09a7f987c289cad50433db38a738f25f667e6: Status 404 returned error can't find the container with id 327a0f33663b715a7e9818f167e09a7f987c289cad50433db38a738f25f667e6
Oct 30 01:20:42.965893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:42.965860    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx" event={"ID":"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b","Type":"ContainerStarted","Data":"327a0f33663b715a7e9818f167e09a7f987c289cad50433db38a738f25f667e6"}
Oct 30 01:20:43.838521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:43.838474    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"]
Oct 30 01:20:44.032852 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:44.032807    4425 generic.go:356] "Generic (PLEG): container finished" podID="fe0ebf75-c6a7-40e4-b749-1b0fda28a42b" containerID="2d7eb593420e07b4ca7546ea4278d94702b55e8ad0938d0826743f1d6507fbf3" exitCode=0
Oct 30 01:20:44.033323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:44.033293    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx" event={"ID":"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b","Type":"ContainerDied","Data":"2d7eb593420e07b4ca7546ea4278d94702b55e8ad0938d0826743f1d6507fbf3"}
Oct 30 01:20:45.218126 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:45.218074    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"
Oct 30 01:20:45.281851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:45.281799    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-maas-api-xks-ca\") pod \"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b\" (UID: \"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b\") "
Oct 30 01:20:45.282126 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:45.282105    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-kube-api-access-rb2gm\" (UniqueName: \"kubernetes.io/projected/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-kube-api-access-rb2gm\") pod \"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b\" (UID: \"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b\") "
Oct 30 01:20:45.282230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:45.282142    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-maas-api-openshift-ca\") pod \"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b\" (UID: \"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b\") "
Oct 30 01:20:45.282641 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:45.282607    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-maas-api-openshift-ca" pod "fe0ebf75-c6a7-40e4-b749-1b0fda28a42b" (UID: "fe0ebf75-c6a7-40e4-b749-1b0fda28a42b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:45.282883 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:45.282849    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:45.284604 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:45.284565    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-kube-api-access-rb2gm" pod "fe0ebf75-c6a7-40e4-b749-1b0fda28a42b" (UID: "fe0ebf75-c6a7-40e4-b749-1b0fda28a42b"). InnerVolumeSpecName "kube-api-access-rb2gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:45.284750 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:45.284721    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-maas-api-xks-ca" pod "fe0ebf75-c6a7-40e4-b749-1b0fda28a42b" (UID: "fe0ebf75-c6a7-40e4-b749-1b0fda28a42b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:45.384030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:45.383981    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:45.384030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:45.384021    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb2gm\" (UniqueName: \"kubernetes.io/projected/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b-kube-api-access-rb2gm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:46.048505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:46.048473    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"
Oct 30 01:20:46.048505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:46.048500    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx" event={"ID":"fe0ebf75-c6a7-40e4-b749-1b0fda28a42b","Type":"ContainerDied","Data":"327a0f33663b715a7e9818f167e09a7f987c289cad50433db38a738f25f667e6"}
Oct 30 01:20:46.048795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:46.048552    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="327a0f33663b715a7e9818f167e09a7f987c289cad50433db38a738f25f667e6"
Oct 30 01:20:47.148513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.148479    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"]
Oct 30 01:20:47.149196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.149176    4425 cpu_manager.go:414] "containerMap: removing container" podUID="fe0ebf75-c6a7-40e4-b749-1b0fda28a42b" containerName="revoke-keys"
Oct 30 01:20:47.149283 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.149198    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe0ebf75-c6a7-40e4-b749-1b0fda28a42b" containerName="revoke-keys"
Oct 30 01:20:47.149401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.149389    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe0ebf75-c6a7-40e4-b749-1b0fda28a42b" containerName="revoke-keys"
Oct 30 01:20:47.153404 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.153379    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"
Oct 30 01:20:47.174223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.174186    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:20:47.183965 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.183926    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:20:47.184227 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.184059    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:20:47.184227 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.184206    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:20:47.189755 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.189721    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"]
Oct 30 01:20:47.311994 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.311952    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc468dd-b7fe-47e1-bc25-69e61924f729-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c947ff9-96krr\" (UID: \"6fc468dd-b7fe-47e1-bc25-69e61924f729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"
Oct 30 01:20:47.312225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.312132    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6vjw\" (UniqueName: \"kubernetes.io/projected/6fc468dd-b7fe-47e1-bc25-69e61924f729-kube-api-access-g6vjw\") pod \"maas-api-revoke-sub-5c947ff9-96krr\" (UID: \"6fc468dd-b7fe-47e1-bc25-69e61924f729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"
Oct 30 01:20:47.312225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.312181    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc468dd-b7fe-47e1-bc25-69e61924f729-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c947ff9-96krr\" (UID: \"6fc468dd-b7fe-47e1-bc25-69e61924f729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"
Oct 30 01:20:47.414374 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.414273    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6vjw\" (UniqueName: \"kubernetes.io/projected/6fc468dd-b7fe-47e1-bc25-69e61924f729-kube-api-access-g6vjw\") pod \"maas-api-revoke-sub-5c947ff9-96krr\" (UID: \"6fc468dd-b7fe-47e1-bc25-69e61924f729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"
Oct 30 01:20:47.414374 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.414323    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc468dd-b7fe-47e1-bc25-69e61924f729-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c947ff9-96krr\" (UID: \"6fc468dd-b7fe-47e1-bc25-69e61924f729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"
Oct 30 01:20:47.414638 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.414442    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc468dd-b7fe-47e1-bc25-69e61924f729-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c947ff9-96krr\" (UID: \"6fc468dd-b7fe-47e1-bc25-69e61924f729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"
Oct 30 01:20:47.415266 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.415237    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc468dd-b7fe-47e1-bc25-69e61924f729-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c947ff9-96krr\" (UID: \"6fc468dd-b7fe-47e1-bc25-69e61924f729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"
Oct 30 01:20:47.417167 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.417122    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc468dd-b7fe-47e1-bc25-69e61924f729-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c947ff9-96krr\" (UID: \"6fc468dd-b7fe-47e1-bc25-69e61924f729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"
Oct 30 01:20:47.438124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.438090    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6vjw\" (UniqueName: \"kubernetes.io/projected/6fc468dd-b7fe-47e1-bc25-69e61924f729-kube-api-access-g6vjw\") pod \"maas-api-revoke-sub-5c947ff9-96krr\" (UID: \"6fc468dd-b7fe-47e1-bc25-69e61924f729\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"
Oct 30 01:20:47.468649 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.468611    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"
Oct 30 01:20:47.777262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:47.777226    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"]
Oct 30 01:20:47.779135 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:20:47.779104    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc468dd_b7fe_47e1_bc25_69e61924f729.slice/crio-3f804b27de74dc162cc9e6a42a444c986758a404febf35eee989f7ae9fd022f6 WatchSource:0}: Error finding container 3f804b27de74dc162cc9e6a42a444c986758a404febf35eee989f7ae9fd022f6: Status 404 returned error can't find the container with id 3f804b27de74dc162cc9e6a42a444c986758a404febf35eee989f7ae9fd022f6
Oct 30 01:20:48.066725 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:48.066693    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr" event={"ID":"6fc468dd-b7fe-47e1-bc25-69e61924f729","Type":"ContainerStarted","Data":"3f804b27de74dc162cc9e6a42a444c986758a404febf35eee989f7ae9fd022f6"}
Oct 30 01:20:49.076128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:49.076090    4425 generic.go:356] "Generic (PLEG): container finished" podID="6fc468dd-b7fe-47e1-bc25-69e61924f729" containerID="66b01135e9e4ad433f3228cd37b3b68c3f92daa06586f1a017cb007ceb91ee9c" exitCode=0
Oct 30 01:20:49.076590 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:49.076170    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr" event={"ID":"6fc468dd-b7fe-47e1-bc25-69e61924f729","Type":"ContainerDied","Data":"66b01135e9e4ad433f3228cd37b3b68c3f92daa06586f1a017cb007ceb91ee9c"}
Oct 30 01:20:50.269734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:50.269704    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"
Oct 30 01:20:50.360969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:50.360876    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc468dd-b7fe-47e1-bc25-69e61924f729-kube-api-access-g6vjw\" (UniqueName: \"kubernetes.io/projected/6fc468dd-b7fe-47e1-bc25-69e61924f729-kube-api-access-g6vjw\") pod \"6fc468dd-b7fe-47e1-bc25-69e61924f729\" (UID: \"6fc468dd-b7fe-47e1-bc25-69e61924f729\") "
Oct 30 01:20:50.360969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:50.360915    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc468dd-b7fe-47e1-bc25-69e61924f729-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc468dd-b7fe-47e1-bc25-69e61924f729-maas-api-openshift-ca\") pod \"6fc468dd-b7fe-47e1-bc25-69e61924f729\" (UID: \"6fc468dd-b7fe-47e1-bc25-69e61924f729\") "
Oct 30 01:20:50.361296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:50.361017    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fc468dd-b7fe-47e1-bc25-69e61924f729-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc468dd-b7fe-47e1-bc25-69e61924f729-maas-api-xks-ca\") pod \"6fc468dd-b7fe-47e1-bc25-69e61924f729\" (UID: \"6fc468dd-b7fe-47e1-bc25-69e61924f729\") "
Oct 30 01:20:50.361435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:50.361411    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc468dd-b7fe-47e1-bc25-69e61924f729-maas-api-openshift-ca" pod "6fc468dd-b7fe-47e1-bc25-69e61924f729" (UID: "6fc468dd-b7fe-47e1-bc25-69e61924f729"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:50.361595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:50.361577    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc468dd-b7fe-47e1-bc25-69e61924f729-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:50.363162 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:50.363126    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc468dd-b7fe-47e1-bc25-69e61924f729-maas-api-xks-ca" pod "6fc468dd-b7fe-47e1-bc25-69e61924f729" (UID: "6fc468dd-b7fe-47e1-bc25-69e61924f729"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:50.363250 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:50.363199    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc468dd-b7fe-47e1-bc25-69e61924f729-kube-api-access-g6vjw" pod "6fc468dd-b7fe-47e1-bc25-69e61924f729" (UID: "6fc468dd-b7fe-47e1-bc25-69e61924f729"). InnerVolumeSpecName "kube-api-access-g6vjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:50.462998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:50.462948    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6vjw\" (UniqueName: \"kubernetes.io/projected/6fc468dd-b7fe-47e1-bc25-69e61924f729-kube-api-access-g6vjw\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:50.462998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:50.462991    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc468dd-b7fe-47e1-bc25-69e61924f729-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:51.093943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:51.093910    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr" event={"ID":"6fc468dd-b7fe-47e1-bc25-69e61924f729","Type":"ContainerDied","Data":"3f804b27de74dc162cc9e6a42a444c986758a404febf35eee989f7ae9fd022f6"}
Oct 30 01:20:51.094119 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:51.093959    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f804b27de74dc162cc9e6a42a444c986758a404febf35eee989f7ae9fd022f6"
Oct 30 01:20:51.094119 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:51.093927    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"
Oct 30 01:20:52.716949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.716913    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"]
Oct 30 01:20:52.717604 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.717588    4425 cpu_manager.go:414] "containerMap: removing container" podUID="6fc468dd-b7fe-47e1-bc25-69e61924f729" containerName="revoke-keys"
Oct 30 01:20:52.717658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.717605    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc468dd-b7fe-47e1-bc25-69e61924f729" containerName="revoke-keys"
Oct 30 01:20:52.717716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.717707    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc468dd-b7fe-47e1-bc25-69e61924f729" containerName="revoke-keys"
Oct 30 01:20:52.722198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.722120    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"
Oct 30 01:20:52.742134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.742101    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:20:52.747131 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.747080    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"]
Oct 30 01:20:52.752136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.752106    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:20:52.752339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.752308    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:20:52.794446 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.794237    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5lwt\" (UniqueName: \"kubernetes.io/projected/ca312417-9947-410e-b0da-f8f703a63272-kube-api-access-n5lwt\") pod \"maas-api-revoke-sub-04ed4ee2-j56t4\" (UID: \"ca312417-9947-410e-b0da-f8f703a63272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"
Oct 30 01:20:52.794645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.794591    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca312417-9947-410e-b0da-f8f703a63272-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04ed4ee2-j56t4\" (UID: \"ca312417-9947-410e-b0da-f8f703a63272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"
Oct 30 01:20:52.794645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.794621    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca312417-9947-410e-b0da-f8f703a63272-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04ed4ee2-j56t4\" (UID: \"ca312417-9947-410e-b0da-f8f703a63272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"
Oct 30 01:20:52.842438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.842403    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:20:52.895593 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.895541    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5lwt\" (UniqueName: \"kubernetes.io/projected/ca312417-9947-410e-b0da-f8f703a63272-kube-api-access-n5lwt\") pod \"maas-api-revoke-sub-04ed4ee2-j56t4\" (UID: \"ca312417-9947-410e-b0da-f8f703a63272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"
Oct 30 01:20:52.895791 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.895762    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca312417-9947-410e-b0da-f8f703a63272-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04ed4ee2-j56t4\" (UID: \"ca312417-9947-410e-b0da-f8f703a63272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"
Oct 30 01:20:52.895862 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.895804    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca312417-9947-410e-b0da-f8f703a63272-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04ed4ee2-j56t4\" (UID: \"ca312417-9947-410e-b0da-f8f703a63272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"
Oct 30 01:20:52.896572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.896546    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca312417-9947-410e-b0da-f8f703a63272-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04ed4ee2-j56t4\" (UID: \"ca312417-9947-410e-b0da-f8f703a63272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"
Oct 30 01:20:52.898312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.898291    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca312417-9947-410e-b0da-f8f703a63272-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04ed4ee2-j56t4\" (UID: \"ca312417-9947-410e-b0da-f8f703a63272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"
Oct 30 01:20:52.917045 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:52.917010    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5lwt\" (UniqueName: \"kubernetes.io/projected/ca312417-9947-410e-b0da-f8f703a63272-kube-api-access-n5lwt\") pod \"maas-api-revoke-sub-04ed4ee2-j56t4\" (UID: \"ca312417-9947-410e-b0da-f8f703a63272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"
Oct 30 01:20:53.038569 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:53.038526    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"
Oct 30 01:20:53.346178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:53.346129    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"]
Oct 30 01:20:53.348393 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:20:53.348346    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca312417_9947_410e_b0da_f8f703a63272.slice/crio-851e98b6d4151466f5e2b264931a78e7cbeb6d45aaaf023194b838a53dbf822e WatchSource:0}: Error finding container 851e98b6d4151466f5e2b264931a78e7cbeb6d45aaaf023194b838a53dbf822e: Status 404 returned error can't find the container with id 851e98b6d4151466f5e2b264931a78e7cbeb6d45aaaf023194b838a53dbf822e
Oct 30 01:20:54.136057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:54.135342    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4" event={"ID":"ca312417-9947-410e-b0da-f8f703a63272","Type":"ContainerStarted","Data":"b9a497d26f8e46b550fcfa3123eb19496841d46ab2e7a63f79c4b205b6890e1a"}
Oct 30 01:20:54.136057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:54.135424    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4" event={"ID":"ca312417-9947-410e-b0da-f8f703a63272","Type":"ContainerStarted","Data":"851e98b6d4151466f5e2b264931a78e7cbeb6d45aaaf023194b838a53dbf822e"}
Oct 30 01:20:54.203458 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:54.203397    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4" podStartSLOduration=2.203379679 podStartE2EDuration="2.203379679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:54.198761875 +0000 UTC m=+2362.948417183" watchObservedRunningTime="2026-10-30 01:20:54.203379679 +0000 UTC m=+2362.953035021"
Oct 30 01:20:55.146225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:55.146183    4425 generic.go:356] "Generic (PLEG): container finished" podID="ca312417-9947-410e-b0da-f8f703a63272" containerID="b9a497d26f8e46b550fcfa3123eb19496841d46ab2e7a63f79c4b205b6890e1a" exitCode=0
Oct 30 01:20:55.146697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:55.146252    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4" event={"ID":"ca312417-9947-410e-b0da-f8f703a63272","Type":"ContainerDied","Data":"b9a497d26f8e46b550fcfa3123eb19496841d46ab2e7a63f79c4b205b6890e1a"}
Oct 30 01:20:55.860429 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:55.860388    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"]
Oct 30 01:20:55.860676 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:55.860645    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8" podUID="1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" containerName="istio-proxy" containerID="cri-o://196947cd3b579255d01288142b6f45ae83f374d29291d6470066b5b575c89741" gracePeriod=30
Oct 30 01:20:56.329763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:56.329734    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"
Oct 30 01:20:56.474954 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:56.474860    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca312417-9947-410e-b0da-f8f703a63272-kube-api-access-n5lwt\" (UniqueName: \"kubernetes.io/projected/ca312417-9947-410e-b0da-f8f703a63272-kube-api-access-n5lwt\") pod \"ca312417-9947-410e-b0da-f8f703a63272\" (UID: \"ca312417-9947-410e-b0da-f8f703a63272\") "
Oct 30 01:20:56.474954 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:56.474917    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca312417-9947-410e-b0da-f8f703a63272-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca312417-9947-410e-b0da-f8f703a63272-maas-api-xks-ca\") pod \"ca312417-9947-410e-b0da-f8f703a63272\" (UID: \"ca312417-9947-410e-b0da-f8f703a63272\") "
Oct 30 01:20:56.475232 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:56.474960    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca312417-9947-410e-b0da-f8f703a63272-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca312417-9947-410e-b0da-f8f703a63272-maas-api-openshift-ca\") pod \"ca312417-9947-410e-b0da-f8f703a63272\" (UID: \"ca312417-9947-410e-b0da-f8f703a63272\") "
Oct 30 01:20:56.475460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:56.475432    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca312417-9947-410e-b0da-f8f703a63272-maas-api-openshift-ca" pod "ca312417-9947-410e-b0da-f8f703a63272" (UID: "ca312417-9947-410e-b0da-f8f703a63272"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:56.476679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:56.476649    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca312417-9947-410e-b0da-f8f703a63272-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:56.477544 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:56.477518    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca312417-9947-410e-b0da-f8f703a63272-maas-api-xks-ca" pod "ca312417-9947-410e-b0da-f8f703a63272" (UID: "ca312417-9947-410e-b0da-f8f703a63272"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:56.477697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:56.477676    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca312417-9947-410e-b0da-f8f703a63272-kube-api-access-n5lwt" pod "ca312417-9947-410e-b0da-f8f703a63272" (UID: "ca312417-9947-410e-b0da-f8f703a63272"). InnerVolumeSpecName "kube-api-access-n5lwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:56.578343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:56.578304    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5lwt\" (UniqueName: \"kubernetes.io/projected/ca312417-9947-410e-b0da-f8f703a63272-kube-api-access-n5lwt\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:56.578343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:56.578341    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca312417-9947-410e-b0da-f8f703a63272-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:57.162036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.162006    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"
Oct 30 01:20:57.162036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.162031    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4" event={"ID":"ca312417-9947-410e-b0da-f8f703a63272","Type":"ContainerDied","Data":"851e98b6d4151466f5e2b264931a78e7cbeb6d45aaaf023194b838a53dbf822e"}
Oct 30 01:20:57.162430 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.162083    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="851e98b6d4151466f5e2b264931a78e7cbeb6d45aaaf023194b838a53dbf822e"
Oct 30 01:20:57.504894 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.504816    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"]
Oct 30 01:20:57.505469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.505452    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ca312417-9947-410e-b0da-f8f703a63272" containerName="revoke-keys"
Oct 30 01:20:57.505529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.505470    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca312417-9947-410e-b0da-f8f703a63272" containerName="revoke-keys"
Oct 30 01:20:57.505628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.505615    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca312417-9947-410e-b0da-f8f703a63272" containerName="revoke-keys"
Oct 30 01:20:57.509451 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.509431    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"
Oct 30 01:20:57.529199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.529170    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"]
Oct 30 01:20:57.531771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.531747    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:20:57.531771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.531759    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:20:57.531957 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.531792    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:20:57.531957 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.531838    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:20:57.590567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.590533    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a650e14b-1236-47e8-b903-fdc7dd48c486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e3df17c-zlxnn\" (UID: \"a650e14b-1236-47e8-b903-fdc7dd48c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"
Oct 30 01:20:57.590734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.590656    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a650e14b-1236-47e8-b903-fdc7dd48c486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e3df17c-zlxnn\" (UID: \"a650e14b-1236-47e8-b903-fdc7dd48c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"
Oct 30 01:20:57.590734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.590695    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dnhg\" (UniqueName: \"kubernetes.io/projected/a650e14b-1236-47e8-b903-fdc7dd48c486-kube-api-access-6dnhg\") pod \"maas-api-revoke-sub-8e3df17c-zlxnn\" (UID: \"a650e14b-1236-47e8-b903-fdc7dd48c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"
Oct 30 01:20:57.691884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.691842    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnhg\" (UniqueName: \"kubernetes.io/projected/a650e14b-1236-47e8-b903-fdc7dd48c486-kube-api-access-6dnhg\") pod \"maas-api-revoke-sub-8e3df17c-zlxnn\" (UID: \"a650e14b-1236-47e8-b903-fdc7dd48c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"
Oct 30 01:20:57.692101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.692081    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a650e14b-1236-47e8-b903-fdc7dd48c486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e3df17c-zlxnn\" (UID: \"a650e14b-1236-47e8-b903-fdc7dd48c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"
Oct 30 01:20:57.692245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.692215    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a650e14b-1236-47e8-b903-fdc7dd48c486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e3df17c-zlxnn\" (UID: \"a650e14b-1236-47e8-b903-fdc7dd48c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"
Oct 30 01:20:57.692794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.692765    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a650e14b-1236-47e8-b903-fdc7dd48c486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e3df17c-zlxnn\" (UID: \"a650e14b-1236-47e8-b903-fdc7dd48c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"
Oct 30 01:20:57.694556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.694535    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a650e14b-1236-47e8-b903-fdc7dd48c486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e3df17c-zlxnn\" (UID: \"a650e14b-1236-47e8-b903-fdc7dd48c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"
Oct 30 01:20:57.713955 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.713918    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dnhg\" (UniqueName: \"kubernetes.io/projected/a650e14b-1236-47e8-b903-fdc7dd48c486-kube-api-access-6dnhg\") pod \"maas-api-revoke-sub-8e3df17c-zlxnn\" (UID: \"a650e14b-1236-47e8-b903-fdc7dd48c486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"
Oct 30 01:20:57.823973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:57.823935    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"
Oct 30 01:20:58.133054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:58.133021    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"]
Oct 30 01:20:58.133971 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:20:58.133942    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda650e14b_1236_47e8_b903_fdc7dd48c486.slice/crio-5a7f2720b35b5e9775d6467e4b95a0f119aea292ac2f29ff9e3f1cda8e628990 WatchSource:0}: Error finding container 5a7f2720b35b5e9775d6467e4b95a0f119aea292ac2f29ff9e3f1cda8e628990: Status 404 returned error can't find the container with id 5a7f2720b35b5e9775d6467e4b95a0f119aea292ac2f29ff9e3f1cda8e628990
Oct 30 01:20:58.179192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:58.178719    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn" event={"ID":"a650e14b-1236-47e8-b903-fdc7dd48c486","Type":"ContainerStarted","Data":"5a7f2720b35b5e9775d6467e4b95a0f119aea292ac2f29ff9e3f1cda8e628990"}
Oct 30 01:20:58.913609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:58.913561    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c" podUID="e3477fee-0b54-47ee-9a25-b1c67c27e698" containerName="main" containerID="cri-o://b1c809ee3967409ab4a441fa70ee170caea41ea5aff3b9b3d0081c80b8115a22" gracePeriod=45
Oct 30 01:20:59.189132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.189100    4425 generic.go:356] "Generic (PLEG): container finished" podID="a650e14b-1236-47e8-b903-fdc7dd48c486" containerID="687f2cc89e63d7babc30a1752307764af5e0133e3fe8984b7c45095c89d7bae6" exitCode=0
Oct 30 01:20:59.189303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.189190    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn" event={"ID":"a650e14b-1236-47e8-b903-fdc7dd48c486","Type":"ContainerDied","Data":"687f2cc89e63d7babc30a1752307764af5e0133e3fe8984b7c45095c89d7bae6"}
Oct 30 01:20:59.190994 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.190957    4425 generic.go:356] "Generic (PLEG): container finished" podID="e3477fee-0b54-47ee-9a25-b1c67c27e698" containerID="b1c809ee3967409ab4a441fa70ee170caea41ea5aff3b9b3d0081c80b8115a22" exitCode=0
Oct 30 01:20:59.191150 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.191105    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c" event={"ID":"e3477fee-0b54-47ee-9a25-b1c67c27e698","Type":"ContainerDied","Data":"b1c809ee3967409ab4a441fa70ee170caea41ea5aff3b9b3d0081c80b8115a22"}
Oct 30 01:20:59.211438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.211413    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:20:59.314490 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.314458    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3477fee-0b54-47ee-9a25-b1c67c27e698-kube-api-access-26djw\" (UniqueName: \"kubernetes.io/projected/e3477fee-0b54-47ee-9a25-b1c67c27e698-kube-api-access-26djw\") pod \"e3477fee-0b54-47ee-9a25-b1c67c27e698\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") "
Oct 30 01:20:59.314667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.314522    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-model-cache\") pod \"e3477fee-0b54-47ee-9a25-b1c67c27e698\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") "
Oct 30 01:20:59.314667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.314621    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-tmp-dir\") pod \"e3477fee-0b54-47ee-9a25-b1c67c27e698\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") "
Oct 30 01:20:59.314752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.314670    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-home\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-home\") pod \"e3477fee-0b54-47ee-9a25-b1c67c27e698\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") "
Oct 30 01:20:59.314752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.314692    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3477fee-0b54-47ee-9a25-b1c67c27e698-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3477fee-0b54-47ee-9a25-b1c67c27e698-tls-certs\") pod \"e3477fee-0b54-47ee-9a25-b1c67c27e698\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") "
Oct 30 01:20:59.314840 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.314781    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-dshm\") pod \"e3477fee-0b54-47ee-9a25-b1c67c27e698\" (UID: \"e3477fee-0b54-47ee-9a25-b1c67c27e698\") "
Oct 30 01:20:59.314923 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.314884    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-model-cache" pod "e3477fee-0b54-47ee-9a25-b1c67c27e698" (UID: "e3477fee-0b54-47ee-9a25-b1c67c27e698"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:59.314923 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.314899    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-home" pod "e3477fee-0b54-47ee-9a25-b1c67c27e698" (UID: "e3477fee-0b54-47ee-9a25-b1c67c27e698"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:59.315054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.314923    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-tmp-dir" pod "e3477fee-0b54-47ee-9a25-b1c67c27e698" (UID: "e3477fee-0b54-47ee-9a25-b1c67c27e698"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:59.315464 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.315442    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:59.315570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.315468    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:59.315570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.315477    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:59.316992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.316963    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3477fee-0b54-47ee-9a25-b1c67c27e698-tls-certs" pod "e3477fee-0b54-47ee-9a25-b1c67c27e698" (UID: "e3477fee-0b54-47ee-9a25-b1c67c27e698"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:59.317102 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.316998    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3477fee-0b54-47ee-9a25-b1c67c27e698-kube-api-access-26djw" pod "e3477fee-0b54-47ee-9a25-b1c67c27e698" (UID: "e3477fee-0b54-47ee-9a25-b1c67c27e698"). InnerVolumeSpecName "kube-api-access-26djw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:59.317102 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.317049    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-dshm" pod "e3477fee-0b54-47ee-9a25-b1c67c27e698" (UID: "e3477fee-0b54-47ee-9a25-b1c67c27e698"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:20:59.416602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.416569    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26djw\" (UniqueName: \"kubernetes.io/projected/e3477fee-0b54-47ee-9a25-b1c67c27e698-kube-api-access-26djw\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:59.416602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.416597    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3477fee-0b54-47ee-9a25-b1c67c27e698-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:20:59.416602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:20:59.416610    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3477fee-0b54-47ee-9a25-b1c67c27e698-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:00.199881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:00.199847    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"
Oct 30 01:21:00.199881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:00.199862    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c" event={"ID":"e3477fee-0b54-47ee-9a25-b1c67c27e698","Type":"ContainerDied","Data":"a7b20ed968526bac4936161ec7dfcea252bb826adaae2b133b12c88eea889a06"}
Oct 30 01:21:00.200439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:00.199919    4425 scope.go:122] "RemoveContainer" containerID="b1c809ee3967409ab4a441fa70ee170caea41ea5aff3b9b3d0081c80b8115a22"
Oct 30 01:21:00.247140 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:00.247107    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"]
Oct 30 01:21:00.256581 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:00.256547    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0e1a98/auth-test-model-0e1a98-kserve-865fbdc49-79g6c"]
Oct 30 01:21:00.373476 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:00.373447    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"
Oct 30 01:21:00.531371 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:00.531336    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a650e14b-1236-47e8-b903-fdc7dd48c486-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a650e14b-1236-47e8-b903-fdc7dd48c486-maas-api-xks-ca\") pod \"a650e14b-1236-47e8-b903-fdc7dd48c486\" (UID: \"a650e14b-1236-47e8-b903-fdc7dd48c486\") "
Oct 30 01:21:00.531525 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:00.531467    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a650e14b-1236-47e8-b903-fdc7dd48c486-kube-api-access-6dnhg\" (UniqueName: \"kubernetes.io/projected/a650e14b-1236-47e8-b903-fdc7dd48c486-kube-api-access-6dnhg\") pod \"a650e14b-1236-47e8-b903-fdc7dd48c486\" (UID: \"a650e14b-1236-47e8-b903-fdc7dd48c486\") "
Oct 30 01:21:00.531525 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:00.531511    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a650e14b-1236-47e8-b903-fdc7dd48c486-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a650e14b-1236-47e8-b903-fdc7dd48c486-maas-api-openshift-ca\") pod \"a650e14b-1236-47e8-b903-fdc7dd48c486\" (UID: \"a650e14b-1236-47e8-b903-fdc7dd48c486\") "
Oct 30 01:21:00.531927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:00.531902    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a650e14b-1236-47e8-b903-fdc7dd48c486-maas-api-openshift-ca" pod "a650e14b-1236-47e8-b903-fdc7dd48c486" (UID: "a650e14b-1236-47e8-b903-fdc7dd48c486"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:00.533605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:00.533584    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a650e14b-1236-47e8-b903-fdc7dd48c486-kube-api-access-6dnhg" pod "a650e14b-1236-47e8-b903-fdc7dd48c486" (UID: "a650e14b-1236-47e8-b903-fdc7dd48c486"). InnerVolumeSpecName "kube-api-access-6dnhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:00.533689 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:00.533600    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a650e14b-1236-47e8-b903-fdc7dd48c486-maas-api-xks-ca" pod "a650e14b-1236-47e8-b903-fdc7dd48c486" (UID: "a650e14b-1236-47e8-b903-fdc7dd48c486"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:00.633017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:00.632984    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dnhg\" (UniqueName: \"kubernetes.io/projected/a650e14b-1236-47e8-b903-fdc7dd48c486-kube-api-access-6dnhg\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:00.633017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:00.633015    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a650e14b-1236-47e8-b903-fdc7dd48c486-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:00.633017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:00.633025    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a650e14b-1236-47e8-b903-fdc7dd48c486-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:01.135074 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.135010    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:21:01.209891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.209855    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"
Oct 30 01:21:01.210395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.209865    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn" event={"ID":"a650e14b-1236-47e8-b903-fdc7dd48c486","Type":"ContainerDied","Data":"5a7f2720b35b5e9775d6467e4b95a0f119aea292ac2f29ff9e3f1cda8e628990"}
Oct 30 01:21:01.210395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.210040    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a7f2720b35b5e9775d6467e4b95a0f119aea292ac2f29ff9e3f1cda8e628990"
Oct 30 01:21:01.211701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.211672    4425 generic.go:356] "Generic (PLEG): container finished" podID="1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" containerID="196947cd3b579255d01288142b6f45ae83f374d29291d6470066b5b575c89741" exitCode=0
Oct 30 01:21:01.211834 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.211780    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8" event={"ID":"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9","Type":"ContainerDied","Data":"196947cd3b579255d01288142b6f45ae83f374d29291d6470066b5b575c89741"}
Oct 30 01:21:01.211834 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.211816    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8" event={"ID":"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9","Type":"ContainerDied","Data":"c2df390cd36d40c585e02e45a80637afa8c96bece1cf533ab18796bf07db074d"}
Oct 30 01:21:01.211911 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.211841    4425 scope.go:122] "RemoveContainer" containerID="196947cd3b579255d01288142b6f45ae83f374d29291d6470066b5b575c89741"
Oct 30 01:21:01.211944 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.211919    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"
Oct 30 01:21:01.226519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.226496    4425 scope.go:122] "RemoveContainer" containerID="196947cd3b579255d01288142b6f45ae83f374d29291d6470066b5b575c89741"
Oct 30 01:21:01.226855 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:21:01.226830    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196947cd3b579255d01288142b6f45ae83f374d29291d6470066b5b575c89741\": container with ID starting with 196947cd3b579255d01288142b6f45ae83f374d29291d6470066b5b575c89741 not found: ID does not exist" containerID="196947cd3b579255d01288142b6f45ae83f374d29291d6470066b5b575c89741"
Oct 30 01:21:01.226957 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.226867    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196947cd3b579255d01288142b6f45ae83f374d29291d6470066b5b575c89741"} err="failed to get container status \"196947cd3b579255d01288142b6f45ae83f374d29291d6470066b5b575c89741\": rpc error: code = NotFound desc = could not find container \"196947cd3b579255d01288142b6f45ae83f374d29291d6470066b5b575c89741\": container with ID starting with 196947cd3b579255d01288142b6f45ae83f374d29291d6470066b5b575c89741 not found: ID does not exist"
Oct 30 01:21:01.240546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.240523    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-token\" (UniqueName: \"kubernetes.io/projected/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-token\") pod \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") "
Oct 30 01:21:01.240658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.240584    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-kube-api-access-qff77\" (UniqueName: \"kubernetes.io/projected/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-kube-api-access-qff77\") pod \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") "
Oct 30 01:21:01.240658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.240631    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-workload-socket\") pod \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") "
Oct 30 01:21:01.240658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.240653    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-podinfo\") pod \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") "
Oct 30 01:21:01.240780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.240695    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-credential-socket\") pod \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") "
Oct 30 01:21:01.240780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.240743    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-data\") pod \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") "
Oct 30 01:21:01.240884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.240796    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istiod-ca-cert\") pod \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") "
Oct 30 01:21:01.240940 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.240904    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-envoy\") pod \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") "
Oct 30 01:21:01.241000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.240932    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-workload-socket" pod "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" (UID: "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:01.241000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.240955    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-workload-certs\") pod \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\" (UID: \"1617c4a9-d9b2-41b7-9907-c1d2d8878eb9\") "
Oct 30 01:21:01.241109 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.241013    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-credential-socket" pod "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" (UID: "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:01.241109 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.241030    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-data" pod "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" (UID: "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:01.241246 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.241131    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-workload-certs" pod "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" (UID: "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:01.241384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.241358    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istiod-ca-cert" pod "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" (UID: "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:01.241598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.241580    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:01.241681 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.241606    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:01.241681 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.241621    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:01.241681 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.241636    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:01.241681 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.241649    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:01.242869 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.242845    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-kube-api-access-qff77" pod "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" (UID: "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9"). InnerVolumeSpecName "kube-api-access-qff77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:01.243027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.243006    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-envoy" pod "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" (UID: "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:01.243271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.243246    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-podinfo" pod "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" (UID: "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:01.243357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.243283    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-token" pod "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" (UID: "1617c4a9-d9b2-41b7-9907-c1d2d8878eb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:01.342465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.342432    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:01.342465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.342460    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:01.342465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.342470    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qff77\" (UniqueName: \"kubernetes.io/projected/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-kube-api-access-qff77\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:01.342771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.342484    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:01.423130 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.423049    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3477fee-0b54-47ee-9a25-b1c67c27e698" path="/var/lib/kubelet/pods/e3477fee-0b54-47ee-9a25-b1c67c27e698/volumes"
Oct 30 01:21:01.548858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.548823    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"]
Oct 30 01:21:01.554331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:01.554309    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-5kjx8"]
Oct 30 01:21:03.425964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:03.425925    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" path="/var/lib/kubelet/pods/1617c4a9-d9b2-41b7-9907-c1d2d8878eb9/volumes"
Oct 30 01:21:04.678648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:04.678612    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"]
Oct 30 01:21:07.555887 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:21:07.555845    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:21:08.513818 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:08.513782    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"]
Oct 30 01:21:09.610316 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:09.610281    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"]
Oct 30 01:21:09.612719 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:09.610494    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn" podUID="dabf895c-bcac-422b-96f8-1a7a548415a2" containerName="istio-proxy" containerID="cri-o://78fe4f8d25747628e95b836ef2c3c2e73667d19eafc5269714086759081b01f4" gracePeriod=30
Oct 30 01:21:13.205543 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:13.205511    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:21:14.915454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.915426    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:21:14.921512 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.921491    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-token\" (UniqueName: \"kubernetes.io/projected/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-token\") pod \"dabf895c-bcac-422b-96f8-1a7a548415a2\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") "
Oct 30 01:21:14.921613 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.921540    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-credential-socket\") pod \"dabf895c-bcac-422b-96f8-1a7a548415a2\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") "
Oct 30 01:21:14.921613 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.921570    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-workload-socket\") pod \"dabf895c-bcac-422b-96f8-1a7a548415a2\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") "
Oct 30 01:21:14.921736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.921635    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-envoy\") pod \"dabf895c-bcac-422b-96f8-1a7a548415a2\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") "
Oct 30 01:21:14.921736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.921689    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-podinfo\") pod \"dabf895c-bcac-422b-96f8-1a7a548415a2\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") "
Oct 30 01:21:14.921736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.921709    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-workload-certs\") pod \"dabf895c-bcac-422b-96f8-1a7a548415a2\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") "
Oct 30 01:21:14.921901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.921752    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dabf895c-bcac-422b-96f8-1a7a548415a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dabf895c-bcac-422b-96f8-1a7a548415a2-istiod-ca-cert\") pod \"dabf895c-bcac-422b-96f8-1a7a548415a2\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") "
Oct 30 01:21:14.921901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.921845    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dabf895c-bcac-422b-96f8-1a7a548415a2-kube-api-access-l4kkk\" (UniqueName: \"kubernetes.io/projected/dabf895c-bcac-422b-96f8-1a7a548415a2-kube-api-access-l4kkk\") pod \"dabf895c-bcac-422b-96f8-1a7a548415a2\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") "
Oct 30 01:21:14.922009 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.921920    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-data\") pod \"dabf895c-bcac-422b-96f8-1a7a548415a2\" (UID: \"dabf895c-bcac-422b-96f8-1a7a548415a2\") "
Oct 30 01:21:14.922009 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.921921    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-credential-socket" pod "dabf895c-bcac-422b-96f8-1a7a548415a2" (UID: "dabf895c-bcac-422b-96f8-1a7a548415a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:14.922185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.922003    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-workload-certs" pod "dabf895c-bcac-422b-96f8-1a7a548415a2" (UID: "dabf895c-bcac-422b-96f8-1a7a548415a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:14.922185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.921907    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-workload-socket" pod "dabf895c-bcac-422b-96f8-1a7a548415a2" (UID: "dabf895c-bcac-422b-96f8-1a7a548415a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:14.922185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.922128    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-data" pod "dabf895c-bcac-422b-96f8-1a7a548415a2" (UID: "dabf895c-bcac-422b-96f8-1a7a548415a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:14.922185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.922167    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dabf895c-bcac-422b-96f8-1a7a548415a2-istiod-ca-cert" pod "dabf895c-bcac-422b-96f8-1a7a548415a2" (UID: "dabf895c-bcac-422b-96f8-1a7a548415a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:14.922606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.922523    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dabf895c-bcac-422b-96f8-1a7a548415a2-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:14.922606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.922549    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:14.922606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.922562    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:14.922606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.922576    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:14.922606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.922588    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:14.923912 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.923891    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-token" pod "dabf895c-bcac-422b-96f8-1a7a548415a2" (UID: "dabf895c-bcac-422b-96f8-1a7a548415a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:14.924018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.924000    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dabf895c-bcac-422b-96f8-1a7a548415a2-kube-api-access-l4kkk" pod "dabf895c-bcac-422b-96f8-1a7a548415a2" (UID: "dabf895c-bcac-422b-96f8-1a7a548415a2"). InnerVolumeSpecName "kube-api-access-l4kkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:14.924506 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.924481    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-podinfo" pod "dabf895c-bcac-422b-96f8-1a7a548415a2" (UID: "dabf895c-bcac-422b-96f8-1a7a548415a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:14.924594 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.924533    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-envoy" pod "dabf895c-bcac-422b-96f8-1a7a548415a2" (UID: "dabf895c-bcac-422b-96f8-1a7a548415a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:14.992494 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.992454    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"]
Oct 30 01:21:14.992751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:14.992721    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz" podUID="a4598591-0f1b-4498-8b16-9353767f087f" containerName="istio-proxy" containerID="cri-o://e376804362c700266225c7697c8fd3348ecb6a9d8e1e46f34dda0819115dcefb" gracePeriod=30
Oct 30 01:21:15.023598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:15.023568    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:15.023598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:15.023599    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:15.023598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:15.023608    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dabf895c-bcac-422b-96f8-1a7a548415a2-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:15.023850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:15.023622    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4kkk\" (UniqueName: \"kubernetes.io/projected/dabf895c-bcac-422b-96f8-1a7a548415a2-kube-api-access-l4kkk\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:15.313238 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:15.313206    4425 generic.go:356] "Generic (PLEG): container finished" podID="dabf895c-bcac-422b-96f8-1a7a548415a2" containerID="78fe4f8d25747628e95b836ef2c3c2e73667d19eafc5269714086759081b01f4" exitCode=0
Oct 30 01:21:15.313438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:15.313266    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"
Oct 30 01:21:15.313438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:15.313300    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn" event={"ID":"dabf895c-bcac-422b-96f8-1a7a548415a2","Type":"ContainerDied","Data":"78fe4f8d25747628e95b836ef2c3c2e73667d19eafc5269714086759081b01f4"}
Oct 30 01:21:15.313438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:15.313360    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn" event={"ID":"dabf895c-bcac-422b-96f8-1a7a548415a2","Type":"ContainerDied","Data":"6b5f4a3605b184a30cdd2d4c2956db9b4fcf375f7f4641e05e3ed97efd6d359d"}
Oct 30 01:21:15.313597 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:15.313420    4425 scope.go:122] "RemoveContainer" containerID="78fe4f8d25747628e95b836ef2c3c2e73667d19eafc5269714086759081b01f4"
Oct 30 01:21:15.324759 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:15.324740    4425 scope.go:122] "RemoveContainer" containerID="78fe4f8d25747628e95b836ef2c3c2e73667d19eafc5269714086759081b01f4"
Oct 30 01:21:15.325097 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:21:15.325075    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78fe4f8d25747628e95b836ef2c3c2e73667d19eafc5269714086759081b01f4\": container with ID starting with 78fe4f8d25747628e95b836ef2c3c2e73667d19eafc5269714086759081b01f4 not found: ID does not exist" containerID="78fe4f8d25747628e95b836ef2c3c2e73667d19eafc5269714086759081b01f4"
Oct 30 01:21:15.325211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:15.325105    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78fe4f8d25747628e95b836ef2c3c2e73667d19eafc5269714086759081b01f4"} err="failed to get container status \"78fe4f8d25747628e95b836ef2c3c2e73667d19eafc5269714086759081b01f4\": rpc error: code = NotFound desc = could not find container \"78fe4f8d25747628e95b836ef2c3c2e73667d19eafc5269714086759081b01f4\": container with ID starting with 78fe4f8d25747628e95b836ef2c3c2e73667d19eafc5269714086759081b01f4 not found: ID does not exist"
Oct 30 01:21:15.363255 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:15.363218    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"]
Oct 30 01:21:15.373862 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:15.373819    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-r8tjn"]
Oct 30 01:21:15.441218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:15.435932    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dabf895c-bcac-422b-96f8-1a7a548415a2" path="/var/lib/kubelet/pods/dabf895c-bcac-422b-96f8-1a7a548415a2/volumes"
Oct 30 01:21:17.306228 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.306190    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"]
Oct 30 01:21:17.306772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.306758    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a650e14b-1236-47e8-b903-fdc7dd48c486" containerName="revoke-keys"
Oct 30 01:21:17.306817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.306773    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a650e14b-1236-47e8-b903-fdc7dd48c486" containerName="revoke-keys"
Oct 30 01:21:17.306817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.306789    4425 cpu_manager.go:414] "containerMap: removing container" podUID="e3477fee-0b54-47ee-9a25-b1c67c27e698" containerName="main"
Oct 30 01:21:17.306817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.306794    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3477fee-0b54-47ee-9a25-b1c67c27e698" containerName="main"
Oct 30 01:21:17.306927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.306819    4425 cpu_manager.go:414] "containerMap: removing container" podUID="1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" containerName="istio-proxy"
Oct 30 01:21:17.306927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.306825    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" containerName="istio-proxy"
Oct 30 01:21:17.306927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.306838    4425 cpu_manager.go:414] "containerMap: removing container" podUID="dabf895c-bcac-422b-96f8-1a7a548415a2" containerName="istio-proxy"
Oct 30 01:21:17.306927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.306843    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dabf895c-bcac-422b-96f8-1a7a548415a2" containerName="istio-proxy"
Oct 30 01:21:17.306927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.306918    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="1617c4a9-d9b2-41b7-9907-c1d2d8878eb9" containerName="istio-proxy"
Oct 30 01:21:17.307082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.306929    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="dabf895c-bcac-422b-96f8-1a7a548415a2" containerName="istio-proxy"
Oct 30 01:21:17.307082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.306939    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a650e14b-1236-47e8-b903-fdc7dd48c486" containerName="revoke-keys"
Oct 30 01:21:17.307082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.306946    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3477fee-0b54-47ee-9a25-b1c67c27e698" containerName="main"
Oct 30 01:21:17.312275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.312252    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"
Oct 30 01:21:17.326922 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.326893    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:21:17.336841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.336805    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:21:17.336994 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.336818    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:21:17.337228 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.337208    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:21:17.342456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.342434    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"]
Oct 30 01:21:17.357670 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.357640    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c982fa3b-8d27-4200-afcf-ddbe8f098d95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f154713c-swxmm\" (UID: \"c982fa3b-8d27-4200-afcf-ddbe8f098d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"
Oct 30 01:21:17.357805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.357697    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c982fa3b-8d27-4200-afcf-ddbe8f098d95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f154713c-swxmm\" (UID: \"c982fa3b-8d27-4200-afcf-ddbe8f098d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"
Oct 30 01:21:17.357875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.357802    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggzm9\" (UniqueName: \"kubernetes.io/projected/c982fa3b-8d27-4200-afcf-ddbe8f098d95-kube-api-access-ggzm9\") pod \"maas-api-revoke-sub-f154713c-swxmm\" (UID: \"c982fa3b-8d27-4200-afcf-ddbe8f098d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"
Oct 30 01:21:17.458655 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.458622    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggzm9\" (UniqueName: \"kubernetes.io/projected/c982fa3b-8d27-4200-afcf-ddbe8f098d95-kube-api-access-ggzm9\") pod \"maas-api-revoke-sub-f154713c-swxmm\" (UID: \"c982fa3b-8d27-4200-afcf-ddbe8f098d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"
Oct 30 01:21:17.458830 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.458760    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c982fa3b-8d27-4200-afcf-ddbe8f098d95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f154713c-swxmm\" (UID: \"c982fa3b-8d27-4200-afcf-ddbe8f098d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"
Oct 30 01:21:17.458830 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.458807    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c982fa3b-8d27-4200-afcf-ddbe8f098d95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f154713c-swxmm\" (UID: \"c982fa3b-8d27-4200-afcf-ddbe8f098d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"
Oct 30 01:21:17.459793 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.459696    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c982fa3b-8d27-4200-afcf-ddbe8f098d95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f154713c-swxmm\" (UID: \"c982fa3b-8d27-4200-afcf-ddbe8f098d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"
Oct 30 01:21:17.461741 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.461717    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c982fa3b-8d27-4200-afcf-ddbe8f098d95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f154713c-swxmm\" (UID: \"c982fa3b-8d27-4200-afcf-ddbe8f098d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"
Oct 30 01:21:17.479751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.479723    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggzm9\" (UniqueName: \"kubernetes.io/projected/c982fa3b-8d27-4200-afcf-ddbe8f098d95-kube-api-access-ggzm9\") pod \"maas-api-revoke-sub-f154713c-swxmm\" (UID: \"c982fa3b-8d27-4200-afcf-ddbe8f098d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"
Oct 30 01:21:17.628101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.628006    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"
Oct 30 01:21:17.933804 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:17.933768    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"]
Oct 30 01:21:17.935977 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:21:17.935948    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc982fa3b_8d27_4200_afcf_ddbe8f098d95.slice/crio-4384a9f578db1870a61f663c53cd78fa75e2811c154ec1880316df1ecf42851e WatchSource:0}: Error finding container 4384a9f578db1870a61f663c53cd78fa75e2811c154ec1880316df1ecf42851e: Status 404 returned error can't find the container with id 4384a9f578db1870a61f663c53cd78fa75e2811c154ec1880316df1ecf42851e
Oct 30 01:21:18.340233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:18.340190    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm" event={"ID":"c982fa3b-8d27-4200-afcf-ddbe8f098d95","Type":"ContainerStarted","Data":"4384a9f578db1870a61f663c53cd78fa75e2811c154ec1880316df1ecf42851e"}
Oct 30 01:21:19.350170 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:19.350122    4425 generic.go:356] "Generic (PLEG): container finished" podID="c982fa3b-8d27-4200-afcf-ddbe8f098d95" containerID="fd15cc29229eaeb3d694ecf9f7efe71b5675c82f67e0acbd401e1145a5a655a6" exitCode=0
Oct 30 01:21:19.350596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:19.350207    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm" event={"ID":"c982fa3b-8d27-4200-afcf-ddbe8f098d95","Type":"ContainerDied","Data":"fd15cc29229eaeb3d694ecf9f7efe71b5675c82f67e0acbd401e1145a5a655a6"}
Oct 30 01:21:19.700029 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:19.699911    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk" podUID="3dbbc68a-083b-4a93-9f03-55c2660151aa" containerName="main" containerID="cri-o://ea94241c38d56f6fb716a8ddc25dced28851c06eb0086f885035674a4f7a6802" gracePeriod=45
Oct 30 01:21:20.000064 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.000034    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:21:20.093610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.093574    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-model-cache\") pod \"3dbbc68a-083b-4a93-9f03-55c2660151aa\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") "
Oct 30 01:21:20.093782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.093640    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-home\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-home\") pod \"3dbbc68a-083b-4a93-9f03-55c2660151aa\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") "
Oct 30 01:21:20.093782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.093702    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dbbc68a-083b-4a93-9f03-55c2660151aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/3dbbc68a-083b-4a93-9f03-55c2660151aa-tls-certs\") pod \"3dbbc68a-083b-4a93-9f03-55c2660151aa\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") "
Oct 30 01:21:20.093876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.093802    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-dshm\") pod \"3dbbc68a-083b-4a93-9f03-55c2660151aa\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") "
Oct 30 01:21:20.093876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.093846    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-model-cache" pod "3dbbc68a-083b-4a93-9f03-55c2660151aa" (UID: "3dbbc68a-083b-4a93-9f03-55c2660151aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:20.093991 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.093899    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-home" pod "3dbbc68a-083b-4a93-9f03-55c2660151aa" (UID: "3dbbc68a-083b-4a93-9f03-55c2660151aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:20.094053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.093999    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-tmp-dir\") pod \"3dbbc68a-083b-4a93-9f03-55c2660151aa\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") "
Oct 30 01:21:20.094114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.094067    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbbc68a-083b-4a93-9f03-55c2660151aa-kube-api-access-4lrfm\" (UniqueName: \"kubernetes.io/projected/3dbbc68a-083b-4a93-9f03-55c2660151aa-kube-api-access-4lrfm\") pod \"3dbbc68a-083b-4a93-9f03-55c2660151aa\" (UID: \"3dbbc68a-083b-4a93-9f03-55c2660151aa\") "
Oct 30 01:21:20.094268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.094238    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-tmp-dir" pod "3dbbc68a-083b-4a93-9f03-55c2660151aa" (UID: "3dbbc68a-083b-4a93-9f03-55c2660151aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:20.094625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.094606    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.094743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.094628    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.094743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.094642    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.096007 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.095983    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-dshm" pod "3dbbc68a-083b-4a93-9f03-55c2660151aa" (UID: "3dbbc68a-083b-4a93-9f03-55c2660151aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:20.096117 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.096101    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbbc68a-083b-4a93-9f03-55c2660151aa-kube-api-access-4lrfm" pod "3dbbc68a-083b-4a93-9f03-55c2660151aa" (UID: "3dbbc68a-083b-4a93-9f03-55c2660151aa"). InnerVolumeSpecName "kube-api-access-4lrfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:20.096182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.096125    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dbbc68a-083b-4a93-9f03-55c2660151aa-tls-certs" pod "3dbbc68a-083b-4a93-9f03-55c2660151aa" (UID: "3dbbc68a-083b-4a93-9f03-55c2660151aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:20.195866 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.195831    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dbbc68a-083b-4a93-9f03-55c2660151aa-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.195866 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.195866    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lrfm\" (UniqueName: \"kubernetes.io/projected/3dbbc68a-083b-4a93-9f03-55c2660151aa-kube-api-access-4lrfm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.196120 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.195881    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dbbc68a-083b-4a93-9f03-55c2660151aa-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.263079 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.263052    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:21:20.297251 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.297205    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-workload-socket\") pod \"a4598591-0f1b-4498-8b16-9353767f087f\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") "
Oct 30 01:21:20.297455 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.297271    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-istio-envoy\") pod \"a4598591-0f1b-4498-8b16-9353767f087f\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") "
Oct 30 01:21:20.297455 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.297332    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4598591-0f1b-4498-8b16-9353767f087f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4598591-0f1b-4498-8b16-9353767f087f-istiod-ca-cert\") pod \"a4598591-0f1b-4498-8b16-9353767f087f\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") "
Oct 30 01:21:20.297586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.297501    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-workload-socket" pod "a4598591-0f1b-4498-8b16-9353767f087f" (UID: "a4598591-0f1b-4498-8b16-9353767f087f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:20.297586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.297514    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4598591-0f1b-4498-8b16-9353767f087f-kube-api-access-znjkk\" (UniqueName: \"kubernetes.io/projected/a4598591-0f1b-4498-8b16-9353767f087f-kube-api-access-znjkk\") pod \"a4598591-0f1b-4498-8b16-9353767f087f\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") "
Oct 30 01:21:20.297697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.297641    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4598591-0f1b-4498-8b16-9353767f087f-istio-token\" (UniqueName: \"kubernetes.io/projected/a4598591-0f1b-4498-8b16-9353767f087f-istio-token\") pod \"a4598591-0f1b-4498-8b16-9353767f087f\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") "
Oct 30 01:21:20.297697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.297677    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-workload-certs\") pod \"a4598591-0f1b-4498-8b16-9353767f087f\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") "
Oct 30 01:21:20.297779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.297731    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4598591-0f1b-4498-8b16-9353767f087f-istiod-ca-cert" pod "a4598591-0f1b-4498-8b16-9353767f087f" (UID: "a4598591-0f1b-4498-8b16-9353767f087f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:20.297779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.297735    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-credential-socket\") pod \"a4598591-0f1b-4498-8b16-9353767f087f\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") "
Oct 30 01:21:20.297878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.297800    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-istio-data\") pod \"a4598591-0f1b-4498-8b16-9353767f087f\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") "
Oct 30 01:21:20.297949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.297847    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4598591-0f1b-4498-8b16-9353767f087f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4598591-0f1b-4498-8b16-9353767f087f-istio-podinfo\") pod \"a4598591-0f1b-4498-8b16-9353767f087f\" (UID: \"a4598591-0f1b-4498-8b16-9353767f087f\") "
Oct 30 01:21:20.298056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.297936    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-credential-socket" pod "a4598591-0f1b-4498-8b16-9353767f087f" (UID: "a4598591-0f1b-4498-8b16-9353767f087f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:20.298391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.298365    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-workload-certs" pod "a4598591-0f1b-4498-8b16-9353767f087f" (UID: "a4598591-0f1b-4498-8b16-9353767f087f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:20.298519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.298374    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-istio-data" pod "a4598591-0f1b-4498-8b16-9353767f087f" (UID: "a4598591-0f1b-4498-8b16-9353767f087f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:20.298624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.298414    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.298624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.298543    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.298624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.298561    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4598591-0f1b-4498-8b16-9353767f087f-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.299977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.299956    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-istio-envoy" pod "a4598591-0f1b-4498-8b16-9353767f087f" (UID: "a4598591-0f1b-4498-8b16-9353767f087f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:20.300077 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.300012    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4598591-0f1b-4498-8b16-9353767f087f-istio-token" pod "a4598591-0f1b-4498-8b16-9353767f087f" (UID: "a4598591-0f1b-4498-8b16-9353767f087f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:20.300077 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.300037    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4598591-0f1b-4498-8b16-9353767f087f-istio-podinfo" pod "a4598591-0f1b-4498-8b16-9353767f087f" (UID: "a4598591-0f1b-4498-8b16-9353767f087f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:20.300261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.300245    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4598591-0f1b-4498-8b16-9353767f087f-kube-api-access-znjkk" pod "a4598591-0f1b-4498-8b16-9353767f087f" (UID: "a4598591-0f1b-4498-8b16-9353767f087f"). InnerVolumeSpecName "kube-api-access-znjkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:20.359319 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.359288    4425 generic.go:356] "Generic (PLEG): container finished" podID="a4598591-0f1b-4498-8b16-9353767f087f" containerID="e376804362c700266225c7697c8fd3348ecb6a9d8e1e46f34dda0819115dcefb" exitCode=0
Oct 30 01:21:20.359756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.359335    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"
Oct 30 01:21:20.359756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.359368    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz" event={"ID":"a4598591-0f1b-4498-8b16-9353767f087f","Type":"ContainerDied","Data":"e376804362c700266225c7697c8fd3348ecb6a9d8e1e46f34dda0819115dcefb"}
Oct 30 01:21:20.359756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.359441    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz" event={"ID":"a4598591-0f1b-4498-8b16-9353767f087f","Type":"ContainerDied","Data":"6b301d5d9eb000920534d829b65870d2797dcf0fe3925d52ee5839f93a3d5c97"}
Oct 30 01:21:20.359756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.359466    4425 scope.go:122] "RemoveContainer" containerID="e376804362c700266225c7697c8fd3348ecb6a9d8e1e46f34dda0819115dcefb"
Oct 30 01:21:20.360924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.360901    4425 generic.go:356] "Generic (PLEG): container finished" podID="3dbbc68a-083b-4a93-9f03-55c2660151aa" containerID="ea94241c38d56f6fb716a8ddc25dced28851c06eb0086f885035674a4f7a6802" exitCode=0
Oct 30 01:21:20.361030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.360943    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"
Oct 30 01:21:20.361030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.360985    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk" event={"ID":"3dbbc68a-083b-4a93-9f03-55c2660151aa","Type":"ContainerDied","Data":"ea94241c38d56f6fb716a8ddc25dced28851c06eb0086f885035674a4f7a6802"}
Oct 30 01:21:20.361030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.361021    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk" event={"ID":"3dbbc68a-083b-4a93-9f03-55c2660151aa","Type":"ContainerDied","Data":"d3f0e6b8d28b781a8f8fba4c7aba44936275671a3ab0504fb45b09a36b6ecde1"}
Oct 30 01:21:20.371531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.371505    4425 scope.go:122] "RemoveContainer" containerID="e376804362c700266225c7697c8fd3348ecb6a9d8e1e46f34dda0819115dcefb"
Oct 30 01:21:20.371853 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:21:20.371831    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e376804362c700266225c7697c8fd3348ecb6a9d8e1e46f34dda0819115dcefb\": container with ID starting with e376804362c700266225c7697c8fd3348ecb6a9d8e1e46f34dda0819115dcefb not found: ID does not exist" containerID="e376804362c700266225c7697c8fd3348ecb6a9d8e1e46f34dda0819115dcefb"
Oct 30 01:21:20.371964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.371862    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e376804362c700266225c7697c8fd3348ecb6a9d8e1e46f34dda0819115dcefb"} err="failed to get container status \"e376804362c700266225c7697c8fd3348ecb6a9d8e1e46f34dda0819115dcefb\": rpc error: code = NotFound desc = could not find container \"e376804362c700266225c7697c8fd3348ecb6a9d8e1e46f34dda0819115dcefb\": container with ID starting with e376804362c700266225c7697c8fd3348ecb6a9d8e1e46f34dda0819115dcefb not found: ID does not exist"
Oct 30 01:21:20.371964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.371881    4425 scope.go:122] "RemoveContainer" containerID="ea94241c38d56f6fb716a8ddc25dced28851c06eb0086f885035674a4f7a6802"
Oct 30 01:21:20.385714 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.385290    4425 scope.go:122] "RemoveContainer" containerID="ea94241c38d56f6fb716a8ddc25dced28851c06eb0086f885035674a4f7a6802"
Oct 30 01:21:20.385973 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:21:20.385941    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea94241c38d56f6fb716a8ddc25dced28851c06eb0086f885035674a4f7a6802\": container with ID starting with ea94241c38d56f6fb716a8ddc25dced28851c06eb0086f885035674a4f7a6802 not found: ID does not exist" containerID="ea94241c38d56f6fb716a8ddc25dced28851c06eb0086f885035674a4f7a6802"
Oct 30 01:21:20.386083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.385983    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea94241c38d56f6fb716a8ddc25dced28851c06eb0086f885035674a4f7a6802"} err="failed to get container status \"ea94241c38d56f6fb716a8ddc25dced28851c06eb0086f885035674a4f7a6802\": rpc error: code = NotFound desc = could not find container \"ea94241c38d56f6fb716a8ddc25dced28851c06eb0086f885035674a4f7a6802\": container with ID starting with ea94241c38d56f6fb716a8ddc25dced28851c06eb0086f885035674a4f7a6802 not found: ID does not exist"
Oct 30 01:21:20.399595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.399571    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.399595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.399597    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znjkk\" (UniqueName: \"kubernetes.io/projected/a4598591-0f1b-4498-8b16-9353767f087f-kube-api-access-znjkk\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.399765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.399606    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4598591-0f1b-4498-8b16-9353767f087f-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.399765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.399621    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.399765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.399633    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4598591-0f1b-4498-8b16-9353767f087f-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.399765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.399644    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4598591-0f1b-4498-8b16-9353767f087f-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.408885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.408831    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"]
Oct 30 01:21:20.413506 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.413479    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-rlnwz"]
Oct 30 01:21:20.441895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.441864    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"]
Oct 30 01:21:20.448910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.448878    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk"]
Oct 30 01:21:20.506420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.506354    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"
Oct 30 01:21:20.601667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.601634    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c982fa3b-8d27-4200-afcf-ddbe8f098d95-kube-api-access-ggzm9\" (UniqueName: \"kubernetes.io/projected/c982fa3b-8d27-4200-afcf-ddbe8f098d95-kube-api-access-ggzm9\") pod \"c982fa3b-8d27-4200-afcf-ddbe8f098d95\" (UID: \"c982fa3b-8d27-4200-afcf-ddbe8f098d95\") "
Oct 30 01:21:20.601868 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.601709    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c982fa3b-8d27-4200-afcf-ddbe8f098d95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c982fa3b-8d27-4200-afcf-ddbe8f098d95-maas-api-xks-ca\") pod \"c982fa3b-8d27-4200-afcf-ddbe8f098d95\" (UID: \"c982fa3b-8d27-4200-afcf-ddbe8f098d95\") "
Oct 30 01:21:20.601868 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.601804    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c982fa3b-8d27-4200-afcf-ddbe8f098d95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c982fa3b-8d27-4200-afcf-ddbe8f098d95-maas-api-openshift-ca\") pod \"c982fa3b-8d27-4200-afcf-ddbe8f098d95\" (UID: \"c982fa3b-8d27-4200-afcf-ddbe8f098d95\") "
Oct 30 01:21:20.602147 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.602121    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c982fa3b-8d27-4200-afcf-ddbe8f098d95-maas-api-openshift-ca" pod "c982fa3b-8d27-4200-afcf-ddbe8f098d95" (UID: "c982fa3b-8d27-4200-afcf-ddbe8f098d95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:20.602345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.602327    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c982fa3b-8d27-4200-afcf-ddbe8f098d95-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.603841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.603817    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c982fa3b-8d27-4200-afcf-ddbe8f098d95-maas-api-xks-ca" pod "c982fa3b-8d27-4200-afcf-ddbe8f098d95" (UID: "c982fa3b-8d27-4200-afcf-ddbe8f098d95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:20.603924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.603904    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c982fa3b-8d27-4200-afcf-ddbe8f098d95-kube-api-access-ggzm9" pod "c982fa3b-8d27-4200-afcf-ddbe8f098d95" (UID: "c982fa3b-8d27-4200-afcf-ddbe8f098d95"). InnerVolumeSpecName "kube-api-access-ggzm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:20.704022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.703987    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggzm9\" (UniqueName: \"kubernetes.io/projected/c982fa3b-8d27-4200-afcf-ddbe8f098d95-kube-api-access-ggzm9\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.704022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.704017    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c982fa3b-8d27-4200-afcf-ddbe8f098d95-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:20.989446 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:20.989395    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-1854c4/auth-test-model-1854c4-kserve-7955bdbb97-jdnxk" podUID="3dbbc68a-083b-4a93-9f03-55c2660151aa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:21:21.370305 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.370275    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"
Oct 30 01:21:21.370744 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.370274    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm" event={"ID":"c982fa3b-8d27-4200-afcf-ddbe8f098d95","Type":"ContainerDied","Data":"4384a9f578db1870a61f663c53cd78fa75e2811c154ec1880316df1ecf42851e"}
Oct 30 01:21:21.370744 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.370415    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4384a9f578db1870a61f663c53cd78fa75e2811c154ec1880316df1ecf42851e"
Oct 30 01:21:21.398644 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.398605    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"]
Oct 30 01:21:21.399281 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.399264    4425 cpu_manager.go:414] "containerMap: removing container" podUID="3dbbc68a-083b-4a93-9f03-55c2660151aa" containerName="main"
Oct 30 01:21:21.399327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.399283    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbbc68a-083b-4a93-9f03-55c2660151aa" containerName="main"
Oct 30 01:21:21.399327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.399297    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a4598591-0f1b-4498-8b16-9353767f087f" containerName="istio-proxy"
Oct 30 01:21:21.399327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.399303    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4598591-0f1b-4498-8b16-9353767f087f" containerName="istio-proxy"
Oct 30 01:21:21.399327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.399312    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c982fa3b-8d27-4200-afcf-ddbe8f098d95" containerName="revoke-keys"
Oct 30 01:21:21.399327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.399317    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c982fa3b-8d27-4200-afcf-ddbe8f098d95" containerName="revoke-keys"
Oct 30 01:21:21.399489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.399425    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4598591-0f1b-4498-8b16-9353767f087f" containerName="istio-proxy"
Oct 30 01:21:21.399489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.399434    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c982fa3b-8d27-4200-afcf-ddbe8f098d95" containerName="revoke-keys"
Oct 30 01:21:21.399489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.399447    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbbc68a-083b-4a93-9f03-55c2660151aa" containerName="main"
Oct 30 01:21:21.403440 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.403420    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"
Oct 30 01:21:21.423139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.423090    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-499482\""
Oct 30 01:21:21.424207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.423198    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-499482-dockercfg-6m9s7\""
Oct 30 01:21:21.426950 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.426924    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbbc68a-083b-4a93-9f03-55c2660151aa" path="/var/lib/kubelet/pods/3dbbc68a-083b-4a93-9f03-55c2660151aa/volumes"
Oct 30 01:21:21.427491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.427474    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4598591-0f1b-4498-8b16-9353767f087f" path="/var/lib/kubelet/pods/a4598591-0f1b-4498-8b16-9353767f087f/volumes"
Oct 30 01:21:21.427897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.427879    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"]
Oct 30 01:21:21.504279 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.500236    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"]
Oct 30 01:21:21.508416 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.508382    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"
Oct 30 01:21:21.513044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.513005    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b86ac41-fb47-4c59-8723-9223d01f49f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t\" (UID: \"8b86ac41-fb47-4c59-8723-9223d01f49f3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"
Oct 30 01:21:21.513345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.513296    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snzqf\" (UniqueName: \"kubernetes.io/projected/8b86ac41-fb47-4c59-8723-9223d01f49f3-kube-api-access-snzqf\") pod \"payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t\" (UID: \"8b86ac41-fb47-4c59-8723-9223d01f49f3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"
Oct 30 01:21:21.527203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.521039    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"]
Oct 30 01:21:21.614403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.614366    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fecb6d-0b3b-4b1c-9806-74e68559265a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm\" (UID: \"d2fecb6d-0b3b-4b1c-9806-74e68559265a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"
Oct 30 01:21:21.614403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.614408    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b86ac41-fb47-4c59-8723-9223d01f49f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t\" (UID: \"8b86ac41-fb47-4c59-8723-9223d01f49f3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"
Oct 30 01:21:21.614672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.614496    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmrcz\" (UniqueName: \"kubernetes.io/projected/d2fecb6d-0b3b-4b1c-9806-74e68559265a-kube-api-access-hmrcz\") pod \"payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm\" (UID: \"d2fecb6d-0b3b-4b1c-9806-74e68559265a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"
Oct 30 01:21:21.614739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.614701    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snzqf\" (UniqueName: \"kubernetes.io/projected/8b86ac41-fb47-4c59-8723-9223d01f49f3-kube-api-access-snzqf\") pod \"payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t\" (UID: \"8b86ac41-fb47-4c59-8723-9223d01f49f3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"
Oct 30 01:21:21.615147 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.615127    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b86ac41-fb47-4c59-8723-9223d01f49f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t\" (UID: \"8b86ac41-fb47-4c59-8723-9223d01f49f3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"
Oct 30 01:21:21.636314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.636282    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snzqf\" (UniqueName: \"kubernetes.io/projected/8b86ac41-fb47-4c59-8723-9223d01f49f3-kube-api-access-snzqf\") pod \"payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t\" (UID: \"8b86ac41-fb47-4c59-8723-9223d01f49f3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"
Oct 30 01:21:21.715669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.715630    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fecb6d-0b3b-4b1c-9806-74e68559265a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm\" (UID: \"d2fecb6d-0b3b-4b1c-9806-74e68559265a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"
Oct 30 01:21:21.715878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.715685    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmrcz\" (UniqueName: \"kubernetes.io/projected/d2fecb6d-0b3b-4b1c-9806-74e68559265a-kube-api-access-hmrcz\") pod \"payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm\" (UID: \"d2fecb6d-0b3b-4b1c-9806-74e68559265a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"
Oct 30 01:21:21.716401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.716377    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fecb6d-0b3b-4b1c-9806-74e68559265a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm\" (UID: \"d2fecb6d-0b3b-4b1c-9806-74e68559265a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"
Oct 30 01:21:21.718262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.718242    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"
Oct 30 01:21:21.742647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.742599    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmrcz\" (UniqueName: \"kubernetes.io/projected/d2fecb6d-0b3b-4b1c-9806-74e68559265a-kube-api-access-hmrcz\") pod \"payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm\" (UID: \"d2fecb6d-0b3b-4b1c-9806-74e68559265a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"
Oct 30 01:21:21.846242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:21.846202    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"
Oct 30 01:21:22.064024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:22.063999    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"]
Oct 30 01:21:22.065636 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:21:22.065610    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b86ac41_fb47_4c59_8723_9223d01f49f3.slice/crio-13d1f9b8292c9ceac4da0e5d3ebf424853264f4d2ab0a8ef0128d755e02e80dc WatchSource:0}: Error finding container 13d1f9b8292c9ceac4da0e5d3ebf424853264f4d2ab0a8ef0128d755e02e80dc: Status 404 returned error can't find the container with id 13d1f9b8292c9ceac4da0e5d3ebf424853264f4d2ab0a8ef0128d755e02e80dc
Oct 30 01:21:22.150649 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:22.150617    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"]
Oct 30 01:21:22.151048 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:21:22.151023    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2fecb6d_0b3b_4b1c_9806_74e68559265a.slice/crio-66e8b7cc31f04549b6d80a4a59dea67601afcc29d3f5561d0238797bc399ec13 WatchSource:0}: Error finding container 66e8b7cc31f04549b6d80a4a59dea67601afcc29d3f5561d0238797bc399ec13: Status 404 returned error can't find the container with id 66e8b7cc31f04549b6d80a4a59dea67601afcc29d3f5561d0238797bc399ec13
Oct 30 01:21:22.382101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:22.381997    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t" event={"ID":"8b86ac41-fb47-4c59-8723-9223d01f49f3","Type":"ContainerStarted","Data":"c58c54a09c3ae43aa427c5d4177dacfd9fa0fcc23903af8ed1e20a9cecc66f41"}
Oct 30 01:21:22.382101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:22.382079    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t" event={"ID":"8b86ac41-fb47-4c59-8723-9223d01f49f3","Type":"ContainerStarted","Data":"13d1f9b8292c9ceac4da0e5d3ebf424853264f4d2ab0a8ef0128d755e02e80dc"}
Oct 30 01:21:22.382653 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:22.382107    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"
Oct 30 01:21:22.384102 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:22.384071    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm" event={"ID":"d2fecb6d-0b3b-4b1c-9806-74e68559265a","Type":"ContainerStarted","Data":"0ad555280f5a67d164712906536283da051527390cd413001c6b33790be0ff44"}
Oct 30 01:21:22.384265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:22.384123    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm" event={"ID":"d2fecb6d-0b3b-4b1c-9806-74e68559265a","Type":"ContainerStarted","Data":"66e8b7cc31f04549b6d80a4a59dea67601afcc29d3f5561d0238797bc399ec13"}
Oct 30 01:21:22.384265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:22.384176    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"
Oct 30 01:21:22.423223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:22.423167    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t" podStartSLOduration=1.423136043 podStartE2EDuration="1.423136043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:22.419314999 +0000 UTC m=+2391.168970289" watchObservedRunningTime="2026-10-30 01:21:22.423136043 +0000 UTC m=+2391.172791395"
Oct 30 01:21:22.455927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:22.455872    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm" podStartSLOduration=1.455859518 podStartE2EDuration="1.455859518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:22.455667267 +0000 UTC m=+2391.205322576" watchObservedRunningTime="2026-10-30 01:21:22.455859518 +0000 UTC m=+2391.205514827"
Oct 30 01:21:23.539484 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:23.539448    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv" podUID="0cc6610f-e2c9-4802-bba2-e3df8f40c9e0" containerName="main" containerID="cri-o://a56f8d29e4a1e58e5a4896247d50d595b5c5f2df36d83d76c1a29f74a642d07f" gracePeriod=45
Oct 30 01:21:23.816263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:23.816236    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:21:23.945956 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:23.945923    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-tmp-dir\") pod \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") "
Oct 30 01:21:23.946135 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:23.945979    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-home\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-home\") pod \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") "
Oct 30 01:21:23.946135 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:23.946030    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-kube-api-access-f9j8f\" (UniqueName: \"kubernetes.io/projected/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-kube-api-access-f9j8f\") pod \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") "
Oct 30 01:21:23.946135 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:23.946102    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-tls-certs\") pod \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") "
Oct 30 01:21:23.946267 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:23.946255    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-model-cache\") pod \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") "
Oct 30 01:21:23.946309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:23.946281    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-tmp-dir" pod "0cc6610f-e2c9-4802-bba2-e3df8f40c9e0" (UID: "0cc6610f-e2c9-4802-bba2-e3df8f40c9e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:23.946351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:23.946307    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-dshm\") pod \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\" (UID: \"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0\") "
Oct 30 01:21:23.946351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:23.946293    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-home" pod "0cc6610f-e2c9-4802-bba2-e3df8f40c9e0" (UID: "0cc6610f-e2c9-4802-bba2-e3df8f40c9e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:23.946504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:23.946487    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-model-cache" pod "0cc6610f-e2c9-4802-bba2-e3df8f40c9e0" (UID: "0cc6610f-e2c9-4802-bba2-e3df8f40c9e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:23.946753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:23.946730    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:23.946753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:23.946748    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:23.946753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:23.946758    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:23.948498 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:23.948474    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-tls-certs" pod "0cc6610f-e2c9-4802-bba2-e3df8f40c9e0" (UID: "0cc6610f-e2c9-4802-bba2-e3df8f40c9e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:23.948626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:23.948505    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-kube-api-access-f9j8f" pod "0cc6610f-e2c9-4802-bba2-e3df8f40c9e0" (UID: "0cc6610f-e2c9-4802-bba2-e3df8f40c9e0"). InnerVolumeSpecName "kube-api-access-f9j8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:23.948626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:23.948510    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-dshm" pod "0cc6610f-e2c9-4802-bba2-e3df8f40c9e0" (UID: "0cc6610f-e2c9-4802-bba2-e3df8f40c9e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:24.047573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:24.047489    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9j8f\" (UniqueName: \"kubernetes.io/projected/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-kube-api-access-f9j8f\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:24.047573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:24.047517    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:24.047573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:24.047532    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:24.397596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:24.397510    4425 generic.go:356] "Generic (PLEG): container finished" podID="0cc6610f-e2c9-4802-bba2-e3df8f40c9e0" containerID="a56f8d29e4a1e58e5a4896247d50d595b5c5f2df36d83d76c1a29f74a642d07f" exitCode=0
Oct 30 01:21:24.397596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:24.397561    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"
Oct 30 01:21:24.397827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:24.397609    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv" event={"ID":"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0","Type":"ContainerDied","Data":"a56f8d29e4a1e58e5a4896247d50d595b5c5f2df36d83d76c1a29f74a642d07f"}
Oct 30 01:21:24.397827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:24.397669    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv" event={"ID":"0cc6610f-e2c9-4802-bba2-e3df8f40c9e0","Type":"ContainerDied","Data":"1f103f5503f4e553219fabeefcf56e0fd3100f1d4b2c2fc6bf5a1532d71eff18"}
Oct 30 01:21:24.397827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:24.397684    4425 scope.go:122] "RemoveContainer" containerID="a56f8d29e4a1e58e5a4896247d50d595b5c5f2df36d83d76c1a29f74a642d07f"
Oct 30 01:21:24.408995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:24.408976    4425 scope.go:122] "RemoveContainer" containerID="a56f8d29e4a1e58e5a4896247d50d595b5c5f2df36d83d76c1a29f74a642d07f"
Oct 30 01:21:24.409287 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:21:24.409266    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56f8d29e4a1e58e5a4896247d50d595b5c5f2df36d83d76c1a29f74a642d07f\": container with ID starting with a56f8d29e4a1e58e5a4896247d50d595b5c5f2df36d83d76c1a29f74a642d07f not found: ID does not exist" containerID="a56f8d29e4a1e58e5a4896247d50d595b5c5f2df36d83d76c1a29f74a642d07f"
Oct 30 01:21:24.409343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:24.409296    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56f8d29e4a1e58e5a4896247d50d595b5c5f2df36d83d76c1a29f74a642d07f"} err="failed to get container status \"a56f8d29e4a1e58e5a4896247d50d595b5c5f2df36d83d76c1a29f74a642d07f\": rpc error: code = NotFound desc = could not find container \"a56f8d29e4a1e58e5a4896247d50d595b5c5f2df36d83d76c1a29f74a642d07f\": container with ID starting with a56f8d29e4a1e58e5a4896247d50d595b5c5f2df36d83d76c1a29f74a642d07f not found: ID does not exist"
Oct 30 01:21:24.447149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:24.447125    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"]
Oct 30 01:21:24.450710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:24.450689    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8ae41e/auth-test-model-8ae41e-kserve-5cccb5c877-t5vnv"]
Oct 30 01:21:25.449434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:25.449391    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc6610f-e2c9-4802-bba2-e3df8f40c9e0" path="/var/lib/kubelet/pods/0cc6610f-e2c9-4802-bba2-e3df8f40c9e0/volumes"
Oct 30 01:21:27.709822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.709783    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"]
Oct 30 01:21:27.710414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.710398    4425 cpu_manager.go:414] "containerMap: removing container" podUID="0cc6610f-e2c9-4802-bba2-e3df8f40c9e0" containerName="main"
Oct 30 01:21:27.710414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.710415    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc6610f-e2c9-4802-bba2-e3df8f40c9e0" containerName="main"
Oct 30 01:21:27.710570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.710560    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc6610f-e2c9-4802-bba2-e3df8f40c9e0" containerName="main"
Oct 30 01:21:27.716022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.715994    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"
Oct 30 01:21:27.733892 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.733856    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:21:27.735176 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.735131    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"]
Oct 30 01:21:27.740132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.740104    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:21:27.740339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.740320    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:21:27.740446 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.740430    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:21:27.796291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.796249    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc25p\" (UniqueName: \"kubernetes.io/projected/457e9435-2882-4e59-be1d-8bec75b53f1e-kube-api-access-dc25p\") pod \"maas-api-revoke-sub-ca4c4487-27679\" (UID: \"457e9435-2882-4e59-be1d-8bec75b53f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"
Oct 30 01:21:27.796489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.796434    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457e9435-2882-4e59-be1d-8bec75b53f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca4c4487-27679\" (UID: \"457e9435-2882-4e59-be1d-8bec75b53f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"
Oct 30 01:21:27.796489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.796466    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457e9435-2882-4e59-be1d-8bec75b53f1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca4c4487-27679\" (UID: \"457e9435-2882-4e59-be1d-8bec75b53f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"
Oct 30 01:21:27.898391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.898340    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457e9435-2882-4e59-be1d-8bec75b53f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca4c4487-27679\" (UID: \"457e9435-2882-4e59-be1d-8bec75b53f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"
Oct 30 01:21:27.898391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.898403    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457e9435-2882-4e59-be1d-8bec75b53f1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca4c4487-27679\" (UID: \"457e9435-2882-4e59-be1d-8bec75b53f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"
Oct 30 01:21:27.898682 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.898538    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc25p\" (UniqueName: \"kubernetes.io/projected/457e9435-2882-4e59-be1d-8bec75b53f1e-kube-api-access-dc25p\") pod \"maas-api-revoke-sub-ca4c4487-27679\" (UID: \"457e9435-2882-4e59-be1d-8bec75b53f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"
Oct 30 01:21:27.900193 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.899343    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457e9435-2882-4e59-be1d-8bec75b53f1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca4c4487-27679\" (UID: \"457e9435-2882-4e59-be1d-8bec75b53f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"
Oct 30 01:21:27.902636 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.902607    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457e9435-2882-4e59-be1d-8bec75b53f1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca4c4487-27679\" (UID: \"457e9435-2882-4e59-be1d-8bec75b53f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"
Oct 30 01:21:27.923190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.923137    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc25p\" (UniqueName: \"kubernetes.io/projected/457e9435-2882-4e59-be1d-8bec75b53f1e-kube-api-access-dc25p\") pod \"maas-api-revoke-sub-ca4c4487-27679\" (UID: \"457e9435-2882-4e59-be1d-8bec75b53f1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"
Oct 30 01:21:27.926010 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:27.925982    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"]
Oct 30 01:21:28.033184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:28.032417    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"
Oct 30 01:21:28.407185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:28.407143    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"]
Oct 30 01:21:28.410327 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:21:28.410297    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457e9435_2882_4e59_be1d_8bec75b53f1e.slice/crio-afed61e01dca864e6254b61c71270b9aa5675b8a02f8c0147d82d33979060304 WatchSource:0}: Error finding container afed61e01dca864e6254b61c71270b9aa5675b8a02f8c0147d82d33979060304: Status 404 returned error can't find the container with id afed61e01dca864e6254b61c71270b9aa5675b8a02f8c0147d82d33979060304
Oct 30 01:21:28.447468 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:28.447430    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679" event={"ID":"457e9435-2882-4e59-be1d-8bec75b53f1e","Type":"ContainerStarted","Data":"afed61e01dca864e6254b61c71270b9aa5675b8a02f8c0147d82d33979060304"}
Oct 30 01:21:29.456708 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:29.456669    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679" event={"ID":"457e9435-2882-4e59-be1d-8bec75b53f1e","Type":"ContainerStarted","Data":"3c602adc2609262976738c60536fa46aa4187507e8ef2f2c237c9702d657b667"}
Oct 30 01:21:29.494991 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:29.494783    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679" podStartSLOduration=2.4947459690000002 podStartE2EDuration="2.494745969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:29.492822055 +0000 UTC m=+2398.242477364" watchObservedRunningTime="2026-10-30 01:21:29.494745969 +0000 UTC m=+2398.244401280"
Oct 30 01:21:30.466851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:30.466808    4425 generic.go:356] "Generic (PLEG): container finished" podID="457e9435-2882-4e59-be1d-8bec75b53f1e" containerID="3c602adc2609262976738c60536fa46aa4187507e8ef2f2c237c9702d657b667" exitCode=0
Oct 30 01:21:30.467395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:30.466899    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679" event={"ID":"457e9435-2882-4e59-be1d-8bec75b53f1e","Type":"ContainerDied","Data":"3c602adc2609262976738c60536fa46aa4187507e8ef2f2c237c9702d657b667"}
Oct 30 01:21:31.656822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:31.656787    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"
Oct 30 01:21:31.767938 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:31.767910    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/457e9435-2882-4e59-be1d-8bec75b53f1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457e9435-2882-4e59-be1d-8bec75b53f1e-maas-api-openshift-ca\") pod \"457e9435-2882-4e59-be1d-8bec75b53f1e\" (UID: \"457e9435-2882-4e59-be1d-8bec75b53f1e\") "
Oct 30 01:21:31.768094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:31.768067    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457e9435-2882-4e59-be1d-8bec75b53f1e-kube-api-access-dc25p\" (UniqueName: \"kubernetes.io/projected/457e9435-2882-4e59-be1d-8bec75b53f1e-kube-api-access-dc25p\") pod \"457e9435-2882-4e59-be1d-8bec75b53f1e\" (UID: \"457e9435-2882-4e59-be1d-8bec75b53f1e\") "
Oct 30 01:21:31.768139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:31.768112    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/457e9435-2882-4e59-be1d-8bec75b53f1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457e9435-2882-4e59-be1d-8bec75b53f1e-maas-api-xks-ca\") pod \"457e9435-2882-4e59-be1d-8bec75b53f1e\" (UID: \"457e9435-2882-4e59-be1d-8bec75b53f1e\") "
Oct 30 01:21:31.768371 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:31.768350    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457e9435-2882-4e59-be1d-8bec75b53f1e-maas-api-openshift-ca" pod "457e9435-2882-4e59-be1d-8bec75b53f1e" (UID: "457e9435-2882-4e59-be1d-8bec75b53f1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:31.768687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:31.768666    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457e9435-2882-4e59-be1d-8bec75b53f1e-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:31.770264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:31.770238    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457e9435-2882-4e59-be1d-8bec75b53f1e-maas-api-xks-ca" pod "457e9435-2882-4e59-be1d-8bec75b53f1e" (UID: "457e9435-2882-4e59-be1d-8bec75b53f1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:31.770419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:31.770403    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457e9435-2882-4e59-be1d-8bec75b53f1e-kube-api-access-dc25p" pod "457e9435-2882-4e59-be1d-8bec75b53f1e" (UID: "457e9435-2882-4e59-be1d-8bec75b53f1e"). InnerVolumeSpecName "kube-api-access-dc25p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:31.869886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:31.869854    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc25p\" (UniqueName: \"kubernetes.io/projected/457e9435-2882-4e59-be1d-8bec75b53f1e-kube-api-access-dc25p\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:31.869886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:31.869884    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457e9435-2882-4e59-be1d-8bec75b53f1e-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:32.488212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:32.488181    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"
Oct 30 01:21:32.488212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:32.488206    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679" event={"ID":"457e9435-2882-4e59-be1d-8bec75b53f1e","Type":"ContainerDied","Data":"afed61e01dca864e6254b61c71270b9aa5675b8a02f8c0147d82d33979060304"}
Oct 30 01:21:32.488431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:32.488256    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afed61e01dca864e6254b61c71270b9aa5675b8a02f8c0147d82d33979060304"
Oct 30 01:21:33.394298 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:33.394267    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"
Oct 30 01:21:33.394774 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:33.394697    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"
Oct 30 01:21:34.343274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:34.343246    4425 scope.go:122] "RemoveContainer" containerID="9ea2a9d6aab015f4dfc678d3e3e35f0be399e9d74c5f7929a2e2d321a7bb94eb"
Oct 30 01:21:35.758938 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.758897    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"]
Oct 30 01:21:35.759540 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.759522    4425 cpu_manager.go:414] "containerMap: removing container" podUID="457e9435-2882-4e59-be1d-8bec75b53f1e" containerName="revoke-keys"
Oct 30 01:21:35.759540 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.759540    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457e9435-2882-4e59-be1d-8bec75b53f1e" containerName="revoke-keys"
Oct 30 01:21:35.759655 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.759649    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="457e9435-2882-4e59-be1d-8bec75b53f1e" containerName="revoke-keys"
Oct 30 01:21:35.765114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.765095    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"
Oct 30 01:21:35.788804 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.788770    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-2f820e\""
Oct 30 01:21:35.788984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.788916    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-2f820e-dockercfg-qt5lj\""
Oct 30 01:21:35.791193 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.789697    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"]
Oct 30 01:21:35.826142 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.826099    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e763d6f-32ac-4739-9e2d-feca894a8987-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r\" (UID: \"8e763d6f-32ac-4739-9e2d-feca894a8987\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"
Oct 30 01:21:35.826477 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.826454    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpn6f\" (UniqueName: \"kubernetes.io/projected/8e763d6f-32ac-4739-9e2d-feca894a8987-kube-api-access-fpn6f\") pod \"payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r\" (UID: \"8e763d6f-32ac-4739-9e2d-feca894a8987\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"
Oct 30 01:21:35.873203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.873139    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"]
Oct 30 01:21:35.882529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.882495    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"
Oct 30 01:21:35.901769 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.901625    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"]
Oct 30 01:21:35.927958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.927927    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e763d6f-32ac-4739-9e2d-feca894a8987-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r\" (UID: \"8e763d6f-32ac-4739-9e2d-feca894a8987\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"
Oct 30 01:21:35.928146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.927965    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpn6f\" (UniqueName: \"kubernetes.io/projected/8e763d6f-32ac-4739-9e2d-feca894a8987-kube-api-access-fpn6f\") pod \"payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r\" (UID: \"8e763d6f-32ac-4739-9e2d-feca894a8987\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"
Oct 30 01:21:35.928146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.928055    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be776653-8806-4154-96d5-87ce0af15d97-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8\" (UID: \"be776653-8806-4154-96d5-87ce0af15d97\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"
Oct 30 01:21:35.928146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.928085    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ww6j\" (UniqueName: \"kubernetes.io/projected/be776653-8806-4154-96d5-87ce0af15d97-kube-api-access-7ww6j\") pod \"payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8\" (UID: \"be776653-8806-4154-96d5-87ce0af15d97\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"
Oct 30 01:21:35.928875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.928849    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e763d6f-32ac-4739-9e2d-feca894a8987-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r\" (UID: \"8e763d6f-32ac-4739-9e2d-feca894a8987\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"
Oct 30 01:21:35.949464 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:35.949432    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpn6f\" (UniqueName: \"kubernetes.io/projected/8e763d6f-32ac-4739-9e2d-feca894a8987-kube-api-access-fpn6f\") pod \"payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r\" (UID: \"8e763d6f-32ac-4739-9e2d-feca894a8987\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"
Oct 30 01:21:36.029621 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:36.029581    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be776653-8806-4154-96d5-87ce0af15d97-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8\" (UID: \"be776653-8806-4154-96d5-87ce0af15d97\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"
Oct 30 01:21:36.029795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:36.029638    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ww6j\" (UniqueName: \"kubernetes.io/projected/be776653-8806-4154-96d5-87ce0af15d97-kube-api-access-7ww6j\") pod \"payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8\" (UID: \"be776653-8806-4154-96d5-87ce0af15d97\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"
Oct 30 01:21:36.030352 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:36.030332    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be776653-8806-4154-96d5-87ce0af15d97-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8\" (UID: \"be776653-8806-4154-96d5-87ce0af15d97\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"
Oct 30 01:21:36.055531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:36.055495    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ww6j\" (UniqueName: \"kubernetes.io/projected/be776653-8806-4154-96d5-87ce0af15d97-kube-api-access-7ww6j\") pod \"payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8\" (UID: \"be776653-8806-4154-96d5-87ce0af15d97\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"
Oct 30 01:21:36.080223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:36.080192    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"
Oct 30 01:21:36.216437 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:36.216398    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"
Oct 30 01:21:36.384770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:36.384746    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"]
Oct 30 01:21:36.386749 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:21:36.386726    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e763d6f_32ac_4739_9e2d_feca894a8987.slice/crio-2b99b1d7f64f7868fd0eee84929bfc7d9c352646dca06e16bd5e2e8f477b206f WatchSource:0}: Error finding container 2b99b1d7f64f7868fd0eee84929bfc7d9c352646dca06e16bd5e2e8f477b206f: Status 404 returned error can't find the container with id 2b99b1d7f64f7868fd0eee84929bfc7d9c352646dca06e16bd5e2e8f477b206f
Oct 30 01:21:36.521266 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:36.521139    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"]
Oct 30 01:21:36.522814 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:21:36.522782    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe776653_8806_4154_96d5_87ce0af15d97.slice/crio-de92ed495f235e82172a4c380be4ddd4e81fa16a0198e5a523b1616a2598599e WatchSource:0}: Error finding container de92ed495f235e82172a4c380be4ddd4e81fa16a0198e5a523b1616a2598599e: Status 404 returned error can't find the container with id de92ed495f235e82172a4c380be4ddd4e81fa16a0198e5a523b1616a2598599e
Oct 30 01:21:36.528214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:36.528181    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r" event={"ID":"8e763d6f-32ac-4739-9e2d-feca894a8987","Type":"ContainerStarted","Data":"964227f2dd072fd7eb6c8b5fa6ab20ffddfbcb9c23cc8e8a5ec71f5b01a3754b"}
Oct 30 01:21:36.528303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:36.528242    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r" event={"ID":"8e763d6f-32ac-4739-9e2d-feca894a8987","Type":"ContainerStarted","Data":"2b99b1d7f64f7868fd0eee84929bfc7d9c352646dca06e16bd5e2e8f477b206f"}
Oct 30 01:21:36.528356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:36.528299    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"
Oct 30 01:21:36.565596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:36.565546    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r" podStartSLOduration=1.565528174 podStartE2EDuration="1.565528174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:36.564783676 +0000 UTC m=+2405.314438986" watchObservedRunningTime="2026-10-30 01:21:36.565528174 +0000 UTC m=+2405.315183484"
Oct 30 01:21:37.320816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.320770    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"]
Oct 30 01:21:37.325959 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.325926    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"
Oct 30 01:21:37.346336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.346291    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"]
Oct 30 01:21:37.346728 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.346700    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgpz5\" (UniqueName: \"kubernetes.io/projected/0e087042-3c62-41a2-8f9e-3134d594906b-kube-api-access-pgpz5\") pod \"payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg\" (UID: \"0e087042-3c62-41a2-8f9e-3134d594906b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"
Oct 30 01:21:37.346884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.346860    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e087042-3c62-41a2-8f9e-3134d594906b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg\" (UID: \"0e087042-3c62-41a2-8f9e-3134d594906b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"
Oct 30 01:21:37.348359 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.348331    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a47b9a\""
Oct 30 01:21:37.348482 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.348430    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a47b9a-dockercfg-pfk2x\""
Oct 30 01:21:37.451829 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.448736    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgpz5\" (UniqueName: \"kubernetes.io/projected/0e087042-3c62-41a2-8f9e-3134d594906b-kube-api-access-pgpz5\") pod \"payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg\" (UID: \"0e087042-3c62-41a2-8f9e-3134d594906b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"
Oct 30 01:21:37.452200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.452150    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e087042-3c62-41a2-8f9e-3134d594906b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg\" (UID: \"0e087042-3c62-41a2-8f9e-3134d594906b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"
Oct 30 01:21:37.453591 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.453522    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e087042-3c62-41a2-8f9e-3134d594906b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg\" (UID: \"0e087042-3c62-41a2-8f9e-3134d594906b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"
Oct 30 01:21:37.472716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.472669    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgpz5\" (UniqueName: \"kubernetes.io/projected/0e087042-3c62-41a2-8f9e-3134d594906b-kube-api-access-pgpz5\") pod \"payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg\" (UID: \"0e087042-3c62-41a2-8f9e-3134d594906b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"
Oct 30 01:21:37.474360 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.474333    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"]
Oct 30 01:21:37.481284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.481255    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"]
Oct 30 01:21:37.481455 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.481351    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"
Oct 30 01:21:37.537790 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.537742    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8" event={"ID":"be776653-8806-4154-96d5-87ce0af15d97","Type":"ContainerStarted","Data":"399e0e35809f1fff95994362539fa2c16aa092d3c61bcf67c1bc060f80facb1b"}
Oct 30 01:21:37.538018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.537998    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"
Oct 30 01:21:37.538207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.538146    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8" event={"ID":"be776653-8806-4154-96d5-87ce0af15d97","Type":"ContainerStarted","Data":"de92ed495f235e82172a4c380be4ddd4e81fa16a0198e5a523b1616a2598599e"}
Oct 30 01:21:37.553397 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.553364    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf5v8\" (UniqueName: \"kubernetes.io/projected/1a04bb9d-f7a6-4333-af09-e078a2cab36e-kube-api-access-hf5v8\") pod \"payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2\" (UID: \"1a04bb9d-f7a6-4333-af09-e078a2cab36e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"
Oct 30 01:21:37.553570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.553505    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a04bb9d-f7a6-4333-af09-e078a2cab36e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2\" (UID: \"1a04bb9d-f7a6-4333-af09-e078a2cab36e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"
Oct 30 01:21:37.557018 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:21:37.556989    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:21:37.578082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.577908    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8" podStartSLOduration=2.577894562 podStartE2EDuration="2.577894562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:37.575617805 +0000 UTC m=+2406.325273116" watchObservedRunningTime="2026-10-30 01:21:37.577894562 +0000 UTC m=+2406.327549870"
Oct 30 01:21:37.643660 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.643625    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"
Oct 30 01:21:37.655402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.655364    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a04bb9d-f7a6-4333-af09-e078a2cab36e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2\" (UID: \"1a04bb9d-f7a6-4333-af09-e078a2cab36e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"
Oct 30 01:21:37.655715 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.655691    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf5v8\" (UniqueName: \"kubernetes.io/projected/1a04bb9d-f7a6-4333-af09-e078a2cab36e-kube-api-access-hf5v8\") pod \"payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2\" (UID: \"1a04bb9d-f7a6-4333-af09-e078a2cab36e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"
Oct 30 01:21:37.656398 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.656365    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a04bb9d-f7a6-4333-af09-e078a2cab36e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2\" (UID: \"1a04bb9d-f7a6-4333-af09-e078a2cab36e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"
Oct 30 01:21:37.677085 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.677053    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf5v8\" (UniqueName: \"kubernetes.io/projected/1a04bb9d-f7a6-4333-af09-e078a2cab36e-kube-api-access-hf5v8\") pod \"payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2\" (UID: \"1a04bb9d-f7a6-4333-af09-e078a2cab36e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"
Oct 30 01:21:37.804610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.804577    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"
Oct 30 01:21:37.961027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:37.960996    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"]
Oct 30 01:21:37.961526 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:21:37.961498    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e087042_3c62_41a2_8f9e_3134d594906b.slice/crio-ed12ed603ea9ffa83b3b2b2af3f77e523a8d6bce8383d24c47270cfc19104345 WatchSource:0}: Error finding container ed12ed603ea9ffa83b3b2b2af3f77e523a8d6bce8383d24c47270cfc19104345: Status 404 returned error can't find the container with id ed12ed603ea9ffa83b3b2b2af3f77e523a8d6bce8383d24c47270cfc19104345
Oct 30 01:21:38.127979 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:38.127709    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"]
Oct 30 01:21:38.546843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:38.546803    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg" event={"ID":"0e087042-3c62-41a2-8f9e-3134d594906b","Type":"ContainerStarted","Data":"497a2e97b3f5ac0e17f75c6766891b5bbff429d66e906fc29b8594fed7765047"}
Oct 30 01:21:38.547324 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:38.546874    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"
Oct 30 01:21:38.547324 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:38.546898    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg" event={"ID":"0e087042-3c62-41a2-8f9e-3134d594906b","Type":"ContainerStarted","Data":"ed12ed603ea9ffa83b3b2b2af3f77e523a8d6bce8383d24c47270cfc19104345"}
Oct 30 01:21:38.548637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:38.548608    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2" event={"ID":"1a04bb9d-f7a6-4333-af09-e078a2cab36e","Type":"ContainerStarted","Data":"f114039bdcddfe298519874b4810d2cd8165459981831655ad939987eb27dafa"}
Oct 30 01:21:38.548762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:38.548659    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2" event={"ID":"1a04bb9d-f7a6-4333-af09-e078a2cab36e","Type":"ContainerStarted","Data":"d0e0dc0c005e0d6d451ee5d5a78faed6490a0ccf792ead026296f853469f3dc8"}
Oct 30 01:21:38.548828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:38.548798    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"
Oct 30 01:21:38.584564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:38.584522    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg" podStartSLOduration=1.584509684 podStartE2EDuration="1.584509684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:38.582878389 +0000 UTC m=+2407.332533697" watchObservedRunningTime="2026-10-30 01:21:38.584509684 +0000 UTC m=+2407.334165052"
Oct 30 01:21:38.624675 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:38.624624    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2" podStartSLOduration=1.62460592 podStartE2EDuration="1.62460592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:38.620790497 +0000 UTC m=+2407.370445809" watchObservedRunningTime="2026-10-30 01:21:38.62460592 +0000 UTC m=+2407.374261275"
Oct 30 01:21:40.950497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:40.950462    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"]
Oct 30 01:21:42.954291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:42.954249    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r" podUID="0ea9e224-aceb-4b04-b85b-b8730d32fb39" containerName="main" containerID="cri-o://a981ab1b67efd20e4ffcb210f03e0553aaf02c8a597cbc1aee3f445d2425da4f" gracePeriod=45
Oct 30 01:21:43.282254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.282220    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:21:43.393179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.391114    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-dshm\") pod \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") "
Oct 30 01:21:43.393179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.391278    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ea9e224-aceb-4b04-b85b-b8730d32fb39-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea9e224-aceb-4b04-b85b-b8730d32fb39-tls-certs\") pod \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") "
Oct 30 01:21:43.393179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.391313    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea9e224-aceb-4b04-b85b-b8730d32fb39-kube-api-access-q6vpl\" (UniqueName: \"kubernetes.io/projected/0ea9e224-aceb-4b04-b85b-b8730d32fb39-kube-api-access-q6vpl\") pod \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") "
Oct 30 01:21:43.393179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.391345    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-home\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-home\") pod \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") "
Oct 30 01:21:43.393179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.391430    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-tmp-dir\") pod \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") "
Oct 30 01:21:43.393179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.391507    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-model-cache\") pod \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\" (UID: \"0ea9e224-aceb-4b04-b85b-b8730d32fb39\") "
Oct 30 01:21:43.393684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.393621    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-model-cache" pod "0ea9e224-aceb-4b04-b85b-b8730d32fb39" (UID: "0ea9e224-aceb-4b04-b85b-b8730d32fb39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:43.404218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.393852    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-home" pod "0ea9e224-aceb-4b04-b85b-b8730d32fb39" (UID: "0ea9e224-aceb-4b04-b85b-b8730d32fb39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:43.404218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.394046    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-tmp-dir" pod "0ea9e224-aceb-4b04-b85b-b8730d32fb39" (UID: "0ea9e224-aceb-4b04-b85b-b8730d32fb39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:43.408538 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.408474    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea9e224-aceb-4b04-b85b-b8730d32fb39-kube-api-access-q6vpl" pod "0ea9e224-aceb-4b04-b85b-b8730d32fb39" (UID: "0ea9e224-aceb-4b04-b85b-b8730d32fb39"). InnerVolumeSpecName "kube-api-access-q6vpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:43.430997 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.430928    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea9e224-aceb-4b04-b85b-b8730d32fb39-tls-certs" pod "0ea9e224-aceb-4b04-b85b-b8730d32fb39" (UID: "0ea9e224-aceb-4b04-b85b-b8730d32fb39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:43.433703 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.433571    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-dshm" pod "0ea9e224-aceb-4b04-b85b-b8730d32fb39" (UID: "0ea9e224-aceb-4b04-b85b-b8730d32fb39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:43.492961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.492923    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:43.492961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.492952    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea9e224-aceb-4b04-b85b-b8730d32fb39-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:43.493177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.492968    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6vpl\" (UniqueName: \"kubernetes.io/projected/0ea9e224-aceb-4b04-b85b-b8730d32fb39-kube-api-access-q6vpl\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:43.493177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.492980    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:43.493177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.492988    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:43.493177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.492995    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea9e224-aceb-4b04-b85b-b8730d32fb39-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:43.595641 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.595610    4425 generic.go:356] "Generic (PLEG): container finished" podID="0ea9e224-aceb-4b04-b85b-b8730d32fb39" containerID="a981ab1b67efd20e4ffcb210f03e0553aaf02c8a597cbc1aee3f445d2425da4f" exitCode=0
Oct 30 01:21:43.595802 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.595649    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"
Oct 30 01:21:43.595802 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.595692    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r" event={"ID":"0ea9e224-aceb-4b04-b85b-b8730d32fb39","Type":"ContainerDied","Data":"a981ab1b67efd20e4ffcb210f03e0553aaf02c8a597cbc1aee3f445d2425da4f"}
Oct 30 01:21:43.595802 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.595741    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r" event={"ID":"0ea9e224-aceb-4b04-b85b-b8730d32fb39","Type":"ContainerDied","Data":"865f93af1b0189bc8e17d4b3d43fa50982e714c2ba8fb85f75f0b38a38860f10"}
Oct 30 01:21:43.595802 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.595764    4425 scope.go:122] "RemoveContainer" containerID="a981ab1b67efd20e4ffcb210f03e0553aaf02c8a597cbc1aee3f445d2425da4f"
Oct 30 01:21:43.606805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.606776    4425 scope.go:122] "RemoveContainer" containerID="a981ab1b67efd20e4ffcb210f03e0553aaf02c8a597cbc1aee3f445d2425da4f"
Oct 30 01:21:43.607108 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:21:43.607089    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a981ab1b67efd20e4ffcb210f03e0553aaf02c8a597cbc1aee3f445d2425da4f\": container with ID starting with a981ab1b67efd20e4ffcb210f03e0553aaf02c8a597cbc1aee3f445d2425da4f not found: ID does not exist" containerID="a981ab1b67efd20e4ffcb210f03e0553aaf02c8a597cbc1aee3f445d2425da4f"
Oct 30 01:21:43.607214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.607115    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a981ab1b67efd20e4ffcb210f03e0553aaf02c8a597cbc1aee3f445d2425da4f"} err="failed to get container status \"a981ab1b67efd20e4ffcb210f03e0553aaf02c8a597cbc1aee3f445d2425da4f\": rpc error: code = NotFound desc = could not find container \"a981ab1b67efd20e4ffcb210f03e0553aaf02c8a597cbc1aee3f445d2425da4f\": container with ID starting with a981ab1b67efd20e4ffcb210f03e0553aaf02c8a597cbc1aee3f445d2425da4f not found: ID does not exist"
Oct 30 01:21:43.647073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.647038    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"]
Oct 30 01:21:43.650721 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:43.650695    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-35042a/auth-test-model-35042a-kserve-59bd7f6c69-6w72r"]
Oct 30 01:21:45.423468 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:45.423434    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ea9e224-aceb-4b04-b85b-b8730d32fb39" path="/var/lib/kubelet/pods/0ea9e224-aceb-4b04-b85b-b8730d32fb39/volumes"
Oct 30 01:21:46.840218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:46.840185    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"]
Oct 30 01:21:46.840663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:46.840422    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp" podUID="01641f6d-e95b-4aeb-88db-8b5154e79bfe" containerName="istio-proxy" containerID="cri-o://cda91935b4a1719fc24d08266d90b4a84eeac9b713e9908b8ca10607c00286a9" gracePeriod=30
Oct 30 01:21:47.541543 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:47.541508    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"
Oct 30 01:21:48.552747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.552716    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"
Oct 30 01:21:48.758172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.758129    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"]
Oct 30 01:21:48.758797 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.758781    4425 cpu_manager.go:414] "containerMap: removing container" podUID="0ea9e224-aceb-4b04-b85b-b8730d32fb39" containerName="main"
Oct 30 01:21:48.758850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.758799    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea9e224-aceb-4b04-b85b-b8730d32fb39" containerName="main"
Oct 30 01:21:48.758950 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.758940    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea9e224-aceb-4b04-b85b-b8730d32fb39" containerName="main"
Oct 30 01:21:48.764118 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.764097    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"
Oct 30 01:21:48.781311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.781275    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-637e08\""
Oct 30 01:21:48.793268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.793242    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"]
Oct 30 01:21:48.832264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.832179    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"]
Oct 30 01:21:48.836801 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.836778    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"
Oct 30 01:21:48.850651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.850630    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-637e08-dockercfg-b7fkz\""
Oct 30 01:21:48.858933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.858909    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"]
Oct 30 01:21:48.864745 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.864716    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgkzb\" (UniqueName: \"kubernetes.io/projected/64f44cf3-24b7-4aea-a48f-b9a395503795-kube-api-access-hgkzb\") pod \"payload-processing-e2e-autores-637e08-f695688c4-8x5bj\" (UID: \"64f44cf3-24b7-4aea-a48f-b9a395503795\") " pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"
Oct 30 01:21:48.864856 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.864834    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqrfg\" (UniqueName: \"kubernetes.io/projected/f3a20b3b-516e-4e9a-b2c3-ec9d6c826348-kube-api-access-qqrfg\") pod \"payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp\" (UID: \"f3a20b3b-516e-4e9a-b2c3-ec9d6c826348\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"
Oct 30 01:21:48.864936 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.864909    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f44cf3-24b7-4aea-a48f-b9a395503795-plugins-config-volume\") pod \"payload-processing-e2e-autores-637e08-f695688c4-8x5bj\" (UID: \"64f44cf3-24b7-4aea-a48f-b9a395503795\") " pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"
Oct 30 01:21:48.865084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.865064    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a20b3b-516e-4e9a-b2c3-ec9d6c826348-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp\" (UID: \"f3a20b3b-516e-4e9a-b2c3-ec9d6c826348\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"
Oct 30 01:21:48.966809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.966772    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a20b3b-516e-4e9a-b2c3-ec9d6c826348-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp\" (UID: \"f3a20b3b-516e-4e9a-b2c3-ec9d6c826348\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"
Oct 30 01:21:48.967013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.966898    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgkzb\" (UniqueName: \"kubernetes.io/projected/64f44cf3-24b7-4aea-a48f-b9a395503795-kube-api-access-hgkzb\") pod \"payload-processing-e2e-autores-637e08-f695688c4-8x5bj\" (UID: \"64f44cf3-24b7-4aea-a48f-b9a395503795\") " pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"
Oct 30 01:21:48.967013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.966943    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqrfg\" (UniqueName: \"kubernetes.io/projected/f3a20b3b-516e-4e9a-b2c3-ec9d6c826348-kube-api-access-qqrfg\") pod \"payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp\" (UID: \"f3a20b3b-516e-4e9a-b2c3-ec9d6c826348\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"
Oct 30 01:21:48.967013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.966974    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f44cf3-24b7-4aea-a48f-b9a395503795-plugins-config-volume\") pod \"payload-processing-e2e-autores-637e08-f695688c4-8x5bj\" (UID: \"64f44cf3-24b7-4aea-a48f-b9a395503795\") " pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"
Oct 30 01:21:48.967739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.967713    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a20b3b-516e-4e9a-b2c3-ec9d6c826348-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp\" (UID: \"f3a20b3b-516e-4e9a-b2c3-ec9d6c826348\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"
Oct 30 01:21:48.967739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.967731    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f44cf3-24b7-4aea-a48f-b9a395503795-plugins-config-volume\") pod \"payload-processing-e2e-autores-637e08-f695688c4-8x5bj\" (UID: \"64f44cf3-24b7-4aea-a48f-b9a395503795\") " pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"
Oct 30 01:21:48.988639 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.988605    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgkzb\" (UniqueName: \"kubernetes.io/projected/64f44cf3-24b7-4aea-a48f-b9a395503795-kube-api-access-hgkzb\") pod \"payload-processing-e2e-autores-637e08-f695688c4-8x5bj\" (UID: \"64f44cf3-24b7-4aea-a48f-b9a395503795\") " pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"
Oct 30 01:21:48.988639 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:48.988627    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqrfg\" (UniqueName: \"kubernetes.io/projected/f3a20b3b-516e-4e9a-b2c3-ec9d6c826348-kube-api-access-qqrfg\") pod \"payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp\" (UID: \"f3a20b3b-516e-4e9a-b2c3-ec9d6c826348\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"
Oct 30 01:21:49.082191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:49.079134    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"
Oct 30 01:21:49.150654 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:49.150630    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"
Oct 30 01:21:49.395742 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:49.395709    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"]
Oct 30 01:21:49.396467 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:21:49.396445    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f44cf3_24b7_4aea_a48f_b9a395503795.slice/crio-488548c90ff8242063fc62c2f74051c5a1728a733ac8c6caa5237405a0452370 WatchSource:0}: Error finding container 488548c90ff8242063fc62c2f74051c5a1728a733ac8c6caa5237405a0452370: Status 404 returned error can't find the container with id 488548c90ff8242063fc62c2f74051c5a1728a733ac8c6caa5237405a0452370
Oct 30 01:21:49.477535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:49.477512    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"]
Oct 30 01:21:49.478192 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:21:49.478139    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a20b3b_516e_4e9a_b2c3_ec9d6c826348.slice/crio-e7713de134b4a29bc7c365416f31e6a1c82322208deed98a38b76635d98a0148 WatchSource:0}: Error finding container e7713de134b4a29bc7c365416f31e6a1c82322208deed98a38b76635d98a0148: Status 404 returned error can't find the container with id e7713de134b4a29bc7c365416f31e6a1c82322208deed98a38b76635d98a0148
Oct 30 01:21:49.567045 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:49.567020    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"
Oct 30 01:21:49.567625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:49.567601    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"
Oct 30 01:21:49.645081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:49.644991    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj" event={"ID":"64f44cf3-24b7-4aea-a48f-b9a395503795","Type":"ContainerStarted","Data":"cbf2011b5aa21b7e24ed8da44491bfafbca219b61d3d9ce936a59c4f391af6c1"}
Oct 30 01:21:49.645081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:49.645048    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj" event={"ID":"64f44cf3-24b7-4aea-a48f-b9a395503795","Type":"ContainerStarted","Data":"488548c90ff8242063fc62c2f74051c5a1728a733ac8c6caa5237405a0452370"}
Oct 30 01:21:49.645336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:49.645282    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"
Oct 30 01:21:49.647216 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:49.647184    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp" event={"ID":"f3a20b3b-516e-4e9a-b2c3-ec9d6c826348","Type":"ContainerStarted","Data":"1ca22975cde6b9234dc347795aa804a09ed4e073c2ea3ebbd9781e32848c38df"}
Oct 30 01:21:49.647373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:49.647231    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp" event={"ID":"f3a20b3b-516e-4e9a-b2c3-ec9d6c826348","Type":"ContainerStarted","Data":"e7713de134b4a29bc7c365416f31e6a1c82322208deed98a38b76635d98a0148"}
Oct 30 01:21:49.647373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:49.647318    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"
Oct 30 01:21:49.681994 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:49.681921    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj" podStartSLOduration=1.6819090920000002 podStartE2EDuration="1.681909092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:49.6813935 +0000 UTC m=+2418.431048809" watchObservedRunningTime="2026-10-30 01:21:49.681909092 +0000 UTC m=+2418.431564398"
Oct 30 01:21:49.717818 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:49.717755    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp" podStartSLOduration=1.717741136 podStartE2EDuration="1.717741136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:49.716317106 +0000 UTC m=+2418.465972415" watchObservedRunningTime="2026-10-30 01:21:49.717741136 +0000 UTC m=+2418.467396444"
Oct 30 01:21:50.650473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:50.650442    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:21:52.132358 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.132334    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:21:52.210493 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.210456    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-workload-certs\") pod \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") "
Oct 30 01:21:52.210659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.210565    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-credential-socket\") pod \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") "
Oct 30 01:21:52.210659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.210636    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-podinfo\") pod \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") "
Oct 30 01:21:52.210659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.210659    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-envoy\") pod \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") "
Oct 30 01:21:52.210854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.210701    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-token\" (UniqueName: \"kubernetes.io/projected/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-token\") pod \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") "
Oct 30 01:21:52.210854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.210732    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-workload-socket\") pod \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") "
Oct 30 01:21:52.210854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.210739    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-workload-certs" pod "01641f6d-e95b-4aeb-88db-8b5154e79bfe" (UID: "01641f6d-e95b-4aeb-88db-8b5154e79bfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:52.211027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.210857    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istiod-ca-cert\") pod \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") "
Oct 30 01:21:52.211027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.210916    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01641f6d-e95b-4aeb-88db-8b5154e79bfe-kube-api-access-lxn52\" (UniqueName: \"kubernetes.io/projected/01641f6d-e95b-4aeb-88db-8b5154e79bfe-kube-api-access-lxn52\") pod \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") "
Oct 30 01:21:52.211027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.210938    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-credential-socket" pod "01641f6d-e95b-4aeb-88db-8b5154e79bfe" (UID: "01641f6d-e95b-4aeb-88db-8b5154e79bfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:52.211027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.210950    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-data\") pod \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\" (UID: \"01641f6d-e95b-4aeb-88db-8b5154e79bfe\") "
Oct 30 01:21:52.211027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.210987    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-workload-socket" pod "01641f6d-e95b-4aeb-88db-8b5154e79bfe" (UID: "01641f6d-e95b-4aeb-88db-8b5154e79bfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:52.211374 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.211322    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istiod-ca-cert" pod "01641f6d-e95b-4aeb-88db-8b5154e79bfe" (UID: "01641f6d-e95b-4aeb-88db-8b5154e79bfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:52.211594 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.211571    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.211700 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.211601    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.211700 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.211605    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-data" pod "01641f6d-e95b-4aeb-88db-8b5154e79bfe" (UID: "01641f6d-e95b-4aeb-88db-8b5154e79bfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:52.211700 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.211616    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.211700 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.211630    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.213057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.213033    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-podinfo" pod "01641f6d-e95b-4aeb-88db-8b5154e79bfe" (UID: "01641f6d-e95b-4aeb-88db-8b5154e79bfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:52.213450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.213420    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-token" pod "01641f6d-e95b-4aeb-88db-8b5154e79bfe" (UID: "01641f6d-e95b-4aeb-88db-8b5154e79bfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:52.213565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.213539    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-envoy" pod "01641f6d-e95b-4aeb-88db-8b5154e79bfe" (UID: "01641f6d-e95b-4aeb-88db-8b5154e79bfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:52.213974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.213905    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01641f6d-e95b-4aeb-88db-8b5154e79bfe-kube-api-access-lxn52" pod "01641f6d-e95b-4aeb-88db-8b5154e79bfe" (UID: "01641f6d-e95b-4aeb-88db-8b5154e79bfe"). InnerVolumeSpecName "kube-api-access-lxn52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:52.313334 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.313302    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxn52\" (UniqueName: \"kubernetes.io/projected/01641f6d-e95b-4aeb-88db-8b5154e79bfe-kube-api-access-lxn52\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.313334 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.313330    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.313334 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.313339    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.313334 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.313348    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.313690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.313358    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01641f6d-e95b-4aeb-88db-8b5154e79bfe-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.671330 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.671294    4425 generic.go:356] "Generic (PLEG): container finished" podID="01641f6d-e95b-4aeb-88db-8b5154e79bfe" containerID="cda91935b4a1719fc24d08266d90b4a84eeac9b713e9908b8ca10607c00286a9" exitCode=0
Oct 30 01:21:52.671541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.671369    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp" event={"ID":"01641f6d-e95b-4aeb-88db-8b5154e79bfe","Type":"ContainerDied","Data":"cda91935b4a1719fc24d08266d90b4a84eeac9b713e9908b8ca10607c00286a9"}
Oct 30 01:21:52.671541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.671419    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp" event={"ID":"01641f6d-e95b-4aeb-88db-8b5154e79bfe","Type":"ContainerDied","Data":"84159016c4af34cecf3be1885c5a14546b17ee1d53b67a86efda54c11e620e4a"}
Oct 30 01:21:52.671541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.671448    4425 scope.go:122] "RemoveContainer" containerID="cda91935b4a1719fc24d08266d90b4a84eeac9b713e9908b8ca10607c00286a9"
Oct 30 01:21:52.671714 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.671597    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"
Oct 30 01:21:52.684882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.684819    4425 scope.go:122] "RemoveContainer" containerID="cda91935b4a1719fc24d08266d90b4a84eeac9b713e9908b8ca10607c00286a9"
Oct 30 01:21:52.685105 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:21:52.685085    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda91935b4a1719fc24d08266d90b4a84eeac9b713e9908b8ca10607c00286a9\": container with ID starting with cda91935b4a1719fc24d08266d90b4a84eeac9b713e9908b8ca10607c00286a9 not found: ID does not exist" containerID="cda91935b4a1719fc24d08266d90b4a84eeac9b713e9908b8ca10607c00286a9"
Oct 30 01:21:52.685215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.685115    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda91935b4a1719fc24d08266d90b4a84eeac9b713e9908b8ca10607c00286a9"} err="failed to get container status \"cda91935b4a1719fc24d08266d90b4a84eeac9b713e9908b8ca10607c00286a9\": rpc error: code = NotFound desc = could not find container \"cda91935b4a1719fc24d08266d90b4a84eeac9b713e9908b8ca10607c00286a9\": container with ID starting with cda91935b4a1719fc24d08266d90b4a84eeac9b713e9908b8ca10607c00286a9 not found: ID does not exist"
Oct 30 01:21:52.721761 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.721727    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"]
Oct 30 01:21:52.731462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:52.731435    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d5b2b8-openshift-default-fcbcf95f9-2nlcp"]
Oct 30 01:21:53.424399 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:53.424370    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01641f6d-e95b-4aeb-88db-8b5154e79bfe" path="/var/lib/kubelet/pods/01641f6d-e95b-4aeb-88db-8b5154e79bfe/volumes"
Oct 30 01:21:55.978661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:55.978625    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"]
Oct 30 01:21:55.979210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:55.979123    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d" podUID="0f250df5-aab6-45c8-afa8-b911491be2e3" containerName="main" containerID="cri-o://4d6d7ffa777bfac0b10bd720582db2b71b94b2c86ee32625c6306bba0f2a47d1" gracePeriod=45
Oct 30 01:21:56.322984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.322959    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:21:56.362716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.362671    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f250df5-aab6-45c8-afa8-b911491be2e3-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f250df5-aab6-45c8-afa8-b911491be2e3-tls-certs\") pod \"0f250df5-aab6-45c8-afa8-b911491be2e3\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") "
Oct 30 01:21:56.362941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.362772    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-home\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-home\") pod \"0f250df5-aab6-45c8-afa8-b911491be2e3\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") "
Oct 30 01:21:56.362941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.362856    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-model-cache\") pod \"0f250df5-aab6-45c8-afa8-b911491be2e3\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") "
Oct 30 01:21:56.362941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.362929    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-dshm\") pod \"0f250df5-aab6-45c8-afa8-b911491be2e3\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") "
Oct 30 01:21:56.363097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.362992    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-tmp-dir\") pod \"0f250df5-aab6-45c8-afa8-b911491be2e3\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") "
Oct 30 01:21:56.363097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.363029    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f250df5-aab6-45c8-afa8-b911491be2e3-kube-api-access-gxx47\" (UniqueName: \"kubernetes.io/projected/0f250df5-aab6-45c8-afa8-b911491be2e3-kube-api-access-gxx47\") pod \"0f250df5-aab6-45c8-afa8-b911491be2e3\" (UID: \"0f250df5-aab6-45c8-afa8-b911491be2e3\") "
Oct 30 01:21:56.365748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.364203    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-model-cache" pod "0f250df5-aab6-45c8-afa8-b911491be2e3" (UID: "0f250df5-aab6-45c8-afa8-b911491be2e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:56.365748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.364781    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-home" pod "0f250df5-aab6-45c8-afa8-b911491be2e3" (UID: "0f250df5-aab6-45c8-afa8-b911491be2e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:56.365748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.364834    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-tmp-dir" pod "0f250df5-aab6-45c8-afa8-b911491be2e3" (UID: "0f250df5-aab6-45c8-afa8-b911491be2e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:56.371223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.368844    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f250df5-aab6-45c8-afa8-b911491be2e3-kube-api-access-gxx47" pod "0f250df5-aab6-45c8-afa8-b911491be2e3" (UID: "0f250df5-aab6-45c8-afa8-b911491be2e3"). InnerVolumeSpecName "kube-api-access-gxx47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:56.375599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.375573    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-dshm" pod "0f250df5-aab6-45c8-afa8-b911491be2e3" (UID: "0f250df5-aab6-45c8-afa8-b911491be2e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:56.375789 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.375769    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f250df5-aab6-45c8-afa8-b911491be2e3-tls-certs" pod "0f250df5-aab6-45c8-afa8-b911491be2e3" (UID: "0f250df5-aab6-45c8-afa8-b911491be2e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:56.464907 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.464868    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:56.464907 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.464902    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:56.465131 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.464916    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxx47\" (UniqueName: \"kubernetes.io/projected/0f250df5-aab6-45c8-afa8-b911491be2e3-kube-api-access-gxx47\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:56.465131 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.464930    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f250df5-aab6-45c8-afa8-b911491be2e3-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:56.465131 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.464942    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:56.465131 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.464952    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f250df5-aab6-45c8-afa8-b911491be2e3-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:21:56.700343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.700256    4425 generic.go:356] "Generic (PLEG): container finished" podID="0f250df5-aab6-45c8-afa8-b911491be2e3" containerID="4d6d7ffa777bfac0b10bd720582db2b71b94b2c86ee32625c6306bba0f2a47d1" exitCode=0
Oct 30 01:21:56.700343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.700299    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"
Oct 30 01:21:56.700576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.700341    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d" event={"ID":"0f250df5-aab6-45c8-afa8-b911491be2e3","Type":"ContainerDied","Data":"4d6d7ffa777bfac0b10bd720582db2b71b94b2c86ee32625c6306bba0f2a47d1"}
Oct 30 01:21:56.700576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.700387    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d" event={"ID":"0f250df5-aab6-45c8-afa8-b911491be2e3","Type":"ContainerDied","Data":"515e840a8335613b9e5f621da9628aeb7c70f55a0300af0c01da741075165bd1"}
Oct 30 01:21:56.700576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.700408    4425 scope.go:122] "RemoveContainer" containerID="4d6d7ffa777bfac0b10bd720582db2b71b94b2c86ee32625c6306bba0f2a47d1"
Oct 30 01:21:56.711514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.711498    4425 scope.go:122] "RemoveContainer" containerID="4d6d7ffa777bfac0b10bd720582db2b71b94b2c86ee32625c6306bba0f2a47d1"
Oct 30 01:21:56.711789 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:21:56.711768    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d6d7ffa777bfac0b10bd720582db2b71b94b2c86ee32625c6306bba0f2a47d1\": container with ID starting with 4d6d7ffa777bfac0b10bd720582db2b71b94b2c86ee32625c6306bba0f2a47d1 not found: ID does not exist" containerID="4d6d7ffa777bfac0b10bd720582db2b71b94b2c86ee32625c6306bba0f2a47d1"
Oct 30 01:21:56.711838 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.711800    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d6d7ffa777bfac0b10bd720582db2b71b94b2c86ee32625c6306bba0f2a47d1"} err="failed to get container status \"4d6d7ffa777bfac0b10bd720582db2b71b94b2c86ee32625c6306bba0f2a47d1\": rpc error: code = NotFound desc = could not find container \"4d6d7ffa777bfac0b10bd720582db2b71b94b2c86ee32625c6306bba0f2a47d1\": container with ID starting with 4d6d7ffa777bfac0b10bd720582db2b71b94b2c86ee32625c6306bba0f2a47d1 not found: ID does not exist"
Oct 30 01:21:56.745598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.745563    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"]
Oct 30 01:21:56.755000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:56.754973    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-df4db1/auth-test-model-df4db1-kserve-69bbc8dc77-t894d"]
Oct 30 01:21:57.423898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:21:57.423865    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f250df5-aab6-45c8-afa8-b911491be2e3" path="/var/lib/kubelet/pods/0f250df5-aab6-45c8-afa8-b911491be2e3/volumes"
Oct 30 01:22:00.657356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:00.657322    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"
Oct 30 01:22:00.657917 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:00.657406    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"
Oct 30 01:22:03.821103 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.821058    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"]
Oct 30 01:22:03.821694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.821677    4425 cpu_manager.go:414] "containerMap: removing container" podUID="0f250df5-aab6-45c8-afa8-b911491be2e3" containerName="main"
Oct 30 01:22:03.821694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.821692    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f250df5-aab6-45c8-afa8-b911491be2e3" containerName="main"
Oct 30 01:22:03.821841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.821714    4425 cpu_manager.go:414] "containerMap: removing container" podUID="01641f6d-e95b-4aeb-88db-8b5154e79bfe" containerName="istio-proxy"
Oct 30 01:22:03.821841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.821719    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01641f6d-e95b-4aeb-88db-8b5154e79bfe" containerName="istio-proxy"
Oct 30 01:22:03.821841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.821833    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f250df5-aab6-45c8-afa8-b911491be2e3" containerName="main"
Oct 30 01:22:03.822003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.821844    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="01641f6d-e95b-4aeb-88db-8b5154e79bfe" containerName="istio-proxy"
Oct 30 01:22:03.825844 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.825817    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"
Oct 30 01:22:03.846598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.846572    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"]
Oct 30 01:22:03.848146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.848124    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c7bba9\""
Oct 30 01:22:03.848306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.848265    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c7bba9-dockercfg-54sjh\""
Oct 30 01:22:03.876921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.876888    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"]
Oct 30 01:22:03.881581 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.881558    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"
Oct 30 01:22:03.905905 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.905872    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"]
Oct 30 01:22:03.959086 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.959052    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5mt2\" (UniqueName: \"kubernetes.io/projected/0aab37bd-b15d-4064-b67b-640c159e9b19-kube-api-access-t5mt2\") pod \"payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv\" (UID: \"0aab37bd-b15d-4064-b67b-640c159e9b19\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"
Oct 30 01:22:03.959246 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.959096    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73980e8-28ab-4caf-8be2-9455d2b4335f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx\" (UID: \"b73980e8-28ab-4caf-8be2-9455d2b4335f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"
Oct 30 01:22:03.959305 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.959264    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aab37bd-b15d-4064-b67b-640c159e9b19-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv\" (UID: \"0aab37bd-b15d-4064-b67b-640c159e9b19\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"
Oct 30 01:22:03.959305 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:03.959294    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnkcx\" (UniqueName: \"kubernetes.io/projected/b73980e8-28ab-4caf-8be2-9455d2b4335f-kube-api-access-xnkcx\") pod \"payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx\" (UID: \"b73980e8-28ab-4caf-8be2-9455d2b4335f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"
Oct 30 01:22:04.060981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.060945    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aab37bd-b15d-4064-b67b-640c159e9b19-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv\" (UID: \"0aab37bd-b15d-4064-b67b-640c159e9b19\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"
Oct 30 01:22:04.060981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.060987    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnkcx\" (UniqueName: \"kubernetes.io/projected/b73980e8-28ab-4caf-8be2-9455d2b4335f-kube-api-access-xnkcx\") pod \"payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx\" (UID: \"b73980e8-28ab-4caf-8be2-9455d2b4335f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"
Oct 30 01:22:04.061247 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.061200    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mt2\" (UniqueName: \"kubernetes.io/projected/0aab37bd-b15d-4064-b67b-640c159e9b19-kube-api-access-t5mt2\") pod \"payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv\" (UID: \"0aab37bd-b15d-4064-b67b-640c159e9b19\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"
Oct 30 01:22:04.061285 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.061248    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73980e8-28ab-4caf-8be2-9455d2b4335f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx\" (UID: \"b73980e8-28ab-4caf-8be2-9455d2b4335f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"
Oct 30 01:22:04.061792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.061771    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aab37bd-b15d-4064-b67b-640c159e9b19-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv\" (UID: \"0aab37bd-b15d-4064-b67b-640c159e9b19\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"
Oct 30 01:22:04.061961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.061941    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73980e8-28ab-4caf-8be2-9455d2b4335f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx\" (UID: \"b73980e8-28ab-4caf-8be2-9455d2b4335f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"
Oct 30 01:22:04.082626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.082549    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mt2\" (UniqueName: \"kubernetes.io/projected/0aab37bd-b15d-4064-b67b-640c159e9b19-kube-api-access-t5mt2\") pod \"payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv\" (UID: \"0aab37bd-b15d-4064-b67b-640c159e9b19\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"
Oct 30 01:22:04.082626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.082588    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnkcx\" (UniqueName: \"kubernetes.io/projected/b73980e8-28ab-4caf-8be2-9455d2b4335f-kube-api-access-xnkcx\") pod \"payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx\" (UID: \"b73980e8-28ab-4caf-8be2-9455d2b4335f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"
Oct 30 01:22:04.141410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.141375    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"
Oct 30 01:22:04.204735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.197097    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"
Oct 30 01:22:04.445534 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.445505    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"]
Oct 30 01:22:04.446894 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:22:04.446866    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aab37bd_b15d_4064_b67b_640c159e9b19.slice/crio-d5978ca1d6f5e82fba01cbf0019d567cf3ab9066587c99245b35f9446f0e9891 WatchSource:0}: Error finding container d5978ca1d6f5e82fba01cbf0019d567cf3ab9066587c99245b35f9446f0e9891: Status 404 returned error can't find the container with id d5978ca1d6f5e82fba01cbf0019d567cf3ab9066587c99245b35f9446f0e9891
Oct 30 01:22:04.536126 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.536097    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"]
Oct 30 01:22:04.536861 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:22:04.536834    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73980e8_28ab_4caf_8be2_9455d2b4335f.slice/crio-c863832f84a8918e4eabfcc500887ddc323153d2b0dc0f3d063d6c76a12286fa WatchSource:0}: Error finding container c863832f84a8918e4eabfcc500887ddc323153d2b0dc0f3d063d6c76a12286fa: Status 404 returned error can't find the container with id c863832f84a8918e4eabfcc500887ddc323153d2b0dc0f3d063d6c76a12286fa
Oct 30 01:22:04.761760 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.761714    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv" event={"ID":"0aab37bd-b15d-4064-b67b-640c159e9b19","Type":"ContainerStarted","Data":"b04f27da98b3ba48d91a9f179ef3e1856128bcee5ef942c055630e9a40259436"}
Oct 30 01:22:04.761941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.761790    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv" event={"ID":"0aab37bd-b15d-4064-b67b-640c159e9b19","Type":"ContainerStarted","Data":"d5978ca1d6f5e82fba01cbf0019d567cf3ab9066587c99245b35f9446f0e9891"}
Oct 30 01:22:04.761941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.761877    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"
Oct 30 01:22:04.763742 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.763719    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx" event={"ID":"b73980e8-28ab-4caf-8be2-9455d2b4335f","Type":"ContainerStarted","Data":"9d282886bbac855c685db09d3529ab51fe49d776a11b964f00193655aef185da"}
Oct 30 01:22:04.763845 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.763753    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx" event={"ID":"b73980e8-28ab-4caf-8be2-9455d2b4335f","Type":"ContainerStarted","Data":"c863832f84a8918e4eabfcc500887ddc323153d2b0dc0f3d063d6c76a12286fa"}
Oct 30 01:22:04.763901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.763853    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"
Oct 30 01:22:04.802833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.802782    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv" podStartSLOduration=1.802766286 podStartE2EDuration="1.802766286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:04.798520718 +0000 UTC m=+2433.548176028" watchObservedRunningTime="2026-10-30 01:22:04.802766286 +0000 UTC m=+2433.552421596"
Oct 30 01:22:04.836245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:04.836201    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx" podStartSLOduration=1.836187317 podStartE2EDuration="1.836187317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:04.833554248 +0000 UTC m=+2433.583209557" watchObservedRunningTime="2026-10-30 01:22:04.836187317 +0000 UTC m=+2433.585842625"
Oct 30 01:22:07.556111 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:22:07.556068    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:22:11.003826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.003768    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4" podUID="6706d150-ebe5-4bdd-ba5f-34126ade3970" containerName="main" containerID="cri-o://ab9a6c76edebcd389bef4a41d1dc27b8b5df73977780b720c34fe4c5f96b68ae" gracePeriod=45
Oct 30 01:22:11.284990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.284964    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:22:11.353967 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.353930    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-dshm\") pod \"6706d150-ebe5-4bdd-ba5f-34126ade3970\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") "
Oct 30 01:22:11.354177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.354032    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6706d150-ebe5-4bdd-ba5f-34126ade3970-kube-api-access-gbxc2\" (UniqueName: \"kubernetes.io/projected/6706d150-ebe5-4bdd-ba5f-34126ade3970-kube-api-access-gbxc2\") pod \"6706d150-ebe5-4bdd-ba5f-34126ade3970\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") "
Oct 30 01:22:11.354177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.354117    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-model-cache\") pod \"6706d150-ebe5-4bdd-ba5f-34126ade3970\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") "
Oct 30 01:22:11.354310 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.354225    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-home\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-home\") pod \"6706d150-ebe5-4bdd-ba5f-34126ade3970\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") "
Oct 30 01:22:11.354310 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.354288    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-tmp-dir\") pod \"6706d150-ebe5-4bdd-ba5f-34126ade3970\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") "
Oct 30 01:22:11.354418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.354345    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6706d150-ebe5-4bdd-ba5f-34126ade3970-tls-certs\" (UniqueName: \"kubernetes.io/secret/6706d150-ebe5-4bdd-ba5f-34126ade3970-tls-certs\") pod \"6706d150-ebe5-4bdd-ba5f-34126ade3970\" (UID: \"6706d150-ebe5-4bdd-ba5f-34126ade3970\") "
Oct 30 01:22:11.354606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.354582    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-home" pod "6706d150-ebe5-4bdd-ba5f-34126ade3970" (UID: "6706d150-ebe5-4bdd-ba5f-34126ade3970"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:11.356585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.354739    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.356585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.356574    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-model-cache" pod "6706d150-ebe5-4bdd-ba5f-34126ade3970" (UID: "6706d150-ebe5-4bdd-ba5f-34126ade3970"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:11.356785 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.356681    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-tmp-dir" pod "6706d150-ebe5-4bdd-ba5f-34126ade3970" (UID: "6706d150-ebe5-4bdd-ba5f-34126ade3970"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:11.357100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.357073    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-dshm" pod "6706d150-ebe5-4bdd-ba5f-34126ade3970" (UID: "6706d150-ebe5-4bdd-ba5f-34126ade3970"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:11.358524 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.358502    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6706d150-ebe5-4bdd-ba5f-34126ade3970-tls-certs" pod "6706d150-ebe5-4bdd-ba5f-34126ade3970" (UID: "6706d150-ebe5-4bdd-ba5f-34126ade3970"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:11.358905 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.358873    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6706d150-ebe5-4bdd-ba5f-34126ade3970-kube-api-access-gbxc2" pod "6706d150-ebe5-4bdd-ba5f-34126ade3970" (UID: "6706d150-ebe5-4bdd-ba5f-34126ade3970"). InnerVolumeSpecName "kube-api-access-gbxc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:11.456416 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.456379    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6706d150-ebe5-4bdd-ba5f-34126ade3970-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.456416 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.456405    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.456416 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.456415    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbxc2\" (UniqueName: \"kubernetes.io/projected/6706d150-ebe5-4bdd-ba5f-34126ade3970-kube-api-access-gbxc2\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.456743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.456428    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.456743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.456440    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6706d150-ebe5-4bdd-ba5f-34126ade3970-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.814852 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.814815    4425 generic.go:356] "Generic (PLEG): container finished" podID="6706d150-ebe5-4bdd-ba5f-34126ade3970" containerID="ab9a6c76edebcd389bef4a41d1dc27b8b5df73977780b720c34fe4c5f96b68ae" exitCode=0
Oct 30 01:22:11.815048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.814863    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"
Oct 30 01:22:11.815048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.814896    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4" event={"ID":"6706d150-ebe5-4bdd-ba5f-34126ade3970","Type":"ContainerDied","Data":"ab9a6c76edebcd389bef4a41d1dc27b8b5df73977780b720c34fe4c5f96b68ae"}
Oct 30 01:22:11.815048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.814941    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4" event={"ID":"6706d150-ebe5-4bdd-ba5f-34126ade3970","Type":"ContainerDied","Data":"ba9bb1e25bd4a38973a42f258c5546ca2f89a5e4245da3f7e881271b8cad00f8"}
Oct 30 01:22:11.815048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.814959    4425 scope.go:122] "RemoveContainer" containerID="ab9a6c76edebcd389bef4a41d1dc27b8b5df73977780b720c34fe4c5f96b68ae"
Oct 30 01:22:11.825057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.825040    4425 scope.go:122] "RemoveContainer" containerID="ab9a6c76edebcd389bef4a41d1dc27b8b5df73977780b720c34fe4c5f96b68ae"
Oct 30 01:22:11.825372 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:22:11.825354    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9a6c76edebcd389bef4a41d1dc27b8b5df73977780b720c34fe4c5f96b68ae\": container with ID starting with ab9a6c76edebcd389bef4a41d1dc27b8b5df73977780b720c34fe4c5f96b68ae not found: ID does not exist" containerID="ab9a6c76edebcd389bef4a41d1dc27b8b5df73977780b720c34fe4c5f96b68ae"
Oct 30 01:22:11.825432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.825379    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9a6c76edebcd389bef4a41d1dc27b8b5df73977780b720c34fe4c5f96b68ae"} err="failed to get container status \"ab9a6c76edebcd389bef4a41d1dc27b8b5df73977780b720c34fe4c5f96b68ae\": rpc error: code = NotFound desc = could not find container \"ab9a6c76edebcd389bef4a41d1dc27b8b5df73977780b720c34fe4c5f96b68ae\": container with ID starting with ab9a6c76edebcd389bef4a41d1dc27b8b5df73977780b720c34fe4c5f96b68ae not found: ID does not exist"
Oct 30 01:22:11.860927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.860896    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"]
Oct 30 01:22:11.870360 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:11.870331    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4"]
Oct 30 01:22:12.277586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:12.277549    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-bdf6ca/auth-test-model-bdf6ca-kserve-8977f76c7-hgnf4" podUID="6706d150-ebe5-4bdd-ba5f-34126ade3970" containerName="main" probeResult="failure" output="Get \"https://10.133.0.215:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:22:13.423813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:13.423782    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6706d150-ebe5-4bdd-ba5f-34126ade3970" path="/var/lib/kubelet/pods/6706d150-ebe5-4bdd-ba5f-34126ade3970/volumes"
Oct 30 01:22:15.774095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:15.774068    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"
Oct 30 01:22:15.774608 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:15.774148    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"
Oct 30 01:22:37.555081 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:22:37.555035    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:22:38.746128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.746092    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"]
Oct 30 01:22:38.746839 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.746819    4425 cpu_manager.go:414] "containerMap: removing container" podUID="6706d150-ebe5-4bdd-ba5f-34126ade3970" containerName="main"
Oct 30 01:22:38.746894 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.746840    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6706d150-ebe5-4bdd-ba5f-34126ade3970" containerName="main"
Oct 30 01:22:38.746980 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.746972    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6706d150-ebe5-4bdd-ba5f-34126ade3970" containerName="main"
Oct 30 01:22:38.752063 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.752044    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"
Oct 30 01:22:38.771311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.771290    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-dd80cb-dockercfg-879wp\""
Oct 30 01:22:38.771868 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.771852    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-dd80cb\""
Oct 30 01:22:38.776498 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.776470    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"]
Oct 30 01:22:38.849071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.849035    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"]
Oct 30 01:22:38.854167 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.854136    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"
Oct 30 01:22:38.877394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.877360    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwn8f\" (UniqueName: \"kubernetes.io/projected/a4625d33-6aa0-47dc-a9dc-cad1bad7cf94-kube-api-access-mwn8f\") pod \"payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz\" (UID: \"a4625d33-6aa0-47dc-a9dc-cad1bad7cf94\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"
Oct 30 01:22:38.877607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.877582    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4625d33-6aa0-47dc-a9dc-cad1bad7cf94-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz\" (UID: \"a4625d33-6aa0-47dc-a9dc-cad1bad7cf94\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"
Oct 30 01:22:38.879568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.879543    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"]
Oct 30 01:22:38.978400 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.978367    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4625d33-6aa0-47dc-a9dc-cad1bad7cf94-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz\" (UID: \"a4625d33-6aa0-47dc-a9dc-cad1bad7cf94\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"
Oct 30 01:22:38.978591 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.978418    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltc4f\" (UniqueName: \"kubernetes.io/projected/ef141a5d-f63e-44da-a7a4-4f8b2e2c874a-kube-api-access-ltc4f\") pod \"payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp\" (UID: \"ef141a5d-f63e-44da-a7a4-4f8b2e2c874a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"
Oct 30 01:22:38.978591 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.978449    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef141a5d-f63e-44da-a7a4-4f8b2e2c874a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp\" (UID: \"ef141a5d-f63e-44da-a7a4-4f8b2e2c874a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"
Oct 30 01:22:38.978591 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.978510    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwn8f\" (UniqueName: \"kubernetes.io/projected/a4625d33-6aa0-47dc-a9dc-cad1bad7cf94-kube-api-access-mwn8f\") pod \"payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz\" (UID: \"a4625d33-6aa0-47dc-a9dc-cad1bad7cf94\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"
Oct 30 01:22:38.979199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:38.979144    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4625d33-6aa0-47dc-a9dc-cad1bad7cf94-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz\" (UID: \"a4625d33-6aa0-47dc-a9dc-cad1bad7cf94\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"
Oct 30 01:22:39.000652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:39.000584    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwn8f\" (UniqueName: \"kubernetes.io/projected/a4625d33-6aa0-47dc-a9dc-cad1bad7cf94-kube-api-access-mwn8f\") pod \"payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz\" (UID: \"a4625d33-6aa0-47dc-a9dc-cad1bad7cf94\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"
Oct 30 01:22:39.066861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:39.066831    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"
Oct 30 01:22:39.080402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:39.080376    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltc4f\" (UniqueName: \"kubernetes.io/projected/ef141a5d-f63e-44da-a7a4-4f8b2e2c874a-kube-api-access-ltc4f\") pod \"payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp\" (UID: \"ef141a5d-f63e-44da-a7a4-4f8b2e2c874a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"
Oct 30 01:22:39.080519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:39.080502    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef141a5d-f63e-44da-a7a4-4f8b2e2c874a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp\" (UID: \"ef141a5d-f63e-44da-a7a4-4f8b2e2c874a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"
Oct 30 01:22:39.081188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:39.081137    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef141a5d-f63e-44da-a7a4-4f8b2e2c874a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp\" (UID: \"ef141a5d-f63e-44da-a7a4-4f8b2e2c874a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"
Oct 30 01:22:39.101727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:39.101683    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltc4f\" (UniqueName: \"kubernetes.io/projected/ef141a5d-f63e-44da-a7a4-4f8b2e2c874a-kube-api-access-ltc4f\") pod \"payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp\" (UID: \"ef141a5d-f63e-44da-a7a4-4f8b2e2c874a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"
Oct 30 01:22:39.182237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:39.181307    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"
Oct 30 01:22:39.384357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:39.384328    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"]
Oct 30 01:22:39.385434 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:22:39.385409    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4625d33_6aa0_47dc_a9dc_cad1bad7cf94.slice/crio-cad7d877d7bd968c0d81e0592ca6f6fccda92740302a5d5a3b06dab2caef8727 WatchSource:0}: Error finding container cad7d877d7bd968c0d81e0592ca6f6fccda92740302a5d5a3b06dab2caef8727: Status 404 returned error can't find the container with id cad7d877d7bd968c0d81e0592ca6f6fccda92740302a5d5a3b06dab2caef8727
Oct 30 01:22:39.530657 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:39.530626    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"]
Oct 30 01:22:39.532355 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:22:39.532325    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef141a5d_f63e_44da_a7a4_4f8b2e2c874a.slice/crio-368cfda66087c23d58179cf3595222cff001e6d0b45a9fd39b21b76ccce3ff57 WatchSource:0}: Error finding container 368cfda66087c23d58179cf3595222cff001e6d0b45a9fd39b21b76ccce3ff57: Status 404 returned error can't find the container with id 368cfda66087c23d58179cf3595222cff001e6d0b45a9fd39b21b76ccce3ff57
Oct 30 01:22:40.018081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:40.018044    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp" event={"ID":"ef141a5d-f63e-44da-a7a4-4f8b2e2c874a","Type":"ContainerStarted","Data":"ce116253e383798668390c16effdedc19d62fe7780f655d608976ed92f38a1e8"}
Oct 30 01:22:40.018600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:40.018116    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"
Oct 30 01:22:40.018600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:40.018143    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp" event={"ID":"ef141a5d-f63e-44da-a7a4-4f8b2e2c874a","Type":"ContainerStarted","Data":"368cfda66087c23d58179cf3595222cff001e6d0b45a9fd39b21b76ccce3ff57"}
Oct 30 01:22:40.019747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:40.019713    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz" event={"ID":"a4625d33-6aa0-47dc-a9dc-cad1bad7cf94","Type":"ContainerStarted","Data":"c7453a7af4dd021222d46d0260e0fd3dac834c650bb924fc0739f03c3d43fa46"}
Oct 30 01:22:40.019875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:40.019771    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz" event={"ID":"a4625d33-6aa0-47dc-a9dc-cad1bad7cf94","Type":"ContainerStarted","Data":"cad7d877d7bd968c0d81e0592ca6f6fccda92740302a5d5a3b06dab2caef8727"}
Oct 30 01:22:40.019875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:40.019801    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"
Oct 30 01:22:40.069779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:40.069730    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp" podStartSLOduration=2.069716407 podStartE2EDuration="2.069716407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:40.06888432 +0000 UTC m=+2468.818539632" watchObservedRunningTime="2026-10-30 01:22:40.069716407 +0000 UTC m=+2468.819371716"
Oct 30 01:22:40.110049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:40.109993    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz" podStartSLOduration=2.109974809 podStartE2EDuration="2.109974809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:40.105870488 +0000 UTC m=+2468.855525799" watchObservedRunningTime="2026-10-30 01:22:40.109974809 +0000 UTC m=+2468.859630121"
Oct 30 01:22:51.030312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:51.030234    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"
Oct 30 01:22:51.030312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:51.030301    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"
Oct 30 01:22:56.945818 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:56.945764    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:22:57.799888 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:57.799850    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"]
Oct 30 01:22:57.804976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:57.804956    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"
Oct 30 01:22:57.820092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:57.820059    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-1afb3e\""
Oct 30 01:22:57.830094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:57.830067    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-1afb3e-dockercfg-gttg8\""
Oct 30 01:22:57.836315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:57.836286    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"]
Oct 30 01:22:57.868325 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:57.868297    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"]
Oct 30 01:22:57.873384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:57.873356    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"
Oct 30 01:22:57.895296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:57.895269    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"]
Oct 30 01:22:57.901556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:57.901528    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s67jh\" (UniqueName: \"kubernetes.io/projected/534f39ab-7cfa-43b1-bd72-ddfa1e8f702d-kube-api-access-s67jh\") pod \"payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr\" (UID: \"534f39ab-7cfa-43b1-bd72-ddfa1e8f702d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"
Oct 30 01:22:57.901700 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:57.901567    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/534f39ab-7cfa-43b1-bd72-ddfa1e8f702d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr\" (UID: \"534f39ab-7cfa-43b1-bd72-ddfa1e8f702d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"
Oct 30 01:22:58.004408 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:58.004364    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv\" (UID: \"a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"
Oct 30 01:22:58.004953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:58.004608    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kswdr\" (UniqueName: \"kubernetes.io/projected/a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1-kube-api-access-kswdr\") pod \"payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv\" (UID: \"a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"
Oct 30 01:22:58.004953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:58.004899    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s67jh\" (UniqueName: \"kubernetes.io/projected/534f39ab-7cfa-43b1-bd72-ddfa1e8f702d-kube-api-access-s67jh\") pod \"payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr\" (UID: \"534f39ab-7cfa-43b1-bd72-ddfa1e8f702d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"
Oct 30 01:22:58.005086 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:58.004960    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/534f39ab-7cfa-43b1-bd72-ddfa1e8f702d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr\" (UID: \"534f39ab-7cfa-43b1-bd72-ddfa1e8f702d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"
Oct 30 01:22:58.006033 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:58.006006    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/534f39ab-7cfa-43b1-bd72-ddfa1e8f702d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr\" (UID: \"534f39ab-7cfa-43b1-bd72-ddfa1e8f702d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"
Oct 30 01:22:58.027355 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:58.027310    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s67jh\" (UniqueName: \"kubernetes.io/projected/534f39ab-7cfa-43b1-bd72-ddfa1e8f702d-kube-api-access-s67jh\") pod \"payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr\" (UID: \"534f39ab-7cfa-43b1-bd72-ddfa1e8f702d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"
Oct 30 01:22:58.106312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:58.106226    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv\" (UID: \"a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"
Oct 30 01:22:58.106480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:58.106337    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kswdr\" (UniqueName: \"kubernetes.io/projected/a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1-kube-api-access-kswdr\") pod \"payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv\" (UID: \"a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"
Oct 30 01:22:58.106992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:58.106966    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv\" (UID: \"a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"
Oct 30 01:22:58.120080 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:58.120058    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"
Oct 30 01:22:58.127713 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:58.127687    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kswdr\" (UniqueName: \"kubernetes.io/projected/a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1-kube-api-access-kswdr\") pod \"payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv\" (UID: \"a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"
Oct 30 01:22:58.193195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:58.192394    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"
Oct 30 01:22:58.426225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:58.426196    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"]
Oct 30 01:22:58.510642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:58.510610    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"]
Oct 30 01:22:58.511026 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:22:58.511004    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3800dfa_f3bb_4c6d_ab7f_24cef32d43d1.slice/crio-e2e7cba9c3b9a4080de96a963d0a908cd31a122d16109da5c4cecd661cbda3a2 WatchSource:0}: Error finding container e2e7cba9c3b9a4080de96a963d0a908cd31a122d16109da5c4cecd661cbda3a2: Status 404 returned error can't find the container with id e2e7cba9c3b9a4080de96a963d0a908cd31a122d16109da5c4cecd661cbda3a2
Oct 30 01:22:59.165642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.165601    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr" event={"ID":"534f39ab-7cfa-43b1-bd72-ddfa1e8f702d","Type":"ContainerStarted","Data":"46d0797f7b9535068c592eacfeae58c1f50af2ea61b90ed59205ca49965b459a"}
Oct 30 01:22:59.166208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.165677    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr" event={"ID":"534f39ab-7cfa-43b1-bd72-ddfa1e8f702d","Type":"ContainerStarted","Data":"0c8823fe78ba3f0f7428f836624195a3f952a516dc8c84744c07794545ef9640"}
Oct 30 01:22:59.166208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.165710    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"
Oct 30 01:22:59.167289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.167243    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv" event={"ID":"a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1","Type":"ContainerStarted","Data":"6404b1bb1d27408c32b30d3f43a9bdfe9bb9457f2e711f1a8a4372fd18ead5af"}
Oct 30 01:22:59.167289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.167287    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv" event={"ID":"a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1","Type":"ContainerStarted","Data":"e2e7cba9c3b9a4080de96a963d0a908cd31a122d16109da5c4cecd661cbda3a2"}
Oct 30 01:22:59.167411 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.167369    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"
Oct 30 01:22:59.211846 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.211797    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr" podStartSLOduration=2.211784507 podStartE2EDuration="2.211784507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:59.201446894 +0000 UTC m=+2487.951102204" watchObservedRunningTime="2026-10-30 01:22:59.211784507 +0000 UTC m=+2487.961439852"
Oct 30 01:22:59.250308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.250250    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv" podStartSLOduration=2.250231071 podStartE2EDuration="2.250231071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:59.244513537 +0000 UTC m=+2487.994168848" watchObservedRunningTime="2026-10-30 01:22:59.250231071 +0000 UTC m=+2487.999886380"
Oct 30 01:22:59.449290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.449188    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"]
Oct 30 01:22:59.453922 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.453900    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"
Oct 30 01:22:59.469164 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.469141    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-ae1820\""
Oct 30 01:22:59.469420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.469395    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-ae1820-dockercfg-bmwsb\""
Oct 30 01:22:59.473394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.473361    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"]
Oct 30 01:22:59.530817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.530783    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d78846e-a31f-4911-8c8f-4d0a93748883-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz\" (UID: \"0d78846e-a31f-4911-8c8f-4d0a93748883\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"
Oct 30 01:22:59.531017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.530830    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxhsc\" (UniqueName: \"kubernetes.io/projected/0d78846e-a31f-4911-8c8f-4d0a93748883-kube-api-access-hxhsc\") pod \"payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz\" (UID: \"0d78846e-a31f-4911-8c8f-4d0a93748883\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"
Oct 30 01:22:59.575559 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.574218    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"]
Oct 30 01:22:59.579807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.579782    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"
Oct 30 01:22:59.628599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.628559    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"]
Oct 30 01:22:59.632021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.631993    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d78846e-a31f-4911-8c8f-4d0a93748883-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz\" (UID: \"0d78846e-a31f-4911-8c8f-4d0a93748883\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"
Oct 30 01:22:59.632202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.632040    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxhsc\" (UniqueName: \"kubernetes.io/projected/0d78846e-a31f-4911-8c8f-4d0a93748883-kube-api-access-hxhsc\") pod \"payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz\" (UID: \"0d78846e-a31f-4911-8c8f-4d0a93748883\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"
Oct 30 01:22:59.632760 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.632742    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d78846e-a31f-4911-8c8f-4d0a93748883-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz\" (UID: \"0d78846e-a31f-4911-8c8f-4d0a93748883\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"
Oct 30 01:22:59.687681 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.687646    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxhsc\" (UniqueName: \"kubernetes.io/projected/0d78846e-a31f-4911-8c8f-4d0a93748883-kube-api-access-hxhsc\") pod \"payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz\" (UID: \"0d78846e-a31f-4911-8c8f-4d0a93748883\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"
Oct 30 01:22:59.733617 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.733544    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045aea57-0daf-4b8d-96b7-38073146d269-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg\" (UID: \"045aea57-0daf-4b8d-96b7-38073146d269\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"
Oct 30 01:22:59.733754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.733658    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g72s6\" (UniqueName: \"kubernetes.io/projected/045aea57-0daf-4b8d-96b7-38073146d269-kube-api-access-g72s6\") pod \"payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg\" (UID: \"045aea57-0daf-4b8d-96b7-38073146d269\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"
Oct 30 01:22:59.769122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.769094    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"
Oct 30 01:22:59.835631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.835593    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045aea57-0daf-4b8d-96b7-38073146d269-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg\" (UID: \"045aea57-0daf-4b8d-96b7-38073146d269\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"
Oct 30 01:22:59.835760 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.835735    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g72s6\" (UniqueName: \"kubernetes.io/projected/045aea57-0daf-4b8d-96b7-38073146d269-kube-api-access-g72s6\") pod \"payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg\" (UID: \"045aea57-0daf-4b8d-96b7-38073146d269\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"
Oct 30 01:22:59.836446 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.836426    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045aea57-0daf-4b8d-96b7-38073146d269-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg\" (UID: \"045aea57-0daf-4b8d-96b7-38073146d269\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"
Oct 30 01:22:59.857017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.856993    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g72s6\" (UniqueName: \"kubernetes.io/projected/045aea57-0daf-4b8d-96b7-38073146d269-kube-api-access-g72s6\") pod \"payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg\" (UID: \"045aea57-0daf-4b8d-96b7-38073146d269\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"
Oct 30 01:22:59.898867 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:22:59.898827    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"
Oct 30 01:23:00.080032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:00.080010    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"]
Oct 30 01:23:00.080248 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:23:00.080193    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d78846e_a31f_4911_8c8f_4d0a93748883.slice/crio-7e3a885fac8827b25fe449f49d96f6625afb2a8d918d563dacb0629091b1ee51 WatchSource:0}: Error finding container 7e3a885fac8827b25fe449f49d96f6625afb2a8d918d563dacb0629091b1ee51: Status 404 returned error can't find the container with id 7e3a885fac8827b25fe449f49d96f6625afb2a8d918d563dacb0629091b1ee51
Oct 30 01:23:00.177230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:00.177191    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz" event={"ID":"0d78846e-a31f-4911-8c8f-4d0a93748883","Type":"ContainerStarted","Data":"2c927f7a0cf34a53ae8717e0de68fea2f4a1c04a13c3ff46781a2a5353dcb880"}
Oct 30 01:23:00.177663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:00.177271    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz" event={"ID":"0d78846e-a31f-4911-8c8f-4d0a93748883","Type":"ContainerStarted","Data":"7e3a885fac8827b25fe449f49d96f6625afb2a8d918d563dacb0629091b1ee51"}
Oct 30 01:23:00.177663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:00.177476    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"
Oct 30 01:23:00.216556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:00.216528    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"]
Oct 30 01:23:00.217612 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:23:00.217584    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045aea57_0daf_4b8d_96b7_38073146d269.slice/crio-7445bd0cf6532f725a2c2d09da7c7012e0fb557f8e922bcbd9d1efa5b5d14ef0 WatchSource:0}: Error finding container 7445bd0cf6532f725a2c2d09da7c7012e0fb557f8e922bcbd9d1efa5b5d14ef0: Status 404 returned error can't find the container with id 7445bd0cf6532f725a2c2d09da7c7012e0fb557f8e922bcbd9d1efa5b5d14ef0
Oct 30 01:23:00.220575 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:00.220532    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz" podStartSLOduration=1.220520431 podStartE2EDuration="1.220520431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:00.215040812 +0000 UTC m=+2488.964696124" watchObservedRunningTime="2026-10-30 01:23:00.220520431 +0000 UTC m=+2488.970175731"
Oct 30 01:23:01.187735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:01.187699    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg" event={"ID":"045aea57-0daf-4b8d-96b7-38073146d269","Type":"ContainerStarted","Data":"e4fc60add2e63e88d9bef7b9db6cb1717987420d951212ef50d3c16483b2c3a7"}
Oct 30 01:23:01.188199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:01.187756    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg" event={"ID":"045aea57-0daf-4b8d-96b7-38073146d269","Type":"ContainerStarted","Data":"7445bd0cf6532f725a2c2d09da7c7012e0fb557f8e922bcbd9d1efa5b5d14ef0"}
Oct 30 01:23:01.188199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:01.187916    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"
Oct 30 01:23:01.230701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:01.230650    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg" podStartSLOduration=2.230637664 podStartE2EDuration="2.230637664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:01.226276529 +0000 UTC m=+2489.975931837" watchObservedRunningTime="2026-10-30 01:23:01.230637664 +0000 UTC m=+2489.980292972"
Oct 30 01:23:07.556070 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:23:07.556031    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:23:10.099385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.099351    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"]
Oct 30 01:23:10.105368 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.105342    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.128813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.128784    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"]
Oct 30 01:23:10.136481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.133531    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-637e08\"/\"kube-root-ca.crt\""
Oct 30 01:23:10.136481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.133608    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-637e08\"/\"openshift-service-ca.crt\""
Oct 30 01:23:10.136481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.134324    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-637e08\"/\"default-dockercfg-w55kd\""
Oct 30 01:23:10.143393 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.143370    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-637e08\"/\"autores-model-637e08-kserve-self-signed-certs\""
Oct 30 01:23:10.180966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.180942    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"
Oct 30 01:23:10.181113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.181009    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"
Oct 30 01:23:10.261527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.261488    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-home\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.261738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.261601    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-tls-certs\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.261738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.261698    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-model-cache\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.261884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.261816    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfqkl\" (UniqueName: \"kubernetes.io/projected/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-kube-api-access-zfqkl\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.261988 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.261965    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-tmp-dir\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.262048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.262020    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-dshm\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.366063 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.365953    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-home\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.366311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.366095    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-tls-certs\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.366311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.366133    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-model-cache\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.366311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.366221    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfqkl\" (UniqueName: \"kubernetes.io/projected/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-kube-api-access-zfqkl\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.366511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.366369    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-tmp-dir\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.366511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.366412    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-dshm\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.367663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.367626    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-home\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.379179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.374930    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-tmp-dir\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.379179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.375205    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-model-cache\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.379179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.378064    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-tls-certs\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.379179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.378398    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-dshm\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.396636 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.396604    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfqkl\" (UniqueName: \"kubernetes.io/projected/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-kube-api-access-zfqkl\") pod \"autores-model-637e08-kserve-5594b89b77-wcvbg\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") " pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.423519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.423488    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:10.741529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:10.741498    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"]
Oct 30 01:23:10.742191 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:23:10.742131    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c62899b_4f3e_4ff7_80d3_a3a637a9857b.slice/crio-50b4e11bfa992b3b7769fcb249868a876abf532f2b8a7fc9c9b0c40620d9cdb6 WatchSource:0}: Error finding container 50b4e11bfa992b3b7769fcb249868a876abf532f2b8a7fc9c9b0c40620d9cdb6: Status 404 returned error can't find the container with id 50b4e11bfa992b3b7769fcb249868a876abf532f2b8a7fc9c9b0c40620d9cdb6
Oct 30 01:23:11.190821 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:11.190790    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"
Oct 30 01:23:11.279134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:11.279100    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg" event={"ID":"0c62899b-4f3e-4ff7-80d3-a3a637a9857b","Type":"ContainerStarted","Data":"fa72e29293737987cbbf4a2e4f110b0cd5d2cd3afd2d38df510463662c5dd2b4"}
Oct 30 01:23:11.279309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:11.279146    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg" event={"ID":"0c62899b-4f3e-4ff7-80d3-a3a637a9857b","Type":"ContainerStarted","Data":"50b4e11bfa992b3b7769fcb249868a876abf532f2b8a7fc9c9b0c40620d9cdb6"}
Oct 30 01:23:11.321202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:11.321119    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg" podStartSLOduration=1.321104755 podStartE2EDuration="1.321104755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:11.316903942 +0000 UTC m=+2500.066559250" watchObservedRunningTime="2026-10-30 01:23:11.321104755 +0000 UTC m=+2500.070760064"
Oct 30 01:23:11.425658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:11.425626    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:12.198201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:12.198171    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"
Oct 30 01:23:20.424791 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:20.424758    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:20.437810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:20.437786    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:21.380260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:21.380226    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:23:26.347441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:26.347405    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"]
Oct 30 01:23:26.347954 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:26.347696    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp" podUID="8653a528-c49c-41e3-abbb-739353e81e66" containerName="istio-proxy" containerID="cri-o://4484e85f1819f19c2fca0c087e0f2b601ff27352a2667988ce51170c18edab6f" gracePeriod=30
Oct 30 01:23:30.310106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:30.310072    4425 patch_prober.go:29] interesting pod/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:23:30.310536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:30.310139    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp" podUID="8653a528-c49c-41e3-abbb-739353e81e66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:23:31.070729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.070698    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"]
Oct 30 01:23:31.075146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.075130    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"
Oct 30 01:23:31.087586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.087549    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-94ea1e\""
Oct 30 01:23:31.097497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.097466    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-94ea1e-dockercfg-tffwc\""
Oct 30 01:23:31.101857 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.101835    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"]
Oct 30 01:23:31.169639 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.169595    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"]
Oct 30 01:23:31.175622 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.175587    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"
Oct 30 01:23:31.200945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.200915    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"]
Oct 30 01:23:31.226222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.226187    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/817e5cef-adec-43f8-8d49-439e1379d1be-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r\" (UID: \"817e5cef-adec-43f8-8d49-439e1379d1be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"
Oct 30 01:23:31.226491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.226467    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp5v6\" (UniqueName: \"kubernetes.io/projected/817e5cef-adec-43f8-8d49-439e1379d1be-kube-api-access-zp5v6\") pod \"payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r\" (UID: \"817e5cef-adec-43f8-8d49-439e1379d1be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"
Oct 30 01:23:31.327685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.327609    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f488s\" (UniqueName: \"kubernetes.io/projected/9cab6c8f-afe1-4908-8aa8-e533837c609a-kube-api-access-f488s\") pod \"payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb\" (UID: \"9cab6c8f-afe1-4908-8aa8-e533837c609a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"
Oct 30 01:23:31.328106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.327728    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/817e5cef-adec-43f8-8d49-439e1379d1be-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r\" (UID: \"817e5cef-adec-43f8-8d49-439e1379d1be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"
Oct 30 01:23:31.328106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.327772    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cab6c8f-afe1-4908-8aa8-e533837c609a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb\" (UID: \"9cab6c8f-afe1-4908-8aa8-e533837c609a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"
Oct 30 01:23:31.328106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.327807    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp5v6\" (UniqueName: \"kubernetes.io/projected/817e5cef-adec-43f8-8d49-439e1379d1be-kube-api-access-zp5v6\") pod \"payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r\" (UID: \"817e5cef-adec-43f8-8d49-439e1379d1be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"
Oct 30 01:23:31.328528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.328501    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/817e5cef-adec-43f8-8d49-439e1379d1be-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r\" (UID: \"817e5cef-adec-43f8-8d49-439e1379d1be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"
Oct 30 01:23:31.348810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.348779    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp5v6\" (UniqueName: \"kubernetes.io/projected/817e5cef-adec-43f8-8d49-439e1379d1be-kube-api-access-zp5v6\") pod \"payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r\" (UID: \"817e5cef-adec-43f8-8d49-439e1379d1be\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"
Oct 30 01:23:31.390333 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.390307    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"
Oct 30 01:23:31.429089 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.429059    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f488s\" (UniqueName: \"kubernetes.io/projected/9cab6c8f-afe1-4908-8aa8-e533837c609a-kube-api-access-f488s\") pod \"payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb\" (UID: \"9cab6c8f-afe1-4908-8aa8-e533837c609a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"
Oct 30 01:23:31.429287 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.429185    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cab6c8f-afe1-4908-8aa8-e533837c609a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb\" (UID: \"9cab6c8f-afe1-4908-8aa8-e533837c609a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"
Oct 30 01:23:31.430071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.430049    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cab6c8f-afe1-4908-8aa8-e533837c609a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb\" (UID: \"9cab6c8f-afe1-4908-8aa8-e533837c609a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"
Oct 30 01:23:31.451908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.451880    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f488s\" (UniqueName: \"kubernetes.io/projected/9cab6c8f-afe1-4908-8aa8-e533837c609a-kube-api-access-f488s\") pod \"payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb\" (UID: \"9cab6c8f-afe1-4908-8aa8-e533837c609a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"
Oct 30 01:23:31.528257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.528225    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"
Oct 30 01:23:31.646335 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.646309    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 01:23:31.695859 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:23:31.695828    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817e5cef_adec_43f8_8d49_439e1379d1be.slice/crio-6f754b17d9bfafba4c72416a0487de1d979fbe9d01c8c7e5b82831f71aa44948 WatchSource:0}: Error finding container 6f754b17d9bfafba4c72416a0487de1d979fbe9d01c8c7e5b82831f71aa44948: Status 404 returned error can't find the container with id 6f754b17d9bfafba4c72416a0487de1d979fbe9d01c8c7e5b82831f71aa44948
Oct 30 01:23:31.696936 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.696911    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"]
Oct 30 01:23:31.732394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.732371    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-istio-envoy\") pod \"8653a528-c49c-41e3-abbb-739353e81e66\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") "
Oct 30 01:23:31.732499 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.732484    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8653a528-c49c-41e3-abbb-739353e81e66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8653a528-c49c-41e3-abbb-739353e81e66-istio-podinfo\") pod \"8653a528-c49c-41e3-abbb-739353e81e66\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") "
Oct 30 01:23:31.732554 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.732507    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8653a528-c49c-41e3-abbb-739353e81e66-istio-token\" (UniqueName: \"kubernetes.io/projected/8653a528-c49c-41e3-abbb-739353e81e66-istio-token\") pod \"8653a528-c49c-41e3-abbb-739353e81e66\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") "
Oct 30 01:23:31.732554 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.732533    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-workload-certs\") pod \"8653a528-c49c-41e3-abbb-739353e81e66\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") "
Oct 30 01:23:31.732554 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.732551    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-istio-data\") pod \"8653a528-c49c-41e3-abbb-739353e81e66\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") "
Oct 30 01:23:31.732711 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.732578    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-credential-socket\") pod \"8653a528-c49c-41e3-abbb-739353e81e66\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") "
Oct 30 01:23:31.732711 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.732623    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8653a528-c49c-41e3-abbb-739353e81e66-kube-api-access-wkrkt\" (UniqueName: \"kubernetes.io/projected/8653a528-c49c-41e3-abbb-739353e81e66-kube-api-access-wkrkt\") pod \"8653a528-c49c-41e3-abbb-739353e81e66\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") "
Oct 30 01:23:31.732711 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.732697    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8653a528-c49c-41e3-abbb-739353e81e66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8653a528-c49c-41e3-abbb-739353e81e66-istiod-ca-cert\") pod \"8653a528-c49c-41e3-abbb-739353e81e66\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") "
Oct 30 01:23:31.732881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.732833    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-workload-certs" pod "8653a528-c49c-41e3-abbb-739353e81e66" (UID: "8653a528-c49c-41e3-abbb-739353e81e66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:31.732881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.732857    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-workload-socket\") pod \"8653a528-c49c-41e3-abbb-739353e81e66\" (UID: \"8653a528-c49c-41e3-abbb-739353e81e66\") "
Oct 30 01:23:31.733036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.732992    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-credential-socket" pod "8653a528-c49c-41e3-abbb-739353e81e66" (UID: "8653a528-c49c-41e3-abbb-739353e81e66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:31.733270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.733245    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-istio-data" pod "8653a528-c49c-41e3-abbb-739353e81e66" (UID: "8653a528-c49c-41e3-abbb-739353e81e66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:31.733471 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.733446    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-workload-socket" pod "8653a528-c49c-41e3-abbb-739353e81e66" (UID: "8653a528-c49c-41e3-abbb-739353e81e66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:31.733758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.733547    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8653a528-c49c-41e3-abbb-739353e81e66-istiod-ca-cert" pod "8653a528-c49c-41e3-abbb-739353e81e66" (UID: "8653a528-c49c-41e3-abbb-739353e81e66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:31.733758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.733675    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.733758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.733695    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.733758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.733708    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.733758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.733721    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8653a528-c49c-41e3-abbb-739353e81e66-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.733758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.733734    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.735149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.735123    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-istio-envoy" pod "8653a528-c49c-41e3-abbb-739353e81e66" (UID: "8653a528-c49c-41e3-abbb-739353e81e66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:31.735149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.735135    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8653a528-c49c-41e3-abbb-739353e81e66-istio-token" pod "8653a528-c49c-41e3-abbb-739353e81e66" (UID: "8653a528-c49c-41e3-abbb-739353e81e66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:31.735149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.735136    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8653a528-c49c-41e3-abbb-739353e81e66-istio-podinfo" pod "8653a528-c49c-41e3-abbb-739353e81e66" (UID: "8653a528-c49c-41e3-abbb-739353e81e66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:31.735442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.735426    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8653a528-c49c-41e3-abbb-739353e81e66-kube-api-access-wkrkt" pod "8653a528-c49c-41e3-abbb-739353e81e66" (UID: "8653a528-c49c-41e3-abbb-739353e81e66"). InnerVolumeSpecName "kube-api-access-wkrkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:31.835470 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.835444    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8653a528-c49c-41e3-abbb-739353e81e66-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.835470 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.835471    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8653a528-c49c-41e3-abbb-739353e81e66-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.835629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.835484    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkrkt\" (UniqueName: \"kubernetes.io/projected/8653a528-c49c-41e3-abbb-739353e81e66-kube-api-access-wkrkt\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.835629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.835498    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8653a528-c49c-41e3-abbb-739353e81e66-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.856934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:31.856907    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"]
Oct 30 01:23:31.857773 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:23:31.857716    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cab6c8f_afe1_4908_8aa8_e533837c609a.slice/crio-b8596a64a89f60d6dfaad46e5623410a2d8a92b855d97548daddb71c2622081e WatchSource:0}: Error finding container b8596a64a89f60d6dfaad46e5623410a2d8a92b855d97548daddb71c2622081e: Status 404 returned error can't find the container with id b8596a64a89f60d6dfaad46e5623410a2d8a92b855d97548daddb71c2622081e
Oct 30 01:23:32.447529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:32.447489    4425 generic.go:356] "Generic (PLEG): container finished" podID="8653a528-c49c-41e3-abbb-739353e81e66" containerID="4484e85f1819f19c2fca0c087e0f2b601ff27352a2667988ce51170c18edab6f" exitCode=0
Oct 30 01:23:32.448003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:32.447530    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"
Oct 30 01:23:32.448003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:32.447547    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp" event={"ID":"8653a528-c49c-41e3-abbb-739353e81e66","Type":"ContainerDied","Data":"4484e85f1819f19c2fca0c087e0f2b601ff27352a2667988ce51170c18edab6f"}
Oct 30 01:23:32.448003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:32.447596    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp" event={"ID":"8653a528-c49c-41e3-abbb-739353e81e66","Type":"ContainerDied","Data":"0bf8cc9ff78c68892db3f69e0224ea50b19bbde3005db5ead12ef4285dffa2ed"}
Oct 30 01:23:32.448003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:32.447629    4425 scope.go:122] "RemoveContainer" containerID="4484e85f1819f19c2fca0c087e0f2b601ff27352a2667988ce51170c18edab6f"
Oct 30 01:23:32.449927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:32.449671    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb" event={"ID":"9cab6c8f-afe1-4908-8aa8-e533837c609a","Type":"ContainerStarted","Data":"af25bf99e691dc67db0ddaab693b76cc2e98f3ebe0c9e2566dbbce9f600a976c"}
Oct 30 01:23:32.450242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:32.450223    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"
Oct 30 01:23:32.450343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:32.450262    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb" event={"ID":"9cab6c8f-afe1-4908-8aa8-e533837c609a","Type":"ContainerStarted","Data":"b8596a64a89f60d6dfaad46e5623410a2d8a92b855d97548daddb71c2622081e"}
Oct 30 01:23:32.457817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:32.451969    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r" event={"ID":"817e5cef-adec-43f8-8d49-439e1379d1be","Type":"ContainerStarted","Data":"3a9db16678e97e6a6ea17db2d4149cdb59021401bd2c465958c71cad8e956e36"}
Oct 30 01:23:32.457817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:32.452015    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r" event={"ID":"817e5cef-adec-43f8-8d49-439e1379d1be","Type":"ContainerStarted","Data":"6f754b17d9bfafba4c72416a0487de1d979fbe9d01c8c7e5b82831f71aa44948"}
Oct 30 01:23:32.457817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:32.452055    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"
Oct 30 01:23:32.460920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:32.460899    4425 scope.go:122] "RemoveContainer" containerID="4484e85f1819f19c2fca0c087e0f2b601ff27352a2667988ce51170c18edab6f"
Oct 30 01:23:32.461190 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:23:32.461166    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4484e85f1819f19c2fca0c087e0f2b601ff27352a2667988ce51170c18edab6f\": container with ID starting with 4484e85f1819f19c2fca0c087e0f2b601ff27352a2667988ce51170c18edab6f not found: ID does not exist" containerID="4484e85f1819f19c2fca0c087e0f2b601ff27352a2667988ce51170c18edab6f"
Oct 30 01:23:32.461263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:32.461204    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4484e85f1819f19c2fca0c087e0f2b601ff27352a2667988ce51170c18edab6f"} err="failed to get container status \"4484e85f1819f19c2fca0c087e0f2b601ff27352a2667988ce51170c18edab6f\": rpc error: code = NotFound desc = could not find container \"4484e85f1819f19c2fca0c087e0f2b601ff27352a2667988ce51170c18edab6f\": container with ID starting with 4484e85f1819f19c2fca0c087e0f2b601ff27352a2667988ce51170c18edab6f not found: ID does not exist"
Oct 30 01:23:32.496642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:32.496600    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r" podStartSLOduration=1.4965897 podStartE2EDuration="1.4965897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:32.495648184 +0000 UTC m=+2521.245303506" watchObservedRunningTime="2026-10-30 01:23:32.4965897 +0000 UTC m=+2521.246245205"
Oct 30 01:23:32.533474 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:32.533431    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb" podStartSLOduration=1.533419997 podStartE2EDuration="1.533419997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:32.531264189 +0000 UTC m=+2521.280919497" watchObservedRunningTime="2026-10-30 01:23:32.533419997 +0000 UTC m=+2521.283075305"
Oct 30 01:23:32.565421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:32.565397    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"]
Oct 30 01:23:32.573387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:32.573368    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-39b241-openshift-default-6bd9c8d777-lwjpp"]
Oct 30 01:23:33.425536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:33.425500    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8653a528-c49c-41e3-abbb-739353e81e66" path="/var/lib/kubelet/pods/8653a528-c49c-41e3-abbb-739353e81e66/volumes"
Oct 30 01:23:34.430332 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:34.430297    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:23:34.433805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:34.433778    4425 scope.go:122] "RemoveContainer" containerID="d6baa504d8cb4c23fffde8daeeba42f4a5e32c54194e251be4e090411a65da08"
Oct 30 01:23:37.555514 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:23:37.555480    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:23:43.463433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:43.463407    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"
Oct 30 01:23:43.463911 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:43.463473    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"
Oct 30 01:23:46.626732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:46.626696    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"]
Oct 30 01:23:46.998512 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:46.998428    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"]
Oct 30 01:23:46.999546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:46.999523    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8653a528-c49c-41e3-abbb-739353e81e66" containerName="istio-proxy"
Oct 30 01:23:46.999546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:46.999543    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8653a528-c49c-41e3-abbb-739353e81e66" containerName="istio-proxy"
Oct 30 01:23:46.999708 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:46.999678    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8653a528-c49c-41e3-abbb-739353e81e66" containerName="istio-proxy"
Oct 30 01:23:47.005727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:47.005708    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"
Oct 30 01:23:47.026266 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:47.026242    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"]
Oct 30 01:23:47.030816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:47.030790    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:23:47.030931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:47.030831    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:23:47.030931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:47.030833    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:23:47.030931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:47.030923    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:23:47.117781 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:47.117746    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4fcg\" (UniqueName: \"kubernetes.io/projected/d9647e5c-0609-4ef6-97d0-476df25a3bc3-kube-api-access-m4fcg\") pod \"maas-api-revoke-keys-e2e-autores-637e08-5b4tl\" (UID: \"d9647e5c-0609-4ef6-97d0-476df25a3bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"
Oct 30 01:23:47.117972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:47.117851    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9647e5c-0609-4ef6-97d0-476df25a3bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-637e08-5b4tl\" (UID: \"d9647e5c-0609-4ef6-97d0-476df25a3bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"
Oct 30 01:23:47.118043 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:47.118030    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9647e5c-0609-4ef6-97d0-476df25a3bc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-637e08-5b4tl\" (UID: \"d9647e5c-0609-4ef6-97d0-476df25a3bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"
Oct 30 01:23:47.219401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:47.219360    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9647e5c-0609-4ef6-97d0-476df25a3bc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-637e08-5b4tl\" (UID: \"d9647e5c-0609-4ef6-97d0-476df25a3bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"
Oct 30 01:23:47.219631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:47.219444    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4fcg\" (UniqueName: \"kubernetes.io/projected/d9647e5c-0609-4ef6-97d0-476df25a3bc3-kube-api-access-m4fcg\") pod \"maas-api-revoke-keys-e2e-autores-637e08-5b4tl\" (UID: \"d9647e5c-0609-4ef6-97d0-476df25a3bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"
Oct 30 01:23:47.219631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:47.219516    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9647e5c-0609-4ef6-97d0-476df25a3bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-637e08-5b4tl\" (UID: \"d9647e5c-0609-4ef6-97d0-476df25a3bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"
Oct 30 01:23:47.220436 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:47.220408    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9647e5c-0609-4ef6-97d0-476df25a3bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-637e08-5b4tl\" (UID: \"d9647e5c-0609-4ef6-97d0-476df25a3bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"
Oct 30 01:23:47.221932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:47.221910    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9647e5c-0609-4ef6-97d0-476df25a3bc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-637e08-5b4tl\" (UID: \"d9647e5c-0609-4ef6-97d0-476df25a3bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"
Oct 30 01:23:47.240685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:47.240650    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4fcg\" (UniqueName: \"kubernetes.io/projected/d9647e5c-0609-4ef6-97d0-476df25a3bc3-kube-api-access-m4fcg\") pod \"maas-api-revoke-keys-e2e-autores-637e08-5b4tl\" (UID: \"d9647e5c-0609-4ef6-97d0-476df25a3bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"
Oct 30 01:23:47.321264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:47.321218    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"
Oct 30 01:23:47.631256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:47.631233    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"]
Oct 30 01:23:47.632920 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:23:47.632892    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9647e5c_0609_4ef6_97d0_476df25a3bc3.slice/crio-6b37a573d384a06e8bc4d758a4a41287f2168cd447d07fdb8f198ee6971fa3aa WatchSource:0}: Error finding container 6b37a573d384a06e8bc4d758a4a41287f2168cd447d07fdb8f198ee6971fa3aa: Status 404 returned error can't find the container with id 6b37a573d384a06e8bc4d758a4a41287f2168cd447d07fdb8f198ee6971fa3aa
Oct 30 01:23:48.570754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:48.570706    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl" event={"ID":"d9647e5c-0609-4ef6-97d0-476df25a3bc3","Type":"ContainerStarted","Data":"78ef4e410587a000a0c46cd5d192c61181bb8e3da112e49f098803613a9fa1f6"}
Oct 30 01:23:48.570950 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:48.570767    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl" event={"ID":"d9647e5c-0609-4ef6-97d0-476df25a3bc3","Type":"ContainerStarted","Data":"6b37a573d384a06e8bc4d758a4a41287f2168cd447d07fdb8f198ee6971fa3aa"}
Oct 30 01:23:48.608768 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:48.608716    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl" podStartSLOduration=2.6086997849999998 podStartE2EDuration="2.608699785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:48.606064289 +0000 UTC m=+2537.355719624" watchObservedRunningTime="2026-10-30 01:23:48.608699785 +0000 UTC m=+2537.358355096"
Oct 30 01:23:49.579127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:49.579096    4425 generic.go:356] "Generic (PLEG): container finished" podID="d9647e5c-0609-4ef6-97d0-476df25a3bc3" containerID="78ef4e410587a000a0c46cd5d192c61181bb8e3da112e49f098803613a9fa1f6" exitCode=0
Oct 30 01:23:49.579605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:49.579184    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl" event={"ID":"d9647e5c-0609-4ef6-97d0-476df25a3bc3","Type":"ContainerDied","Data":"78ef4e410587a000a0c46cd5d192c61181bb8e3da112e49f098803613a9fa1f6"}
Oct 30 01:23:50.741553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:50.741527    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"
Oct 30 01:23:50.870860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:50.870725    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9647e5c-0609-4ef6-97d0-476df25a3bc3-kube-api-access-m4fcg\" (UniqueName: \"kubernetes.io/projected/d9647e5c-0609-4ef6-97d0-476df25a3bc3-kube-api-access-m4fcg\") pod \"d9647e5c-0609-4ef6-97d0-476df25a3bc3\" (UID: \"d9647e5c-0609-4ef6-97d0-476df25a3bc3\") "
Oct 30 01:23:50.870860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:50.870771    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9647e5c-0609-4ef6-97d0-476df25a3bc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9647e5c-0609-4ef6-97d0-476df25a3bc3-maas-api-openshift-ca\") pod \"d9647e5c-0609-4ef6-97d0-476df25a3bc3\" (UID: \"d9647e5c-0609-4ef6-97d0-476df25a3bc3\") "
Oct 30 01:23:50.870860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:50.870830    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9647e5c-0609-4ef6-97d0-476df25a3bc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9647e5c-0609-4ef6-97d0-476df25a3bc3-maas-api-xks-ca\") pod \"d9647e5c-0609-4ef6-97d0-476df25a3bc3\" (UID: \"d9647e5c-0609-4ef6-97d0-476df25a3bc3\") "
Oct 30 01:23:50.871257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:50.871236    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9647e5c-0609-4ef6-97d0-476df25a3bc3-maas-api-openshift-ca" pod "d9647e5c-0609-4ef6-97d0-476df25a3bc3" (UID: "d9647e5c-0609-4ef6-97d0-476df25a3bc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:50.871495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:50.871465    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9647e5c-0609-4ef6-97d0-476df25a3bc3-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:23:50.873045 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:50.873016    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9647e5c-0609-4ef6-97d0-476df25a3bc3-kube-api-access-m4fcg" pod "d9647e5c-0609-4ef6-97d0-476df25a3bc3" (UID: "d9647e5c-0609-4ef6-97d0-476df25a3bc3"). InnerVolumeSpecName "kube-api-access-m4fcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:50.873218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:50.873199    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9647e5c-0609-4ef6-97d0-476df25a3bc3-maas-api-xks-ca" pod "d9647e5c-0609-4ef6-97d0-476df25a3bc3" (UID: "d9647e5c-0609-4ef6-97d0-476df25a3bc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:50.972800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:50.972777    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4fcg\" (UniqueName: \"kubernetes.io/projected/d9647e5c-0609-4ef6-97d0-476df25a3bc3-kube-api-access-m4fcg\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:23:50.972800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:50.972799    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9647e5c-0609-4ef6-97d0-476df25a3bc3-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:23:51.594576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:51.594552    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"
Oct 30 01:23:51.594729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:51.594553    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl" event={"ID":"d9647e5c-0609-4ef6-97d0-476df25a3bc3","Type":"ContainerDied","Data":"6b37a573d384a06e8bc4d758a4a41287f2168cd447d07fdb8f198ee6971fa3aa"}
Oct 30 01:23:51.594729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:51.594674    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b37a573d384a06e8bc4d758a4a41287f2168cd447d07fdb8f198ee6971fa3aa"
Oct 30 01:23:57.690483 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:57.690449    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"]
Oct 30 01:23:57.719275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:23:57.719246    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"]
Oct 30 01:24:01.649351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:01.649306    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg" podUID="0c62899b-4f3e-4ff7-80d3-a3a637a9857b" containerName="main" containerID="cri-o://fa72e29293737987cbbf4a2e4f110b0cd5d2cd3afd2d38df510463662c5dd2b4" gracePeriod=45
Oct 30 01:24:01.927178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:01.927127    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:24:02.104860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.104828    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-dshm\") pod \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") "
Oct 30 01:24:02.105037 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.104923    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-model-cache\") pod \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") "
Oct 30 01:24:02.105037 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.104960    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-tmp-dir\") pod \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") "
Oct 30 01:24:02.105134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.105052    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-tls-certs\") pod \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") "
Oct 30 01:24:02.105134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.105121    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-home\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-home\") pod \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") "
Oct 30 01:24:02.105313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.105151    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-kube-api-access-zfqkl\" (UniqueName: \"kubernetes.io/projected/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-kube-api-access-zfqkl\") pod \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\" (UID: \"0c62899b-4f3e-4ff7-80d3-a3a637a9857b\") "
Oct 30 01:24:02.105313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.105223    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-model-cache" pod "0c62899b-4f3e-4ff7-80d3-a3a637a9857b" (UID: "0c62899b-4f3e-4ff7-80d3-a3a637a9857b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:02.105440 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.105362    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-tmp-dir" pod "0c62899b-4f3e-4ff7-80d3-a3a637a9857b" (UID: "0c62899b-4f3e-4ff7-80d3-a3a637a9857b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:02.105762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.105684    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:24:02.105762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.105736    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:24:02.105964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.105781    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-home" pod "0c62899b-4f3e-4ff7-80d3-a3a637a9857b" (UID: "0c62899b-4f3e-4ff7-80d3-a3a637a9857b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:02.107468 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.107441    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-dshm" pod "0c62899b-4f3e-4ff7-80d3-a3a637a9857b" (UID: "0c62899b-4f3e-4ff7-80d3-a3a637a9857b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:02.107588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.107465    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-tls-certs" pod "0c62899b-4f3e-4ff7-80d3-a3a637a9857b" (UID: "0c62899b-4f3e-4ff7-80d3-a3a637a9857b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:02.107588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.107445    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-kube-api-access-zfqkl" pod "0c62899b-4f3e-4ff7-80d3-a3a637a9857b" (UID: "0c62899b-4f3e-4ff7-80d3-a3a637a9857b"). InnerVolumeSpecName "kube-api-access-zfqkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:02.207190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.207096    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:24:02.207190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.207122    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:24:02.207190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.207134    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:24:02.207190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.207149    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfqkl\" (UniqueName: \"kubernetes.io/projected/0c62899b-4f3e-4ff7-80d3-a3a637a9857b-kube-api-access-zfqkl\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:24:02.677568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.677530    4425 generic.go:356] "Generic (PLEG): container finished" podID="0c62899b-4f3e-4ff7-80d3-a3a637a9857b" containerID="fa72e29293737987cbbf4a2e4f110b0cd5d2cd3afd2d38df510463662c5dd2b4" exitCode=0
Oct 30 01:24:02.678040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.677601    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"
Oct 30 01:24:02.678040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.677596    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg" event={"ID":"0c62899b-4f3e-4ff7-80d3-a3a637a9857b","Type":"ContainerDied","Data":"fa72e29293737987cbbf4a2e4f110b0cd5d2cd3afd2d38df510463662c5dd2b4"}
Oct 30 01:24:02.678040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.677715    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg" event={"ID":"0c62899b-4f3e-4ff7-80d3-a3a637a9857b","Type":"ContainerDied","Data":"50b4e11bfa992b3b7769fcb249868a876abf532f2b8a7fc9c9b0c40620d9cdb6"}
Oct 30 01:24:02.678040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.677741    4425 scope.go:122] "RemoveContainer" containerID="fa72e29293737987cbbf4a2e4f110b0cd5d2cd3afd2d38df510463662c5dd2b4"
Oct 30 01:24:02.692794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.692717    4425 scope.go:122] "RemoveContainer" containerID="fa72e29293737987cbbf4a2e4f110b0cd5d2cd3afd2d38df510463662c5dd2b4"
Oct 30 01:24:02.692794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.692780    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj" podUID="64f44cf3-24b7-4aea-a48f-b9a395503795" containerName="payload-processing" containerID="cri-o://cbf2011b5aa21b7e24ed8da44491bfafbca219b61d3d9ce936a59c4f391af6c1" gracePeriod=25
Oct 30 01:24:02.693613 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:24:02.693586    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa72e29293737987cbbf4a2e4f110b0cd5d2cd3afd2d38df510463662c5dd2b4\": container with ID starting with fa72e29293737987cbbf4a2e4f110b0cd5d2cd3afd2d38df510463662c5dd2b4 not found: ID does not exist" containerID="fa72e29293737987cbbf4a2e4f110b0cd5d2cd3afd2d38df510463662c5dd2b4"
Oct 30 01:24:02.693712 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.693624    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa72e29293737987cbbf4a2e4f110b0cd5d2cd3afd2d38df510463662c5dd2b4"} err="failed to get container status \"fa72e29293737987cbbf4a2e4f110b0cd5d2cd3afd2d38df510463662c5dd2b4\": rpc error: code = NotFound desc = could not find container \"fa72e29293737987cbbf4a2e4f110b0cd5d2cd3afd2d38df510463662c5dd2b4\": container with ID starting with fa72e29293737987cbbf4a2e4f110b0cd5d2cd3afd2d38df510463662c5dd2b4 not found: ID does not exist"
Oct 30 01:24:02.719645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.719621    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp" podUID="f3a20b3b-516e-4e9a-b2c3-ec9d6c826348" containerName="payload-pre-processing" containerID="cri-o://1ca22975cde6b9234dc347795aa804a09ed4e073c2ea3ebbd9781e32848c38df" gracePeriod=25
Oct 30 01:24:02.728705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.728681    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"]
Oct 30 01:24:02.734077 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:02.734057    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-637e08/autores-model-637e08-kserve-5594b89b77-wcvbg"]
Oct 30 01:24:03.050357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.050330    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"
Oct 30 01:24:03.081751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.081722    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"
Oct 30 01:24:03.141075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.141044    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3a20b3b-516e-4e9a-b2c3-ec9d6c826348-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a20b3b-516e-4e9a-b2c3-ec9d6c826348-plugins-config-volume\") pod \"f3a20b3b-516e-4e9a-b2c3-ec9d6c826348\" (UID: \"f3a20b3b-516e-4e9a-b2c3-ec9d6c826348\") "
Oct 30 01:24:03.141357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.141105    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3a20b3b-516e-4e9a-b2c3-ec9d6c826348-kube-api-access-qqrfg\" (UniqueName: \"kubernetes.io/projected/f3a20b3b-516e-4e9a-b2c3-ec9d6c826348-kube-api-access-qqrfg\") pod \"f3a20b3b-516e-4e9a-b2c3-ec9d6c826348\" (UID: \"f3a20b3b-516e-4e9a-b2c3-ec9d6c826348\") "
Oct 30 01:24:03.141357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.141150    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f44cf3-24b7-4aea-a48f-b9a395503795-kube-api-access-hgkzb\" (UniqueName: \"kubernetes.io/projected/64f44cf3-24b7-4aea-a48f-b9a395503795-kube-api-access-hgkzb\") pod \"64f44cf3-24b7-4aea-a48f-b9a395503795\" (UID: \"64f44cf3-24b7-4aea-a48f-b9a395503795\") "
Oct 30 01:24:03.141537 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.141354    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f44cf3-24b7-4aea-a48f-b9a395503795-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f44cf3-24b7-4aea-a48f-b9a395503795-plugins-config-volume\") pod \"64f44cf3-24b7-4aea-a48f-b9a395503795\" (UID: \"64f44cf3-24b7-4aea-a48f-b9a395503795\") "
Oct 30 01:24:03.141658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.141538    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a20b3b-516e-4e9a-b2c3-ec9d6c826348-plugins-config-volume" pod "f3a20b3b-516e-4e9a-b2c3-ec9d6c826348" (UID: "f3a20b3b-516e-4e9a-b2c3-ec9d6c826348"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:03.141933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.141907    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f44cf3-24b7-4aea-a48f-b9a395503795-plugins-config-volume" pod "64f44cf3-24b7-4aea-a48f-b9a395503795" (UID: "64f44cf3-24b7-4aea-a48f-b9a395503795"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:03.142038 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.141996    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a20b3b-516e-4e9a-b2c3-ec9d6c826348-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.142038 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.142014    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f44cf3-24b7-4aea-a48f-b9a395503795-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.143897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.143868    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a20b3b-516e-4e9a-b2c3-ec9d6c826348-kube-api-access-qqrfg" pod "f3a20b3b-516e-4e9a-b2c3-ec9d6c826348" (UID: "f3a20b3b-516e-4e9a-b2c3-ec9d6c826348"). InnerVolumeSpecName "kube-api-access-qqrfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:03.144206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.144143    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f44cf3-24b7-4aea-a48f-b9a395503795-kube-api-access-hgkzb" pod "64f44cf3-24b7-4aea-a48f-b9a395503795" (UID: "64f44cf3-24b7-4aea-a48f-b9a395503795"). InnerVolumeSpecName "kube-api-access-hgkzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:03.243536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.243459    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqrfg\" (UniqueName: \"kubernetes.io/projected/f3a20b3b-516e-4e9a-b2c3-ec9d6c826348-kube-api-access-qqrfg\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.243536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.243487    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgkzb\" (UniqueName: \"kubernetes.io/projected/64f44cf3-24b7-4aea-a48f-b9a395503795-kube-api-access-hgkzb\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.424343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.424314    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c62899b-4f3e-4ff7-80d3-a3a637a9857b" path="/var/lib/kubelet/pods/0c62899b-4f3e-4ff7-80d3-a3a637a9857b/volumes"
Oct 30 01:24:03.686422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.686394    4425 generic.go:356] "Generic (PLEG): container finished" podID="f3a20b3b-516e-4e9a-b2c3-ec9d6c826348" containerID="1ca22975cde6b9234dc347795aa804a09ed4e073c2ea3ebbd9781e32848c38df" exitCode=0
Oct 30 01:24:03.686895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.686445    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"
Oct 30 01:24:03.686895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.686480    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp" event={"ID":"f3a20b3b-516e-4e9a-b2c3-ec9d6c826348","Type":"ContainerDied","Data":"1ca22975cde6b9234dc347795aa804a09ed4e073c2ea3ebbd9781e32848c38df"}
Oct 30 01:24:03.686895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.686525    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp" event={"ID":"f3a20b3b-516e-4e9a-b2c3-ec9d6c826348","Type":"ContainerDied","Data":"e7713de134b4a29bc7c365416f31e6a1c82322208deed98a38b76635d98a0148"}
Oct 30 01:24:03.686895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.686550    4425 scope.go:122] "RemoveContainer" containerID="1ca22975cde6b9234dc347795aa804a09ed4e073c2ea3ebbd9781e32848c38df"
Oct 30 01:24:03.689147 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.689121    4425 generic.go:356] "Generic (PLEG): container finished" podID="64f44cf3-24b7-4aea-a48f-b9a395503795" containerID="cbf2011b5aa21b7e24ed8da44491bfafbca219b61d3d9ce936a59c4f391af6c1" exitCode=0
Oct 30 01:24:03.689285 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.689168    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj" event={"ID":"64f44cf3-24b7-4aea-a48f-b9a395503795","Type":"ContainerDied","Data":"cbf2011b5aa21b7e24ed8da44491bfafbca219b61d3d9ce936a59c4f391af6c1"}
Oct 30 01:24:03.689285 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.689210    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"
Oct 30 01:24:03.689285 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.689227    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj" event={"ID":"64f44cf3-24b7-4aea-a48f-b9a395503795","Type":"ContainerDied","Data":"488548c90ff8242063fc62c2f74051c5a1728a733ac8c6caa5237405a0452370"}
Oct 30 01:24:03.697528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.697508    4425 scope.go:122] "RemoveContainer" containerID="1ca22975cde6b9234dc347795aa804a09ed4e073c2ea3ebbd9781e32848c38df"
Oct 30 01:24:03.697807 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:24:03.697786    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca22975cde6b9234dc347795aa804a09ed4e073c2ea3ebbd9781e32848c38df\": container with ID starting with 1ca22975cde6b9234dc347795aa804a09ed4e073c2ea3ebbd9781e32848c38df not found: ID does not exist" containerID="1ca22975cde6b9234dc347795aa804a09ed4e073c2ea3ebbd9781e32848c38df"
Oct 30 01:24:03.697907 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.697813    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca22975cde6b9234dc347795aa804a09ed4e073c2ea3ebbd9781e32848c38df"} err="failed to get container status \"1ca22975cde6b9234dc347795aa804a09ed4e073c2ea3ebbd9781e32848c38df\": rpc error: code = NotFound desc = could not find container \"1ca22975cde6b9234dc347795aa804a09ed4e073c2ea3ebbd9781e32848c38df\": container with ID starting with 1ca22975cde6b9234dc347795aa804a09ed4e073c2ea3ebbd9781e32848c38df not found: ID does not exist"
Oct 30 01:24:03.697907 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.697830    4425 scope.go:122] "RemoveContainer" containerID="cbf2011b5aa21b7e24ed8da44491bfafbca219b61d3d9ce936a59c4f391af6c1"
Oct 30 01:24:03.717834 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.717816    4425 scope.go:122] "RemoveContainer" containerID="cbf2011b5aa21b7e24ed8da44491bfafbca219b61d3d9ce936a59c4f391af6c1"
Oct 30 01:24:03.718087 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:24:03.718069    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf2011b5aa21b7e24ed8da44491bfafbca219b61d3d9ce936a59c4f391af6c1\": container with ID starting with cbf2011b5aa21b7e24ed8da44491bfafbca219b61d3d9ce936a59c4f391af6c1 not found: ID does not exist" containerID="cbf2011b5aa21b7e24ed8da44491bfafbca219b61d3d9ce936a59c4f391af6c1"
Oct 30 01:24:03.718137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.718093    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf2011b5aa21b7e24ed8da44491bfafbca219b61d3d9ce936a59c4f391af6c1"} err="failed to get container status \"cbf2011b5aa21b7e24ed8da44491bfafbca219b61d3d9ce936a59c4f391af6c1\": rpc error: code = NotFound desc = could not find container \"cbf2011b5aa21b7e24ed8da44491bfafbca219b61d3d9ce936a59c4f391af6c1\": container with ID starting with cbf2011b5aa21b7e24ed8da44491bfafbca219b61d3d9ce936a59c4f391af6c1 not found: ID does not exist"
Oct 30 01:24:03.732383 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.732362    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"]
Oct 30 01:24:03.741550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.741530    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-637e08-575b5d99f-bx7hp"]
Oct 30 01:24:03.767449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.767426    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"]
Oct 30 01:24:03.772684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:03.772663    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-637e08-f695688c4-8x5bj"]
Oct 30 01:24:05.424448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:05.424418    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64f44cf3-24b7-4aea-a48f-b9a395503795" path="/var/lib/kubelet/pods/64f44cf3-24b7-4aea-a48f-b9a395503795/volumes"
Oct 30 01:24:05.424818 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:05.424801    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3a20b3b-516e-4e9a-b2c3-ec9d6c826348" path="/var/lib/kubelet/pods/f3a20b3b-516e-4e9a-b2c3-ec9d6c826348/volumes"
Oct 30 01:24:07.555182 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:24:07.555135    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:24:12.363564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.363531    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"]
Oct 30 01:24:12.364209 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.364192    4425 cpu_manager.go:414] "containerMap: removing container" podUID="64f44cf3-24b7-4aea-a48f-b9a395503795" containerName="payload-processing"
Oct 30 01:24:12.364209 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.364208    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64f44cf3-24b7-4aea-a48f-b9a395503795" containerName="payload-processing"
Oct 30 01:24:12.364369 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.364238    4425 cpu_manager.go:414] "containerMap: removing container" podUID="f3a20b3b-516e-4e9a-b2c3-ec9d6c826348" containerName="payload-pre-processing"
Oct 30 01:24:12.364369 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.364244    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3a20b3b-516e-4e9a-b2c3-ec9d6c826348" containerName="payload-pre-processing"
Oct 30 01:24:12.364369 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.364258    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d9647e5c-0609-4ef6-97d0-476df25a3bc3" containerName="revoke-keys"
Oct 30 01:24:12.364369 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.364263    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9647e5c-0609-4ef6-97d0-476df25a3bc3" containerName="revoke-keys"
Oct 30 01:24:12.364369 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.364276    4425 cpu_manager.go:414] "containerMap: removing container" podUID="0c62899b-4f3e-4ff7-80d3-a3a637a9857b" containerName="main"
Oct 30 01:24:12.364369 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.364281    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c62899b-4f3e-4ff7-80d3-a3a637a9857b" containerName="main"
Oct 30 01:24:12.364651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.364377    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="64f44cf3-24b7-4aea-a48f-b9a395503795" containerName="payload-processing"
Oct 30 01:24:12.364651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.364391    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3a20b3b-516e-4e9a-b2c3-ec9d6c826348" containerName="payload-pre-processing"
Oct 30 01:24:12.364651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.364399    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9647e5c-0609-4ef6-97d0-476df25a3bc3" containerName="revoke-keys"
Oct 30 01:24:12.364651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.364407    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c62899b-4f3e-4ff7-80d3-a3a637a9857b" containerName="main"
Oct 30 01:24:12.370350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.370329    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:24:12.383206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.383187    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:24:12.392702 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.392576    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-94ea1e-serving-cert\""
Oct 30 01:24:12.392702 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.392644    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 01:24:12.392702 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.392684    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:24:12.393086 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.393071    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p8xs2\""
Oct 30 01:24:12.398109 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.398085    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"]
Oct 30 01:24:12.465521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.465494    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68f8c428-fa70-411a-bd16-2861555890bf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b\" (UID: \"68f8c428-fa70-411a-bd16-2861555890bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:24:12.465690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.465565    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68f8c428-fa70-411a-bd16-2861555890bf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b\" (UID: \"68f8c428-fa70-411a-bd16-2861555890bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:24:12.465766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.465720    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f8c428-fa70-411a-bd16-2861555890bf-metrics-certs\") pod \"maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b\" (UID: \"68f8c428-fa70-411a-bd16-2861555890bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:24:12.465766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.465756    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xxgf\" (UniqueName: \"kubernetes.io/projected/68f8c428-fa70-411a-bd16-2861555890bf-kube-api-access-8xxgf\") pod \"maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b\" (UID: \"68f8c428-fa70-411a-bd16-2861555890bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:24:12.566883 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.566856    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68f8c428-fa70-411a-bd16-2861555890bf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b\" (UID: \"68f8c428-fa70-411a-bd16-2861555890bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:24:12.567020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.566904    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68f8c428-fa70-411a-bd16-2861555890bf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b\" (UID: \"68f8c428-fa70-411a-bd16-2861555890bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:24:12.567020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.566968    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f8c428-fa70-411a-bd16-2861555890bf-metrics-certs\") pod \"maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b\" (UID: \"68f8c428-fa70-411a-bd16-2861555890bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:24:12.567020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.566989    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xxgf\" (UniqueName: \"kubernetes.io/projected/68f8c428-fa70-411a-bd16-2861555890bf-kube-api-access-8xxgf\") pod \"maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b\" (UID: \"68f8c428-fa70-411a-bd16-2861555890bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:24:12.567639 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.567617    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68f8c428-fa70-411a-bd16-2861555890bf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b\" (UID: \"68f8c428-fa70-411a-bd16-2861555890bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:24:12.569439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.569420    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f8c428-fa70-411a-bd16-2861555890bf-metrics-certs\") pod \"maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b\" (UID: \"68f8c428-fa70-411a-bd16-2861555890bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:24:12.569530 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.569489    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68f8c428-fa70-411a-bd16-2861555890bf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b\" (UID: \"68f8c428-fa70-411a-bd16-2861555890bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:24:12.586806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.586785    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"]
Oct 30 01:24:12.589552 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.589533    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xxgf\" (UniqueName: \"kubernetes.io/projected/68f8c428-fa70-411a-bd16-2861555890bf-kube-api-access-8xxgf\") pod \"maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b\" (UID: \"68f8c428-fa70-411a-bd16-2861555890bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:24:12.591640 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.591625    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:24:12.609137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.609106    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dd80cb\"/\"routing-overlay\""
Oct 30 01:24:12.609366 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.609348    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dd80cb\"/\"openshift-service-ca.crt\""
Oct 30 01:24:12.609450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.609352    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dd80cb\"/\"kube-root-ca.crt\""
Oct 30 01:24:12.609450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.609387    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dd80cb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-dd80cb\""
Oct 30 01:24:12.614195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.614114    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"]
Oct 30 01:24:12.614345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.614329    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dd80cb\"/\"agc-cred-6b76c2a546af\""
Oct 30 01:24:12.614572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.614551    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dd80cb\"/\"payload-processing-external-model-e2e--871060f7-dockercfg-rlvb4\""
Oct 30 01:24:12.667616 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.667589    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3254932d-a86a-408b-8567-869cc43f80ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x\" (UID: \"3254932d-a86a-408b-8567-869cc43f80ba\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:24:12.667760 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.667742    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3254932d-a86a-408b-8567-869cc43f80ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x\" (UID: \"3254932d-a86a-408b-8567-869cc43f80ba\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:24:12.667864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.667847    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3254932d-a86a-408b-8567-869cc43f80ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x\" (UID: \"3254932d-a86a-408b-8567-869cc43f80ba\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:24:12.667961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.667948    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sscb\" (UniqueName: \"kubernetes.io/projected/3254932d-a86a-408b-8567-869cc43f80ba-kube-api-access-7sscb\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x\" (UID: \"3254932d-a86a-408b-8567-869cc43f80ba\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:24:12.684984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.684962    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:24:12.769266 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.768972    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3254932d-a86a-408b-8567-869cc43f80ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x\" (UID: \"3254932d-a86a-408b-8567-869cc43f80ba\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:24:12.769266 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.769102    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3254932d-a86a-408b-8567-869cc43f80ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x\" (UID: \"3254932d-a86a-408b-8567-869cc43f80ba\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:24:12.775288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.770098    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3254932d-a86a-408b-8567-869cc43f80ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x\" (UID: \"3254932d-a86a-408b-8567-869cc43f80ba\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:24:12.775288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.770250    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3254932d-a86a-408b-8567-869cc43f80ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x\" (UID: \"3254932d-a86a-408b-8567-869cc43f80ba\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:24:12.775288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.770339    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sscb\" (UniqueName: \"kubernetes.io/projected/3254932d-a86a-408b-8567-869cc43f80ba-kube-api-access-7sscb\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x\" (UID: \"3254932d-a86a-408b-8567-869cc43f80ba\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:24:12.775288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.771631    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3254932d-a86a-408b-8567-869cc43f80ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x\" (UID: \"3254932d-a86a-408b-8567-869cc43f80ba\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:24:12.775288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.774244    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3254932d-a86a-408b-8567-869cc43f80ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x\" (UID: \"3254932d-a86a-408b-8567-869cc43f80ba\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:24:12.792930 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.792571    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sscb\" (UniqueName: \"kubernetes.io/projected/3254932d-a86a-408b-8567-869cc43f80ba-kube-api-access-7sscb\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x\" (UID: \"3254932d-a86a-408b-8567-869cc43f80ba\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:24:12.915468 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:12.915391    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:24:13.010290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:13.010265    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"]
Oct 30 01:24:13.011551 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:24:13.011527    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f8c428_fa70_411a_bd16_2861555890bf.slice/crio-b652ae6ed1be311ee7bfffbe7b9b1d5f4f4ef63a1685226eb70d91ddc265152e WatchSource:0}: Error finding container b652ae6ed1be311ee7bfffbe7b9b1d5f4f4ef63a1685226eb70d91ddc265152e: Status 404 returned error can't find the container with id b652ae6ed1be311ee7bfffbe7b9b1d5f4f4ef63a1685226eb70d91ddc265152e
Oct 30 01:24:13.012992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:13.012973    4425 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:24:13.220339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:13.220288    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"]
Oct 30 01:24:13.221394 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:24:13.221370    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3254932d_a86a_408b_8567_869cc43f80ba.slice/crio-12e5afaa8299e6ef7b27f18dee8b4d59592f1d51612763147af7e67c60cc88a3 WatchSource:0}: Error finding container 12e5afaa8299e6ef7b27f18dee8b4d59592f1d51612763147af7e67c60cc88a3: Status 404 returned error can't find the container with id 12e5afaa8299e6ef7b27f18dee8b4d59592f1d51612763147af7e67c60cc88a3
Oct 30 01:24:13.776236 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:13.776200    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x" event={"ID":"3254932d-a86a-408b-8567-869cc43f80ba","Type":"ContainerStarted","Data":"d7b591ce02280388e4be03e372e9d3bba0bbcc16613a39ae814b594fa63f15cc"}
Oct 30 01:24:13.776710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:13.776260    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x" event={"ID":"3254932d-a86a-408b-8567-869cc43f80ba","Type":"ContainerStarted","Data":"12e5afaa8299e6ef7b27f18dee8b4d59592f1d51612763147af7e67c60cc88a3"}
Oct 30 01:24:13.776710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:13.776290    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:24:13.777758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:13.777723    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b" event={"ID":"68f8c428-fa70-411a-bd16-2861555890bf","Type":"ContainerStarted","Data":"b652ae6ed1be311ee7bfffbe7b9b1d5f4f4ef63a1685226eb70d91ddc265152e"}
Oct 30 01:24:13.814547 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:13.814497    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x" podStartSLOduration=1.8144827239999999 podStartE2EDuration="1.814482724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:13.812835572 +0000 UTC m=+2562.562490884" watchObservedRunningTime="2026-10-30 01:24:13.814482724 +0000 UTC m=+2562.564138032"
Oct 30 01:24:14.786307 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:14.786263    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b" event={"ID":"68f8c428-fa70-411a-bd16-2861555890bf","Type":"ContainerStarted","Data":"2bf5c4af154cfabaf72b0c2251e266888655b006fbf876e31a94d15cd114734a"}
Oct 30 01:24:14.786827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:14.786363    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:24:14.824188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:14.824097    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b" podStartSLOduration=1.279063951 podStartE2EDuration="2.824084335s" totalImagesPullingTime="1.545020384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:14.822971776 +0000 UTC m=+2563.572627085" watchObservedRunningTime="2026-10-30 01:24:14.824084335 +0000 UTC m=+2563.573739643"
Oct 30 01:24:20.797662 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:20.797633    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:24:24.790596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:24.790565    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:24:32.967015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:32.966969    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"]
Oct 30 01:24:32.975842 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:32.975804    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-637e08-5b4tl"]
Oct 30 01:24:33.241286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:33.241166    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:24:33.426624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:33.426592    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9647e5c-0609-4ef6-97d0-476df25a3bc3" path="/var/lib/kubelet/pods/d9647e5c-0609-4ef6-97d0-476df25a3bc3/volumes"
Oct 30 01:24:37.559783 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:24:37.559747    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:24:43.199919 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.199882    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"]
Oct 30 01:24:43.206331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.206303    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:24:43.229387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.229345    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1afb3e\"/\"kube-root-ca.crt\""
Oct 30 01:24:43.229387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.229363    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1afb3e\"/\"routing-overlay\""
Oct 30 01:24:43.229387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.229356    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1afb3e\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1afb3e\""
Oct 30 01:24:43.229387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.229386    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1afb3e\"/\"openshift-service-ca.crt\""
Oct 30 01:24:43.232870 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.232833    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1afb3e\"/\"agc-cred-1aa3b9ce7f6d\""
Oct 30 01:24:43.233047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.233033    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1afb3e\"/\"payload-processing-external-model-e2e--8ac1e578-dockercfg-wt9sb\""
Oct 30 01:24:43.237183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.237134    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"]
Oct 30 01:24:43.379291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.379251    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8zjv\" (UniqueName: \"kubernetes.io/projected/9f921d86-8439-40e7-bb10-3a3f55b7f918-kube-api-access-s8zjv\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s\" (UID: \"9f921d86-8439-40e7-bb10-3a3f55b7f918\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:24:43.379291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.379297    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f921d86-8439-40e7-bb10-3a3f55b7f918-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s\" (UID: \"9f921d86-8439-40e7-bb10-3a3f55b7f918\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:24:43.379549 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.379335    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f921d86-8439-40e7-bb10-3a3f55b7f918-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s\" (UID: \"9f921d86-8439-40e7-bb10-3a3f55b7f918\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:24:43.379549 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.379502    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f921d86-8439-40e7-bb10-3a3f55b7f918-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s\" (UID: \"9f921d86-8439-40e7-bb10-3a3f55b7f918\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:24:43.480895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.480793    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8zjv\" (UniqueName: \"kubernetes.io/projected/9f921d86-8439-40e7-bb10-3a3f55b7f918-kube-api-access-s8zjv\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s\" (UID: \"9f921d86-8439-40e7-bb10-3a3f55b7f918\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:24:43.480895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.480837    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f921d86-8439-40e7-bb10-3a3f55b7f918-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s\" (UID: \"9f921d86-8439-40e7-bb10-3a3f55b7f918\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:24:43.480895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.480866    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f921d86-8439-40e7-bb10-3a3f55b7f918-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s\" (UID: \"9f921d86-8439-40e7-bb10-3a3f55b7f918\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:24:43.481263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.480971    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f921d86-8439-40e7-bb10-3a3f55b7f918-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s\" (UID: \"9f921d86-8439-40e7-bb10-3a3f55b7f918\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:24:43.481886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.481854    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f921d86-8439-40e7-bb10-3a3f55b7f918-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s\" (UID: \"9f921d86-8439-40e7-bb10-3a3f55b7f918\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:24:43.481886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.481881    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f921d86-8439-40e7-bb10-3a3f55b7f918-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s\" (UID: \"9f921d86-8439-40e7-bb10-3a3f55b7f918\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:24:43.484001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.483970    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f921d86-8439-40e7-bb10-3a3f55b7f918-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s\" (UID: \"9f921d86-8439-40e7-bb10-3a3f55b7f918\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:24:43.502104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.502066    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8zjv\" (UniqueName: \"kubernetes.io/projected/9f921d86-8439-40e7-bb10-3a3f55b7f918-kube-api-access-s8zjv\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s\" (UID: \"9f921d86-8439-40e7-bb10-3a3f55b7f918\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:24:43.522799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.522759    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:24:43.882588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:43.882548    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"]
Oct 30 01:24:43.883894 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:24:43.883857    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f921d86_8439_40e7_bb10_3a3f55b7f918.slice/crio-198a803400ceaf1b712a1d15bff84f56c09c389c5e6fbbad864c4df845f88965 WatchSource:0}: Error finding container 198a803400ceaf1b712a1d15bff84f56c09c389c5e6fbbad864c4df845f88965: Status 404 returned error can't find the container with id 198a803400ceaf1b712a1d15bff84f56c09c389c5e6fbbad864c4df845f88965
Oct 30 01:24:44.011242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.011194    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s" event={"ID":"9f921d86-8439-40e7-bb10-3a3f55b7f918","Type":"ContainerStarted","Data":"1e2cff83b7c1b8472b8404eb6f32049dda034df9b9005aebbaa3fb94bffca693"}
Oct 30 01:24:44.011504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.011270    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s" event={"ID":"9f921d86-8439-40e7-bb10-3a3f55b7f918","Type":"ContainerStarted","Data":"198a803400ceaf1b712a1d15bff84f56c09c389c5e6fbbad864c4df845f88965"}
Oct 30 01:24:44.011504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.011323    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:24:44.053780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.053724    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s" podStartSLOduration=1.053707644 podStartE2EDuration="1.053707644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:44.052771459 +0000 UTC m=+2592.802426801" watchObservedRunningTime="2026-10-30 01:24:44.053707644 +0000 UTC m=+2592.803362953"
Oct 30 01:24:44.080291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.080241    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"]
Oct 30 01:24:44.085247 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.085223    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:24:44.105226 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.105183    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"agc-cred-ddb5d36df12e\""
Oct 30 01:24:44.105418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.105403    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"payload-processing-external-model-e2e--53969a13-dockercfg-h75vd\""
Oct 30 01:24:44.107571 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.107547    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"openshift-service-ca.crt\""
Oct 30 01:24:44.107770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.107751    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ae1820\""
Oct 30 01:24:44.107960 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.107937    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"routing-overlay\""
Oct 30 01:24:44.109551 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.109509    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"kube-root-ca.crt\""
Oct 30 01:24:44.112509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.112481    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"]
Oct 30 01:24:44.191405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.191303    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv\" (UID: \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:24:44.191405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.191373    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8k2s\" (UniqueName: \"kubernetes.io/projected/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-kube-api-access-t8k2s\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv\" (UID: \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:24:44.191610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.191530    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv\" (UID: \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:24:44.191694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.191669    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv\" (UID: \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:24:44.293711 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.293670    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv\" (UID: \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:24:44.293711 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.293718    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv\" (UID: \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:24:44.294418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.293881    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv\" (UID: \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:24:44.294418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.293913    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8k2s\" (UniqueName: \"kubernetes.io/projected/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-kube-api-access-t8k2s\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv\" (UID: \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:24:44.294626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.294590    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv\" (UID: \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:24:44.294751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.294692    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv\" (UID: \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:24:44.296643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.296618    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv\" (UID: \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:24:44.315560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.315526    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8k2s\" (UniqueName: \"kubernetes.io/projected/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-kube-api-access-t8k2s\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv\" (UID: \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:24:44.401390 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.401339    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:24:44.709889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:44.709680    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"]
Oct 30 01:24:45.023984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:45.023947    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv" event={"ID":"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066","Type":"ContainerStarted","Data":"c33ca5a481c754265159f9c7a47aa29f7ac15583fb038e9140ad8e1fa9f9db0f"}
Oct 30 01:24:45.024184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:45.024004    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv" event={"ID":"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066","Type":"ContainerStarted","Data":"4e2d3a8212ebd247dabb209bed707fddd657299c01e9d6dc14eabf4905667f9f"}
Oct 30 01:24:45.024184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:45.024057    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:24:45.075779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:45.075720    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv" podStartSLOduration=1.07570439 podStartE2EDuration="1.07570439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:45.0723596 +0000 UTC m=+2593.822014908" watchObservedRunningTime="2026-10-30 01:24:45.07570439 +0000 UTC m=+2593.825359699"
Oct 30 01:24:55.028167 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:55.028117    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:24:57.042842 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:24:57.042808    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:25:07.558465 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:25:07.558420    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:25:12.108290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.108246    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"]
Oct 30 01:25:12.112108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.112086    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"
Oct 30 01:25:12.125497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.125461    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-1cb829-dockercfg-xdw7n\""
Oct 30 01:25:12.135389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.135351    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-1cb829\""
Oct 30 01:25:12.141975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.141938    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"]
Oct 30 01:25:12.161868 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.161809    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde72706-a81b-4b93-96c6-f02dd6a96f3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm\" (UID: \"fde72706-a81b-4b93-96c6-f02dd6a96f3c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"
Oct 30 01:25:12.161868 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.161823    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"]
Oct 30 01:25:12.162135 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.161919    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47qhb\" (UniqueName: \"kubernetes.io/projected/fde72706-a81b-4b93-96c6-f02dd6a96f3c-kube-api-access-47qhb\") pod \"payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm\" (UID: \"fde72706-a81b-4b93-96c6-f02dd6a96f3c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"
Oct 30 01:25:12.171089 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.171047    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"
Oct 30 01:25:12.198251 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.198011    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"]
Oct 30 01:25:12.268398 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.268346    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde72706-a81b-4b93-96c6-f02dd6a96f3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm\" (UID: \"fde72706-a81b-4b93-96c6-f02dd6a96f3c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"
Oct 30 01:25:12.268633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.268435    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qhb\" (UniqueName: \"kubernetes.io/projected/fde72706-a81b-4b93-96c6-f02dd6a96f3c-kube-api-access-47qhb\") pod \"payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm\" (UID: \"fde72706-a81b-4b93-96c6-f02dd6a96f3c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"
Oct 30 01:25:12.270000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.269968    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde72706-a81b-4b93-96c6-f02dd6a96f3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm\" (UID: \"fde72706-a81b-4b93-96c6-f02dd6a96f3c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"
Oct 30 01:25:12.290116 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.290079    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47qhb\" (UniqueName: \"kubernetes.io/projected/fde72706-a81b-4b93-96c6-f02dd6a96f3c-kube-api-access-47qhb\") pod \"payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm\" (UID: \"fde72706-a81b-4b93-96c6-f02dd6a96f3c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"
Oct 30 01:25:12.370073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.369945    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2htcp\" (UniqueName: \"kubernetes.io/projected/892671e5-857b-4816-9376-f6862e47dd40-kube-api-access-2htcp\") pod \"payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt\" (UID: \"892671e5-857b-4816-9376-f6862e47dd40\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"
Oct 30 01:25:12.370073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.370017    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/892671e5-857b-4816-9376-f6862e47dd40-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt\" (UID: \"892671e5-857b-4816-9376-f6862e47dd40\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"
Oct 30 01:25:12.429128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.429077    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"
Oct 30 01:25:12.471312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.471280    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2htcp\" (UniqueName: \"kubernetes.io/projected/892671e5-857b-4816-9376-f6862e47dd40-kube-api-access-2htcp\") pod \"payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt\" (UID: \"892671e5-857b-4816-9376-f6862e47dd40\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"
Oct 30 01:25:12.471550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.471341    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/892671e5-857b-4816-9376-f6862e47dd40-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt\" (UID: \"892671e5-857b-4816-9376-f6862e47dd40\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"
Oct 30 01:25:12.472309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.472277    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/892671e5-857b-4816-9376-f6862e47dd40-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt\" (UID: \"892671e5-857b-4816-9376-f6862e47dd40\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"
Oct 30 01:25:12.496081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.496046    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2htcp\" (UniqueName: \"kubernetes.io/projected/892671e5-857b-4816-9376-f6862e47dd40-kube-api-access-2htcp\") pod \"payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt\" (UID: \"892671e5-857b-4816-9376-f6862e47dd40\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"
Oct 30 01:25:12.740885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.740792    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"]
Oct 30 01:25:12.745388 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:25:12.745350    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde72706_a81b_4b93_96c6_f02dd6a96f3c.slice/crio-a966910e15cbea6d8ac459a7303638785082006c35a268cf17e2dded9592fdfe WatchSource:0}: Error finding container a966910e15cbea6d8ac459a7303638785082006c35a268cf17e2dded9592fdfe: Status 404 returned error can't find the container with id a966910e15cbea6d8ac459a7303638785082006c35a268cf17e2dded9592fdfe
Oct 30 01:25:12.792327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:12.792279    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"
Oct 30 01:25:13.110559 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:13.110522    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"]
Oct 30 01:25:13.110927 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:25:13.110532    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892671e5_857b_4816_9376_f6862e47dd40.slice/crio-579b6c97e66f1bbe4a78bfd12dcaa3c7af2f1252bbea59b2b5aba4f476bdfa08 WatchSource:0}: Error finding container 579b6c97e66f1bbe4a78bfd12dcaa3c7af2f1252bbea59b2b5aba4f476bdfa08: Status 404 returned error can't find the container with id 579b6c97e66f1bbe4a78bfd12dcaa3c7af2f1252bbea59b2b5aba4f476bdfa08
Oct 30 01:25:13.292990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:13.292940    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm" event={"ID":"fde72706-a81b-4b93-96c6-f02dd6a96f3c","Type":"ContainerStarted","Data":"3f4f2ef4769034a517ef1a0b01cff5625b68888378cbdfa8f47546388451c002"}
Oct 30 01:25:13.293248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:13.293021    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm" event={"ID":"fde72706-a81b-4b93-96c6-f02dd6a96f3c","Type":"ContainerStarted","Data":"a966910e15cbea6d8ac459a7303638785082006c35a268cf17e2dded9592fdfe"}
Oct 30 01:25:13.294291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:13.294265    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"
Oct 30 01:25:13.296133 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:13.296103    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt" event={"ID":"892671e5-857b-4816-9376-f6862e47dd40","Type":"ContainerStarted","Data":"eaab8886600a3a447a871bd80c08af7cc20054d24eb777cc39e8c53d6fa82b95"}
Oct 30 01:25:13.296279 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:13.296151    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt" event={"ID":"892671e5-857b-4816-9376-f6862e47dd40","Type":"ContainerStarted","Data":"579b6c97e66f1bbe4a78bfd12dcaa3c7af2f1252bbea59b2b5aba4f476bdfa08"}
Oct 30 01:25:13.296849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:13.296833    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"
Oct 30 01:25:13.332177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:13.331865    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm" podStartSLOduration=1.3318443389999999 podStartE2EDuration="1.331844339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:13.328943474 +0000 UTC m=+2622.078598796" watchObservedRunningTime="2026-10-30 01:25:13.331844339 +0000 UTC m=+2622.081499648"
Oct 30 01:25:13.370854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:13.370729    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt" podStartSLOduration=1.3707113 podStartE2EDuration="1.3707113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:13.367294535 +0000 UTC m=+2622.116949881" watchObservedRunningTime="2026-10-30 01:25:13.3707113 +0000 UTC m=+2622.120366804"
Oct 30 01:25:14.698830 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.698789    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"]
Oct 30 01:25:14.703337 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.703312    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:25:14.716214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.716173    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"payload-processing-external-model-e2e--33153ae7-dockercfg-gggq4\""
Oct 30 01:25:14.716492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.716473    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"routing-overlay\""
Oct 30 01:25:14.731654 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.731616    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"]
Oct 30 01:25:14.812942 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.812884    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a16443ab-4a60-4682-bcc3-7236f4ab7965-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5\" (UID: \"a16443ab-4a60-4682-bcc3-7236f4ab7965\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:25:14.813235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.812953    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbcqm\" (UniqueName: \"kubernetes.io/projected/a16443ab-4a60-4682-bcc3-7236f4ab7965-kube-api-access-cbcqm\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5\" (UID: \"a16443ab-4a60-4682-bcc3-7236f4ab7965\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:25:14.813405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.813375    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a16443ab-4a60-4682-bcc3-7236f4ab7965-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5\" (UID: \"a16443ab-4a60-4682-bcc3-7236f4ab7965\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:25:14.813489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.813447    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a16443ab-4a60-4682-bcc3-7236f4ab7965-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5\" (UID: \"a16443ab-4a60-4682-bcc3-7236f4ab7965\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:25:14.816430 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.816398    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"payload-processing-external-model-plugins-e2e-extmod-b-94ea1e\""
Oct 30 01:25:14.816669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.816461    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"agc-cred-13675c2fb47c\""
Oct 30 01:25:14.816899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.816874    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"kube-root-ca.crt\""
Oct 30 01:25:14.817041 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.816930    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"openshift-service-ca.crt\""
Oct 30 01:25:14.915040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.914987    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a16443ab-4a60-4682-bcc3-7236f4ab7965-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5\" (UID: \"a16443ab-4a60-4682-bcc3-7236f4ab7965\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:25:14.915270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.915068    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a16443ab-4a60-4682-bcc3-7236f4ab7965-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5\" (UID: \"a16443ab-4a60-4682-bcc3-7236f4ab7965\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:25:14.915270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.915243    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a16443ab-4a60-4682-bcc3-7236f4ab7965-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5\" (UID: \"a16443ab-4a60-4682-bcc3-7236f4ab7965\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:25:14.915379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.915276    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbcqm\" (UniqueName: \"kubernetes.io/projected/a16443ab-4a60-4682-bcc3-7236f4ab7965-kube-api-access-cbcqm\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5\" (UID: \"a16443ab-4a60-4682-bcc3-7236f4ab7965\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:25:14.915920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.915895    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a16443ab-4a60-4682-bcc3-7236f4ab7965-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5\" (UID: \"a16443ab-4a60-4682-bcc3-7236f4ab7965\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:25:14.916045 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.915927    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a16443ab-4a60-4682-bcc3-7236f4ab7965-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5\" (UID: \"a16443ab-4a60-4682-bcc3-7236f4ab7965\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:25:14.917884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.917861    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a16443ab-4a60-4682-bcc3-7236f4ab7965-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5\" (UID: \"a16443ab-4a60-4682-bcc3-7236f4ab7965\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:25:14.936922 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:14.936880    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbcqm\" (UniqueName: \"kubernetes.io/projected/a16443ab-4a60-4682-bcc3-7236f4ab7965-kube-api-access-cbcqm\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5\" (UID: \"a16443ab-4a60-4682-bcc3-7236f4ab7965\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:25:15.020104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:15.020049    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:25:15.333043 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:15.333010    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"]
Oct 30 01:25:15.333901 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:25:15.333875    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16443ab_4a60_4682_bcc3_7236f4ab7965.slice/crio-e0835ba6a500da88f7d94cbe335250a3848e879deb5674ecdb57dd9c8e9f99b2 WatchSource:0}: Error finding container e0835ba6a500da88f7d94cbe335250a3848e879deb5674ecdb57dd9c8e9f99b2: Status 404 returned error can't find the container with id e0835ba6a500da88f7d94cbe335250a3848e879deb5674ecdb57dd9c8e9f99b2
Oct 30 01:25:16.321972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:16.321929    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5" event={"ID":"a16443ab-4a60-4682-bcc3-7236f4ab7965","Type":"ContainerStarted","Data":"0183f77930a8ce4e0beef3179c0aa0976efb1182a31467d2f030087af46452bd"}
Oct 30 01:25:16.322436 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:16.322014    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5" event={"ID":"a16443ab-4a60-4682-bcc3-7236f4ab7965","Type":"ContainerStarted","Data":"e0835ba6a500da88f7d94cbe335250a3848e879deb5674ecdb57dd9c8e9f99b2"}
Oct 30 01:25:16.322436 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:16.322172    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:25:16.360461 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:16.360403    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5" podStartSLOduration=2.3603860340000002 podStartE2EDuration="2.360386034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:16.357497325 +0000 UTC m=+2625.107152636" watchObservedRunningTime="2026-10-30 01:25:16.360386034 +0000 UTC m=+2625.110041343"
Oct 30 01:25:23.030462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:23.030418    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"]
Oct 30 01:25:23.039784 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:23.039751    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bff1fb5-q8fl6"]
Oct 30 01:25:23.425680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:23.425586    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88bd9909-1499-4bc5-84f4-f46c4a9a722c" path="/var/lib/kubelet/pods/88bd9909-1499-4bc5-84f4-f46c4a9a722c/volumes"
Oct 30 01:25:25.316289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:25.316258    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"
Oct 30 01:25:25.316712 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:25.316328    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"
Oct 30 01:25:27.333105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:27.333072    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:25:27.391168 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:27.391110    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:25:30.917765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:30.917672    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"]
Oct 30 01:25:30.922417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:30.922392    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"
Oct 30 01:25:30.946586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:30.946548    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:25:30.946586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:30.946548    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:25:30.951125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:30.951086    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"]
Oct 30 01:25:31.040683 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:31.040637    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxb7v\" (UniqueName: \"kubernetes.io/projected/d6a04a93-2a01-43a4-b392-f3779c455217-kube-api-access-mxb7v\") pod \"maas-api-revoke-sub-a770f91a-kxgw7\" (UID: \"d6a04a93-2a01-43a4-b392-f3779c455217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"
Oct 30 01:25:31.040683 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:31.040685    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a04a93-2a01-43a4-b392-f3779c455217-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a770f91a-kxgw7\" (UID: \"d6a04a93-2a01-43a4-b392-f3779c455217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"
Oct 30 01:25:31.040977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:31.040704    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a04a93-2a01-43a4-b392-f3779c455217-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a770f91a-kxgw7\" (UID: \"d6a04a93-2a01-43a4-b392-f3779c455217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"
Oct 30 01:25:31.142767 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:31.142722    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxb7v\" (UniqueName: \"kubernetes.io/projected/d6a04a93-2a01-43a4-b392-f3779c455217-kube-api-access-mxb7v\") pod \"maas-api-revoke-sub-a770f91a-kxgw7\" (UID: \"d6a04a93-2a01-43a4-b392-f3779c455217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"
Oct 30 01:25:31.142767 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:31.142766    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a04a93-2a01-43a4-b392-f3779c455217-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a770f91a-kxgw7\" (UID: \"d6a04a93-2a01-43a4-b392-f3779c455217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"
Oct 30 01:25:31.143056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:31.142801    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a04a93-2a01-43a4-b392-f3779c455217-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a770f91a-kxgw7\" (UID: \"d6a04a93-2a01-43a4-b392-f3779c455217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"
Oct 30 01:25:31.143606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:31.143577    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a04a93-2a01-43a4-b392-f3779c455217-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a770f91a-kxgw7\" (UID: \"d6a04a93-2a01-43a4-b392-f3779c455217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"
Oct 30 01:25:31.145403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:31.145384    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a04a93-2a01-43a4-b392-f3779c455217-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a770f91a-kxgw7\" (UID: \"d6a04a93-2a01-43a4-b392-f3779c455217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"
Oct 30 01:25:31.163869 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:31.163834    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxb7v\" (UniqueName: \"kubernetes.io/projected/d6a04a93-2a01-43a4-b392-f3779c455217-kube-api-access-mxb7v\") pod \"maas-api-revoke-sub-a770f91a-kxgw7\" (UID: \"d6a04a93-2a01-43a4-b392-f3779c455217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"
Oct 30 01:25:31.238523 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:31.238424    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"
Oct 30 01:25:31.553693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:31.553660    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"]
Oct 30 01:25:31.554451 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:25:31.554423    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a04a93_2a01_43a4_b392_f3779c455217.slice/crio-84a1815e9bb1b526b41bf00bbbefc35046c761cfb1f20e224b4ea0e9aeeb6eab WatchSource:0}: Error finding container 84a1815e9bb1b526b41bf00bbbefc35046c761cfb1f20e224b4ea0e9aeeb6eab: Status 404 returned error can't find the container with id 84a1815e9bb1b526b41bf00bbbefc35046c761cfb1f20e224b4ea0e9aeeb6eab
Oct 30 01:25:32.453329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:32.453283    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7" event={"ID":"d6a04a93-2a01-43a4-b392-f3779c455217","Type":"ContainerStarted","Data":"5e4ed70b47ba1ecdfd19bb942f24eb9aef5364871385b73d617b01d974b69a25"}
Oct 30 01:25:32.453870 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:32.453358    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7" event={"ID":"d6a04a93-2a01-43a4-b392-f3779c455217","Type":"ContainerStarted","Data":"84a1815e9bb1b526b41bf00bbbefc35046c761cfb1f20e224b4ea0e9aeeb6eab"}
Oct 30 01:25:32.494871 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:32.494808    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7" podStartSLOduration=2.494788342 podStartE2EDuration="2.494788342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:32.488398121 +0000 UTC m=+2641.238053431" watchObservedRunningTime="2026-10-30 01:25:32.494788342 +0000 UTC m=+2641.244443652"
Oct 30 01:25:33.472835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:33.472796    4425 generic.go:356] "Generic (PLEG): container finished" podID="d6a04a93-2a01-43a4-b392-f3779c455217" containerID="5e4ed70b47ba1ecdfd19bb942f24eb9aef5364871385b73d617b01d974b69a25" exitCode=0
Oct 30 01:25:33.473372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:33.472987    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7" event={"ID":"d6a04a93-2a01-43a4-b392-f3779c455217","Type":"ContainerDied","Data":"5e4ed70b47ba1ecdfd19bb942f24eb9aef5364871385b73d617b01d974b69a25"}
Oct 30 01:25:34.032643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:34.032598    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"]
Oct 30 01:25:34.042174 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:34.042123    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fa45cdd-rxkth"]
Oct 30 01:25:34.645909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:34.645883    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"
Oct 30 01:25:34.791795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:34.791753    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6a04a93-2a01-43a4-b392-f3779c455217-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a04a93-2a01-43a4-b392-f3779c455217-maas-api-openshift-ca\") pod \"d6a04a93-2a01-43a4-b392-f3779c455217\" (UID: \"d6a04a93-2a01-43a4-b392-f3779c455217\") "
Oct 30 01:25:34.792023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:34.791932    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a04a93-2a01-43a4-b392-f3779c455217-kube-api-access-mxb7v\" (UniqueName: \"kubernetes.io/projected/d6a04a93-2a01-43a4-b392-f3779c455217-kube-api-access-mxb7v\") pod \"d6a04a93-2a01-43a4-b392-f3779c455217\" (UID: \"d6a04a93-2a01-43a4-b392-f3779c455217\") "
Oct 30 01:25:34.792023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:34.791971    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6a04a93-2a01-43a4-b392-f3779c455217-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a04a93-2a01-43a4-b392-f3779c455217-maas-api-xks-ca\") pod \"d6a04a93-2a01-43a4-b392-f3779c455217\" (UID: \"d6a04a93-2a01-43a4-b392-f3779c455217\") "
Oct 30 01:25:34.792203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:34.792150    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a04a93-2a01-43a4-b392-f3779c455217-maas-api-openshift-ca" pod "d6a04a93-2a01-43a4-b392-f3779c455217" (UID: "d6a04a93-2a01-43a4-b392-f3779c455217"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:34.792535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:34.792511    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a04a93-2a01-43a4-b392-f3779c455217-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:25:34.794199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:34.794171    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6a04a93-2a01-43a4-b392-f3779c455217-maas-api-xks-ca" pod "d6a04a93-2a01-43a4-b392-f3779c455217" (UID: "d6a04a93-2a01-43a4-b392-f3779c455217"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:34.794302 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:34.794209    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a04a93-2a01-43a4-b392-f3779c455217-kube-api-access-mxb7v" pod "d6a04a93-2a01-43a4-b392-f3779c455217" (UID: "d6a04a93-2a01-43a4-b392-f3779c455217"). InnerVolumeSpecName "kube-api-access-mxb7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:34.894002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:34.893960    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxb7v\" (UniqueName: \"kubernetes.io/projected/d6a04a93-2a01-43a4-b392-f3779c455217-kube-api-access-mxb7v\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:25:34.894002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:34.893996    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a04a93-2a01-43a4-b392-f3779c455217-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:25:35.424398 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:35.424341    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8073178c-d929-45e4-8dc8-8fd7de27b041" path="/var/lib/kubelet/pods/8073178c-d929-45e4-8dc8-8fd7de27b041/volumes"
Oct 30 01:25:35.489371 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:35.489326    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7" event={"ID":"d6a04a93-2a01-43a4-b392-f3779c455217","Type":"ContainerDied","Data":"84a1815e9bb1b526b41bf00bbbefc35046c761cfb1f20e224b4ea0e9aeeb6eab"}
Oct 30 01:25:35.489371 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:35.489348    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"
Oct 30 01:25:35.489669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:35.489387    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84a1815e9bb1b526b41bf00bbbefc35046c761cfb1f20e224b4ea0e9aeeb6eab"
Oct 30 01:25:37.558550 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:25:37.558507    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:25:42.592732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:42.592689    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"]
Oct 30 01:25:46.112675 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:46.112581    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"]
Oct 30 01:25:46.121454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:46.121385    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ccf6204-67hfx"]
Oct 30 01:25:47.424934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:47.424895    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe0ebf75-c6a7-40e4-b749-1b0fda28a42b" path="/var/lib/kubelet/pods/fe0ebf75-c6a7-40e4-b749-1b0fda28a42b/volumes"
Oct 30 01:25:47.593703 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:47.593642    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x" podUID="3254932d-a86a-408b-8567-869cc43f80ba" containerName="payload-processing" containerID="cri-o://d7b591ce02280388e4be03e372e9d3bba0bbcc16613a39ae814b594fa63f15cc" gracePeriod=25
Oct 30 01:25:47.908312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:47.908283    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:25:47.965815 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:47.965709    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3254932d-a86a-408b-8567-869cc43f80ba-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3254932d-a86a-408b-8567-869cc43f80ba-provider-credentials\") pod \"3254932d-a86a-408b-8567-869cc43f80ba\" (UID: \"3254932d-a86a-408b-8567-869cc43f80ba\") "
Oct 30 01:25:47.965815 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:47.965783    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3254932d-a86a-408b-8567-869cc43f80ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3254932d-a86a-408b-8567-869cc43f80ba-plugins-config-volume\") pod \"3254932d-a86a-408b-8567-869cc43f80ba\" (UID: \"3254932d-a86a-408b-8567-869cc43f80ba\") "
Oct 30 01:25:47.966092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:47.965880    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3254932d-a86a-408b-8567-869cc43f80ba-kube-api-access-7sscb\" (UniqueName: \"kubernetes.io/projected/3254932d-a86a-408b-8567-869cc43f80ba-kube-api-access-7sscb\") pod \"3254932d-a86a-408b-8567-869cc43f80ba\" (UID: \"3254932d-a86a-408b-8567-869cc43f80ba\") "
Oct 30 01:25:47.966179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:47.966093    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3254932d-a86a-408b-8567-869cc43f80ba-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3254932d-a86a-408b-8567-869cc43f80ba-routing-overlay-volume\") pod \"3254932d-a86a-408b-8567-869cc43f80ba\" (UID: \"3254932d-a86a-408b-8567-869cc43f80ba\") "
Oct 30 01:25:47.966320 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:47.966287    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3254932d-a86a-408b-8567-869cc43f80ba-plugins-config-volume" pod "3254932d-a86a-408b-8567-869cc43f80ba" (UID: "3254932d-a86a-408b-8567-869cc43f80ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:47.966648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:47.966620    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3254932d-a86a-408b-8567-869cc43f80ba-routing-overlay-volume" pod "3254932d-a86a-408b-8567-869cc43f80ba" (UID: "3254932d-a86a-408b-8567-869cc43f80ba"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:25:47.966801 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:47.966780    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3254932d-a86a-408b-8567-869cc43f80ba-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:25:47.966844 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:47.966809    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3254932d-a86a-408b-8567-869cc43f80ba-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:25:47.968921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:47.968889    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3254932d-a86a-408b-8567-869cc43f80ba-provider-credentials" pod "3254932d-a86a-408b-8567-869cc43f80ba" (UID: "3254932d-a86a-408b-8567-869cc43f80ba"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:47.968921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:47.968911    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3254932d-a86a-408b-8567-869cc43f80ba-kube-api-access-7sscb" pod "3254932d-a86a-408b-8567-869cc43f80ba" (UID: "3254932d-a86a-408b-8567-869cc43f80ba"). InnerVolumeSpecName "kube-api-access-7sscb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:48.068775 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:48.068731    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sscb\" (UniqueName: \"kubernetes.io/projected/3254932d-a86a-408b-8567-869cc43f80ba-kube-api-access-7sscb\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:25:48.068775 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:48.068766    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3254932d-a86a-408b-8567-869cc43f80ba-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:25:48.597479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:48.597437    4425 generic.go:356] "Generic (PLEG): container finished" podID="3254932d-a86a-408b-8567-869cc43f80ba" containerID="d7b591ce02280388e4be03e372e9d3bba0bbcc16613a39ae814b594fa63f15cc" exitCode=0
Oct 30 01:25:48.598002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:48.597482    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x" event={"ID":"3254932d-a86a-408b-8567-869cc43f80ba","Type":"ContainerDied","Data":"d7b591ce02280388e4be03e372e9d3bba0bbcc16613a39ae814b594fa63f15cc"}
Oct 30 01:25:48.598002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:48.597533    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x" event={"ID":"3254932d-a86a-408b-8567-869cc43f80ba","Type":"ContainerDied","Data":"12e5afaa8299e6ef7b27f18dee8b4d59592f1d51612763147af7e67c60cc88a3"}
Oct 30 01:25:48.598002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:48.597563    4425 scope.go:122] "RemoveContainer" containerID="d7b591ce02280388e4be03e372e9d3bba0bbcc16613a39ae814b594fa63f15cc"
Oct 30 01:25:48.598002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:48.597493    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"
Oct 30 01:25:48.609981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:48.609961    4425 scope.go:122] "RemoveContainer" containerID="d7b591ce02280388e4be03e372e9d3bba0bbcc16613a39ae814b594fa63f15cc"
Oct 30 01:25:48.610432 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:25:48.610405    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b591ce02280388e4be03e372e9d3bba0bbcc16613a39ae814b594fa63f15cc\": container with ID starting with d7b591ce02280388e4be03e372e9d3bba0bbcc16613a39ae814b594fa63f15cc not found: ID does not exist" containerID="d7b591ce02280388e4be03e372e9d3bba0bbcc16613a39ae814b594fa63f15cc"
Oct 30 01:25:48.610545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:48.610444    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b591ce02280388e4be03e372e9d3bba0bbcc16613a39ae814b594fa63f15cc"} err="failed to get container status \"d7b591ce02280388e4be03e372e9d3bba0bbcc16613a39ae814b594fa63f15cc\": rpc error: code = NotFound desc = could not find container \"d7b591ce02280388e4be03e372e9d3bba0bbcc16613a39ae814b594fa63f15cc\": container with ID starting with d7b591ce02280388e4be03e372e9d3bba0bbcc16613a39ae814b594fa63f15cc not found: ID does not exist"
Oct 30 01:25:48.644589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:48.644545    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"]
Oct 30 01:25:48.654071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:48.654033    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-5799qwr7x"]
Oct 30 01:25:49.424476 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:49.424443    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3254932d-a86a-408b-8567-869cc43f80ba" path="/var/lib/kubelet/pods/3254932d-a86a-408b-8567-869cc43f80ba/volumes"
Oct 30 01:25:51.031227 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:51.031185    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"]
Oct 30 01:25:51.040659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:51.040621    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c947ff9-96krr"]
Oct 30 01:25:51.424853 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:51.424762    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc468dd-b7fe-47e1-bc25-69e61924f729" path="/var/lib/kubelet/pods/6fc468dd-b7fe-47e1-bc25-69e61924f729/volumes"
Oct 30 01:25:53.028586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.028546    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"]
Oct 30 01:25:53.029434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.029405    4425 cpu_manager.go:414] "containerMap: removing container" podUID="3254932d-a86a-408b-8567-869cc43f80ba" containerName="payload-processing"
Oct 30 01:25:53.029434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.029432    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3254932d-a86a-408b-8567-869cc43f80ba" containerName="payload-processing"
Oct 30 01:25:53.029619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.029458    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d6a04a93-2a01-43a4-b392-f3779c455217" containerName="revoke-keys"
Oct 30 01:25:53.029619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.029463    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a04a93-2a01-43a4-b392-f3779c455217" containerName="revoke-keys"
Oct 30 01:25:53.029619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.029605    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3254932d-a86a-408b-8567-869cc43f80ba" containerName="payload-processing"
Oct 30 01:25:53.029619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.029616    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a04a93-2a01-43a4-b392-f3779c455217" containerName="revoke-keys"
Oct 30 01:25:53.035049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.035015    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:25:53.059750 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.059708    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-1cb829-serving-cert\""
Oct 30 01:25:53.064537 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.064495    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"]
Oct 30 01:25:53.147634 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.147599    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-metrics-certs\") pod \"maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz\" (UID: \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:25:53.147851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.147761    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz\" (UID: \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:25:53.147851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.147826    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz\" (UID: \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:25:53.147935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.147901    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25bj7\" (UniqueName: \"kubernetes.io/projected/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-kube-api-access-25bj7\") pod \"maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz\" (UID: \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:25:53.250196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.250129    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz\" (UID: \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:25:53.250377 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.250267    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz\" (UID: \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:25:53.250377 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.250304    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25bj7\" (UniqueName: \"kubernetes.io/projected/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-kube-api-access-25bj7\") pod \"maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz\" (UID: \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:25:53.250522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.250432    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-metrics-certs\") pod \"maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz\" (UID: \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:25:53.251240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.251151    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz\" (UID: \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:25:53.253313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.253281    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz\" (UID: \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:25:53.253634 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.253613    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-metrics-certs\") pod \"maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz\" (UID: \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:25:53.272457 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.272411    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25bj7\" (UniqueName: \"kubernetes.io/projected/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-kube-api-access-25bj7\") pod \"maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz\" (UID: \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:25:53.351211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.351098    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:25:53.666229 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:53.666190    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"]
Oct 30 01:25:53.667032 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:25:53.667008    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcc68ce_dd94_4b15_ac6a_fa179b89a3ce.slice/crio-e7d904676f50adad3b7378855e8ad89dd7aa68a226f304684f20561be8dfd6ba WatchSource:0}: Error finding container e7d904676f50adad3b7378855e8ad89dd7aa68a226f304684f20561be8dfd6ba: Status 404 returned error can't find the container with id e7d904676f50adad3b7378855e8ad89dd7aa68a226f304684f20561be8dfd6ba
Oct 30 01:25:54.648250 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:54.648206    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz" event={"ID":"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce","Type":"ContainerStarted","Data":"5c5ac4cef5bcf288c556bd7fdb594f4c70dad7d4e99d8aa98ac23d556264afc9"}
Oct 30 01:25:54.648704 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:54.648287    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:25:54.648704 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:54.648313    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz" event={"ID":"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce","Type":"ContainerStarted","Data":"e7d904676f50adad3b7378855e8ad89dd7aa68a226f304684f20561be8dfd6ba"}
Oct 30 01:25:54.686094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:54.686029    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz" podStartSLOduration=1.30066271 podStartE2EDuration="1.68601532s" totalImagesPullingTime="385.35261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:54.685689575 +0000 UTC m=+2663.435344884" watchObservedRunningTime="2026-10-30 01:25:54.68601532 +0000 UTC m=+2663.435670629"
Oct 30 01:25:57.041192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.041122    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"]
Oct 30 01:25:57.045730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.045703    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04ed4ee2-j56t4"]
Oct 30 01:25:57.424517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.424424    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca312417-9947-410e-b0da-f8f703a63272" path="/var/lib/kubelet/pods/ca312417-9947-410e-b0da-f8f703a63272/volumes"
Oct 30 01:25:57.681761 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.681679    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"]
Oct 30 01:25:57.686914 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.686885    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:25:57.700398 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.700374    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-499482\"/\"e2e-cmp-legacy-179ad8-provider-cred\""
Oct 30 01:25:57.707739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.707706    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"]
Oct 30 01:25:57.710353 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.710326    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-499482\"/\"openshift-service-ca.crt\""
Oct 30 01:25:57.710536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.710387    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-499482\"/\"payload-processing-external-model-plugins-e2e-extmod-a-499482\""
Oct 30 01:25:57.712385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.712354    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcj6s\" (UniqueName: \"kubernetes.io/projected/adf5721c-1a9f-4360-b164-ad125d052f17-kube-api-access-dcj6s\") pod \"payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk\" (UID: \"adf5721c-1a9f-4360-b164-ad125d052f17\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:25:57.712507 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.712462    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf5721c-1a9f-4360-b164-ad125d052f17-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk\" (UID: \"adf5721c-1a9f-4360-b164-ad125d052f17\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:25:57.712628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.712613    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/adf5721c-1a9f-4360-b164-ad125d052f17-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk\" (UID: \"adf5721c-1a9f-4360-b164-ad125d052f17\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:25:57.712670 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.712637    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adf5721c-1a9f-4360-b164-ad125d052f17-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk\" (UID: \"adf5721c-1a9f-4360-b164-ad125d052f17\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:25:57.713041 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.713026    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-499482\"/\"routing-overlay\""
Oct 30 01:25:57.713114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.713097    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-499482\"/\"kube-root-ca.crt\""
Oct 30 01:25:57.800739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.800706    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-499482\"/\"payload-processing-external-model-e2e--b4b1dd99-dockercfg-cqn9r\""
Oct 30 01:25:57.814172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.814122    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/adf5721c-1a9f-4360-b164-ad125d052f17-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk\" (UID: \"adf5721c-1a9f-4360-b164-ad125d052f17\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:25:57.814389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.814175    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adf5721c-1a9f-4360-b164-ad125d052f17-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk\" (UID: \"adf5721c-1a9f-4360-b164-ad125d052f17\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:25:57.814389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.814344    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcj6s\" (UniqueName: \"kubernetes.io/projected/adf5721c-1a9f-4360-b164-ad125d052f17-kube-api-access-dcj6s\") pod \"payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk\" (UID: \"adf5721c-1a9f-4360-b164-ad125d052f17\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:25:57.814389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.814373    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf5721c-1a9f-4360-b164-ad125d052f17-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk\" (UID: \"adf5721c-1a9f-4360-b164-ad125d052f17\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:25:57.815223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.815192    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adf5721c-1a9f-4360-b164-ad125d052f17-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk\" (UID: \"adf5721c-1a9f-4360-b164-ad125d052f17\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:25:57.815341 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.815201    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf5721c-1a9f-4360-b164-ad125d052f17-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk\" (UID: \"adf5721c-1a9f-4360-b164-ad125d052f17\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:25:57.816973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.816950    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/adf5721c-1a9f-4360-b164-ad125d052f17-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk\" (UID: \"adf5721c-1a9f-4360-b164-ad125d052f17\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:25:57.837887 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:57.837842    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcj6s\" (UniqueName: \"kubernetes.io/projected/adf5721c-1a9f-4360-b164-ad125d052f17-kube-api-access-dcj6s\") pod \"payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk\" (UID: \"adf5721c-1a9f-4360-b164-ad125d052f17\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:25:58.004217 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:58.004107    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:25:58.326625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:58.326596    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"]
Oct 30 01:25:58.328731 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:25:58.328692    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf5721c_1a9f_4360_b164_ad125d052f17.slice/crio-e7a0b822ba5dcc81b04f6057bac02922b1805d0e51597b0a72abef28ca7ef80d WatchSource:0}: Error finding container e7a0b822ba5dcc81b04f6057bac02922b1805d0e51597b0a72abef28ca7ef80d: Status 404 returned error can't find the container with id e7a0b822ba5dcc81b04f6057bac02922b1805d0e51597b0a72abef28ca7ef80d
Oct 30 01:25:58.702886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:58.702778    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk" event={"ID":"adf5721c-1a9f-4360-b164-ad125d052f17","Type":"ContainerStarted","Data":"499b145c7eb88d4a1668284f948a75755110b4e4889d2a6934a621f1f24a2e7f"}
Oct 30 01:25:58.702886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:58.702837    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk" event={"ID":"adf5721c-1a9f-4360-b164-ad125d052f17","Type":"ContainerStarted","Data":"e7a0b822ba5dcc81b04f6057bac02922b1805d0e51597b0a72abef28ca7ef80d"}
Oct 30 01:25:58.702886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:58.702881    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:25:58.741529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:25:58.741462    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk" podStartSLOduration=1.7414404719999999 podStartE2EDuration="1.741440472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:58.73855891 +0000 UTC m=+2667.488214220" watchObservedRunningTime="2026-10-30 01:25:58.741440472 +0000 UTC m=+2667.491095794"
Oct 30 01:26:00.670858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:00.670810    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:26:01.042886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.042847    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"]
Oct 30 01:26:01.047948 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.047911    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e3df17c-zlxnn"]
Oct 30 01:26:01.424773 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.424688    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a650e14b-1236-47e8-b903-fdc7dd48c486" path="/var/lib/kubelet/pods/a650e14b-1236-47e8-b903-fdc7dd48c486/volumes"
Oct 30 01:26:01.637377 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.637336    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"]
Oct 30 01:26:01.642510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.642487    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:26:01.661124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.661091    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1afb3e\"/\"agc-cred-525254b7241a\""
Oct 30 01:26:01.665909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.665869    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"]
Oct 30 01:26:01.677132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.677046    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x245f\" (UniqueName: \"kubernetes.io/projected/2a78699a-583f-4438-a5f4-ce704e517df3-kube-api-access-x245f\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt\" (UID: \"2a78699a-583f-4438-a5f4-ce704e517df3\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:26:01.677132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.677107    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a78699a-583f-4438-a5f4-ce704e517df3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt\" (UID: \"2a78699a-583f-4438-a5f4-ce704e517df3\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:26:01.677996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.677337    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a78699a-583f-4438-a5f4-ce704e517df3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt\" (UID: \"2a78699a-583f-4438-a5f4-ce704e517df3\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:26:01.677996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.677487    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a78699a-583f-4438-a5f4-ce704e517df3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt\" (UID: \"2a78699a-583f-4438-a5f4-ce704e517df3\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:26:01.779166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.779126    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a78699a-583f-4438-a5f4-ce704e517df3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt\" (UID: \"2a78699a-583f-4438-a5f4-ce704e517df3\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:26:01.779383 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.779278    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x245f\" (UniqueName: \"kubernetes.io/projected/2a78699a-583f-4438-a5f4-ce704e517df3-kube-api-access-x245f\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt\" (UID: \"2a78699a-583f-4438-a5f4-ce704e517df3\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:26:01.779383 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.779332    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a78699a-583f-4438-a5f4-ce704e517df3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt\" (UID: \"2a78699a-583f-4438-a5f4-ce704e517df3\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:26:01.779530 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.779513    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a78699a-583f-4438-a5f4-ce704e517df3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt\" (UID: \"2a78699a-583f-4438-a5f4-ce704e517df3\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:26:01.780350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.780293    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a78699a-583f-4438-a5f4-ce704e517df3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt\" (UID: \"2a78699a-583f-4438-a5f4-ce704e517df3\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:26:01.780487 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.780450    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a78699a-583f-4438-a5f4-ce704e517df3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt\" (UID: \"2a78699a-583f-4438-a5f4-ce704e517df3\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:26:01.782122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.782095    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a78699a-583f-4438-a5f4-ce704e517df3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt\" (UID: \"2a78699a-583f-4438-a5f4-ce704e517df3\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:26:01.800827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.800786    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x245f\" (UniqueName: \"kubernetes.io/projected/2a78699a-583f-4438-a5f4-ce704e517df3-kube-api-access-x245f\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt\" (UID: \"2a78699a-583f-4438-a5f4-ce704e517df3\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:26:01.959346 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:01.959235    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:26:02.269660 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.269627    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"]
Oct 30 01:26:02.271337 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:26:02.271306    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a78699a_583f_4438_a5f4_ce704e517df3.slice/crio-20a3a734cc638d8252762afae17043cc916654d1f6472aa0148496f2699174ed WatchSource:0}: Error finding container 20a3a734cc638d8252762afae17043cc916654d1f6472aa0148496f2699174ed: Status 404 returned error can't find the container with id 20a3a734cc638d8252762afae17043cc916654d1f6472aa0148496f2699174ed
Oct 30 01:26:02.693248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.693139    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"]
Oct 30 01:26:02.698024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.697997    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:02.710408 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.710378    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"routing-overlay\""
Oct 30 01:26:02.713283 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.713262    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"agc-cred-25d1cd39e6b0\""
Oct 30 01:26:02.718086 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.718044    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"]
Oct 30 01:26:02.720348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.720320    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"payload-processing-external-model-e2e--59030ca4-dockercfg-jwrp9\""
Oct 30 01:26:02.747229 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.747189    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt" event={"ID":"2a78699a-583f-4438-a5f4-ce704e517df3","Type":"ContainerStarted","Data":"c695b153c9c7f536d4f8d531cc5825bab1a4e110f944163256af91c72f3039f9"}
Oct 30 01:26:02.747424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.747252    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt" event={"ID":"2a78699a-583f-4438-a5f4-ce704e517df3","Type":"ContainerStarted","Data":"20a3a734cc638d8252762afae17043cc916654d1f6472aa0148496f2699174ed"}
Oct 30 01:26:02.747424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.747302    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:26:02.785749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.785676    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt" podStartSLOduration=1.785655694 podStartE2EDuration="1.785655694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:02.78301036 +0000 UTC m=+2671.532665663" watchObservedRunningTime="2026-10-30 01:26:02.785655694 +0000 UTC m=+2671.535311003"
Oct 30 01:26:02.798427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.798291    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8915869a-a352-4b75-a761-7462858e6ce7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv\" (UID: \"8915869a-a352-4b75-a761-7462858e6ce7\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:02.799213 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.799175    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crj4r\" (UniqueName: \"kubernetes.io/projected/8915869a-a352-4b75-a761-7462858e6ce7-kube-api-access-crj4r\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv\" (UID: \"8915869a-a352-4b75-a761-7462858e6ce7\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:02.799364 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.799259    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8915869a-a352-4b75-a761-7462858e6ce7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv\" (UID: \"8915869a-a352-4b75-a761-7462858e6ce7\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:02.799424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.799384    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8915869a-a352-4b75-a761-7462858e6ce7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv\" (UID: \"8915869a-a352-4b75-a761-7462858e6ce7\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:02.810780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.810733    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2f820e\""
Oct 30 01:26:02.810780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.810733    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"openshift-service-ca.crt\""
Oct 30 01:26:02.810780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.810750    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"kube-root-ca.crt\""
Oct 30 01:26:02.901123 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.901075    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8915869a-a352-4b75-a761-7462858e6ce7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv\" (UID: \"8915869a-a352-4b75-a761-7462858e6ce7\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:02.901340 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.901292    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crj4r\" (UniqueName: \"kubernetes.io/projected/8915869a-a352-4b75-a761-7462858e6ce7-kube-api-access-crj4r\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv\" (UID: \"8915869a-a352-4b75-a761-7462858e6ce7\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:02.901340 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.901333    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8915869a-a352-4b75-a761-7462858e6ce7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv\" (UID: \"8915869a-a352-4b75-a761-7462858e6ce7\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:02.902772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.902743    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8915869a-a352-4b75-a761-7462858e6ce7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv\" (UID: \"8915869a-a352-4b75-a761-7462858e6ce7\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:02.903200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.901393    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8915869a-a352-4b75-a761-7462858e6ce7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv\" (UID: \"8915869a-a352-4b75-a761-7462858e6ce7\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:02.903923 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.903894    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8915869a-a352-4b75-a761-7462858e6ce7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv\" (UID: \"8915869a-a352-4b75-a761-7462858e6ce7\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:02.910935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.910889    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8915869a-a352-4b75-a761-7462858e6ce7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv\" (UID: \"8915869a-a352-4b75-a761-7462858e6ce7\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:02.928548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:02.928476    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crj4r\" (UniqueName: \"kubernetes.io/projected/8915869a-a352-4b75-a761-7462858e6ce7-kube-api-access-crj4r\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv\" (UID: \"8915869a-a352-4b75-a761-7462858e6ce7\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:03.013127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:03.013084    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:03.327414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:03.327380    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"]
Oct 30 01:26:03.329000 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:26:03.328965    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8915869a_a352_4b75_a761_7462858e6ce7.slice/crio-d20344ebffd30415f3211c8d819c8087a7a9db2c63c51898c58bd878e8854783 WatchSource:0}: Error finding container d20344ebffd30415f3211c8d819c8087a7a9db2c63c51898c58bd878e8854783: Status 404 returned error can't find the container with id d20344ebffd30415f3211c8d819c8087a7a9db2c63c51898c58bd878e8854783
Oct 30 01:26:03.756497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:03.756398    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv" event={"ID":"8915869a-a352-4b75-a761-7462858e6ce7","Type":"ContainerStarted","Data":"70ea0c5a00d144e9e15a923009db47730b56207a47606206a8706bf8b4b78a6e"}
Oct 30 01:26:03.756497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:03.756459    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv" event={"ID":"8915869a-a352-4b75-a761-7462858e6ce7","Type":"ContainerStarted","Data":"d20344ebffd30415f3211c8d819c8087a7a9db2c63c51898c58bd878e8854783"}
Oct 30 01:26:03.757023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:03.756734    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:03.795838 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:03.795763    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv" podStartSLOduration=1.795739149 podStartE2EDuration="1.795739149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:03.792801697 +0000 UTC m=+2672.542457005" watchObservedRunningTime="2026-10-30 01:26:03.795739149 +0000 UTC m=+2672.545394458"
Oct 30 01:26:07.557236 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:26:07.557189    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:26:09.713900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:09.713787    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:26:13.760239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:13.760209    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:26:13.835623 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:13.835587    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"]
Oct 30 01:26:14.768450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:14.768411    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:18.001890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:18.001848    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"]
Oct 30 01:26:18.836565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:18.836517    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s" podUID="9f921d86-8439-40e7-bb10-3a3f55b7f918" containerName="payload-processing" containerID="cri-o://1e2cff83b7c1b8472b8404eb6f32049dda034df9b9005aebbaa3fb94bffca693" gracePeriod=25
Oct 30 01:26:19.124944 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.124914    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:26:19.221223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.221181    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f921d86-8439-40e7-bb10-3a3f55b7f918-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f921d86-8439-40e7-bb10-3a3f55b7f918-provider-credentials\") pod \"9f921d86-8439-40e7-bb10-3a3f55b7f918\" (UID: \"9f921d86-8439-40e7-bb10-3a3f55b7f918\") "
Oct 30 01:26:19.221431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.221273    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f921d86-8439-40e7-bb10-3a3f55b7f918-kube-api-access-s8zjv\" (UniqueName: \"kubernetes.io/projected/9f921d86-8439-40e7-bb10-3a3f55b7f918-kube-api-access-s8zjv\") pod \"9f921d86-8439-40e7-bb10-3a3f55b7f918\" (UID: \"9f921d86-8439-40e7-bb10-3a3f55b7f918\") "
Oct 30 01:26:19.221482 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.221454    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f921d86-8439-40e7-bb10-3a3f55b7f918-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f921d86-8439-40e7-bb10-3a3f55b7f918-routing-overlay-volume\") pod \"9f921d86-8439-40e7-bb10-3a3f55b7f918\" (UID: \"9f921d86-8439-40e7-bb10-3a3f55b7f918\") "
Oct 30 01:26:19.221643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.221621    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f921d86-8439-40e7-bb10-3a3f55b7f918-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f921d86-8439-40e7-bb10-3a3f55b7f918-plugins-config-volume\") pod \"9f921d86-8439-40e7-bb10-3a3f55b7f918\" (UID: \"9f921d86-8439-40e7-bb10-3a3f55b7f918\") "
Oct 30 01:26:19.221985 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.221950    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f921d86-8439-40e7-bb10-3a3f55b7f918-routing-overlay-volume" pod "9f921d86-8439-40e7-bb10-3a3f55b7f918" (UID: "9f921d86-8439-40e7-bb10-3a3f55b7f918"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:19.222146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.221997    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f921d86-8439-40e7-bb10-3a3f55b7f918-plugins-config-volume" pod "9f921d86-8439-40e7-bb10-3a3f55b7f918" (UID: "9f921d86-8439-40e7-bb10-3a3f55b7f918"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:19.222281 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.222263    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f921d86-8439-40e7-bb10-3a3f55b7f918-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:26:19.222337 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.222288    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f921d86-8439-40e7-bb10-3a3f55b7f918-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:26:19.223810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.223786    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f921d86-8439-40e7-bb10-3a3f55b7f918-kube-api-access-s8zjv" pod "9f921d86-8439-40e7-bb10-3a3f55b7f918" (UID: "9f921d86-8439-40e7-bb10-3a3f55b7f918"). InnerVolumeSpecName "kube-api-access-s8zjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:19.223926 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.223842    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f921d86-8439-40e7-bb10-3a3f55b7f918-provider-credentials" pod "9f921d86-8439-40e7-bb10-3a3f55b7f918" (UID: "9f921d86-8439-40e7-bb10-3a3f55b7f918"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:19.323404 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.323363    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f921d86-8439-40e7-bb10-3a3f55b7f918-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:26:19.323404 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.323398    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8zjv\" (UniqueName: \"kubernetes.io/projected/9f921d86-8439-40e7-bb10-3a3f55b7f918-kube-api-access-s8zjv\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:26:19.888589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.888551    4425 generic.go:356] "Generic (PLEG): container finished" podID="9f921d86-8439-40e7-bb10-3a3f55b7f918" containerID="1e2cff83b7c1b8472b8404eb6f32049dda034df9b9005aebbaa3fb94bffca693" exitCode=0
Oct 30 01:26:19.888786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.888608    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"
Oct 30 01:26:19.888786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.888641    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s" event={"ID":"9f921d86-8439-40e7-bb10-3a3f55b7f918","Type":"ContainerDied","Data":"1e2cff83b7c1b8472b8404eb6f32049dda034df9b9005aebbaa3fb94bffca693"}
Oct 30 01:26:19.888786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.888699    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s" event={"ID":"9f921d86-8439-40e7-bb10-3a3f55b7f918","Type":"ContainerDied","Data":"198a803400ceaf1b712a1d15bff84f56c09c389c5e6fbbad864c4df845f88965"}
Oct 30 01:26:19.888786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.888721    4425 scope.go:122] "RemoveContainer" containerID="1e2cff83b7c1b8472b8404eb6f32049dda034df9b9005aebbaa3fb94bffca693"
Oct 30 01:26:19.899932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.899908    4425 scope.go:122] "RemoveContainer" containerID="1e2cff83b7c1b8472b8404eb6f32049dda034df9b9005aebbaa3fb94bffca693"
Oct 30 01:26:19.900297 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:26:19.900271    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2cff83b7c1b8472b8404eb6f32049dda034df9b9005aebbaa3fb94bffca693\": container with ID starting with 1e2cff83b7c1b8472b8404eb6f32049dda034df9b9005aebbaa3fb94bffca693 not found: ID does not exist" containerID="1e2cff83b7c1b8472b8404eb6f32049dda034df9b9005aebbaa3fb94bffca693"
Oct 30 01:26:19.900406 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.900308    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2cff83b7c1b8472b8404eb6f32049dda034df9b9005aebbaa3fb94bffca693"} err="failed to get container status \"1e2cff83b7c1b8472b8404eb6f32049dda034df9b9005aebbaa3fb94bffca693\": rpc error: code = NotFound desc = could not find container \"1e2cff83b7c1b8472b8404eb6f32049dda034df9b9005aebbaa3fb94bffca693\": container with ID starting with 1e2cff83b7c1b8472b8404eb6f32049dda034df9b9005aebbaa3fb94bffca693 not found: ID does not exist"
Oct 30 01:26:19.935080 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.935037    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"]
Oct 30 01:26:19.944937 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:19.944893    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-74fflvq2s"]
Oct 30 01:26:21.030071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:21.030034    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"]
Oct 30 01:26:21.044410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:21.044372    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f154713c-swxmm"]
Oct 30 01:26:21.424872 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:21.424792    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f921d86-8439-40e7-bb10-3a3f55b7f918" path="/var/lib/kubelet/pods/9f921d86-8439-40e7-bb10-3a3f55b7f918/volumes"
Oct 30 01:26:21.425324 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:21.425309    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c982fa3b-8d27-4200-afcf-ddbe8f098d95" path="/var/lib/kubelet/pods/c982fa3b-8d27-4200-afcf-ddbe8f098d95/volumes"
Oct 30 01:26:23.002716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.002656    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv" podUID="8915869a-a352-4b75-a761-7462858e6ce7" containerName="payload-processing" containerID="cri-o://70ea0c5a00d144e9e15a923009db47730b56207a47606206a8706bf8b4b78a6e" gracePeriod=25
Oct 30 01:26:23.301858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.301826    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:23.380665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.380602    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8915869a-a352-4b75-a761-7462858e6ce7-kube-api-access-crj4r\" (UniqueName: \"kubernetes.io/projected/8915869a-a352-4b75-a761-7462858e6ce7-kube-api-access-crj4r\") pod \"8915869a-a352-4b75-a761-7462858e6ce7\" (UID: \"8915869a-a352-4b75-a761-7462858e6ce7\") "
Oct 30 01:26:23.380864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.380775    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8915869a-a352-4b75-a761-7462858e6ce7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8915869a-a352-4b75-a761-7462858e6ce7-plugins-config-volume\") pod \"8915869a-a352-4b75-a761-7462858e6ce7\" (UID: \"8915869a-a352-4b75-a761-7462858e6ce7\") "
Oct 30 01:26:23.380921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.380902    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8915869a-a352-4b75-a761-7462858e6ce7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8915869a-a352-4b75-a761-7462858e6ce7-provider-credentials\") pod \"8915869a-a352-4b75-a761-7462858e6ce7\" (UID: \"8915869a-a352-4b75-a761-7462858e6ce7\") "
Oct 30 01:26:23.381016 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.380997    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8915869a-a352-4b75-a761-7462858e6ce7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8915869a-a352-4b75-a761-7462858e6ce7-routing-overlay-volume\") pod \"8915869a-a352-4b75-a761-7462858e6ce7\" (UID: \"8915869a-a352-4b75-a761-7462858e6ce7\") "
Oct 30 01:26:23.381430 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.381394    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8915869a-a352-4b75-a761-7462858e6ce7-plugins-config-volume" pod "8915869a-a352-4b75-a761-7462858e6ce7" (UID: "8915869a-a352-4b75-a761-7462858e6ce7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:23.381603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.381435    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8915869a-a352-4b75-a761-7462858e6ce7-routing-overlay-volume" pod "8915869a-a352-4b75-a761-7462858e6ce7" (UID: "8915869a-a352-4b75-a761-7462858e6ce7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:23.381752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.381709    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8915869a-a352-4b75-a761-7462858e6ce7-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:26:23.381752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.381736    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8915869a-a352-4b75-a761-7462858e6ce7-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:26:23.383344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.383309    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8915869a-a352-4b75-a761-7462858e6ce7-kube-api-access-crj4r" pod "8915869a-a352-4b75-a761-7462858e6ce7" (UID: "8915869a-a352-4b75-a761-7462858e6ce7"). InnerVolumeSpecName "kube-api-access-crj4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:23.383344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.383329    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8915869a-a352-4b75-a761-7462858e6ce7-provider-credentials" pod "8915869a-a352-4b75-a761-7462858e6ce7" (UID: "8915869a-a352-4b75-a761-7462858e6ce7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:23.483319 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.483280    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crj4r\" (UniqueName: \"kubernetes.io/projected/8915869a-a352-4b75-a761-7462858e6ce7-kube-api-access-crj4r\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:26:23.483319 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.483311    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8915869a-a352-4b75-a761-7462858e6ce7-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:26:23.924173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.924123    4425 generic.go:356] "Generic (PLEG): container finished" podID="8915869a-a352-4b75-a761-7462858e6ce7" containerID="70ea0c5a00d144e9e15a923009db47730b56207a47606206a8706bf8b4b78a6e" exitCode=0
Oct 30 01:26:23.924373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.924186    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"
Oct 30 01:26:23.924373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.924203    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv" event={"ID":"8915869a-a352-4b75-a761-7462858e6ce7","Type":"ContainerDied","Data":"70ea0c5a00d144e9e15a923009db47730b56207a47606206a8706bf8b4b78a6e"}
Oct 30 01:26:23.924373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.924266    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv" event={"ID":"8915869a-a352-4b75-a761-7462858e6ce7","Type":"ContainerDied","Data":"d20344ebffd30415f3211c8d819c8087a7a9db2c63c51898c58bd878e8854783"}
Oct 30 01:26:23.924373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.924289    4425 scope.go:122] "RemoveContainer" containerID="70ea0c5a00d144e9e15a923009db47730b56207a47606206a8706bf8b4b78a6e"
Oct 30 01:26:23.935587 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.935557    4425 scope.go:122] "RemoveContainer" containerID="70ea0c5a00d144e9e15a923009db47730b56207a47606206a8706bf8b4b78a6e"
Oct 30 01:26:23.935901 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:26:23.935877    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ea0c5a00d144e9e15a923009db47730b56207a47606206a8706bf8b4b78a6e\": container with ID starting with 70ea0c5a00d144e9e15a923009db47730b56207a47606206a8706bf8b4b78a6e not found: ID does not exist" containerID="70ea0c5a00d144e9e15a923009db47730b56207a47606206a8706bf8b4b78a6e"
Oct 30 01:26:23.936019 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.935908    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ea0c5a00d144e9e15a923009db47730b56207a47606206a8706bf8b4b78a6e"} err="failed to get container status \"70ea0c5a00d144e9e15a923009db47730b56207a47606206a8706bf8b4b78a6e\": rpc error: code = NotFound desc = could not find container \"70ea0c5a00d144e9e15a923009db47730b56207a47606206a8706bf8b4b78a6e\": container with ID starting with 70ea0c5a00d144e9e15a923009db47730b56207a47606206a8706bf8b4b78a6e not found: ID does not exist"
Oct 30 01:26:23.970524 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.970490    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"]
Oct 30 01:26:23.979878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:23.979844    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6686qxlkv"]
Oct 30 01:26:25.424745 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:25.424706    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8915869a-a352-4b75-a761-7462858e6ce7" path="/var/lib/kubelet/pods/8915869a-a352-4b75-a761-7462858e6ce7/volumes"
Oct 30 01:26:26.908381 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:26.908343    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:26:32.030807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:32.030766    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"]
Oct 30 01:26:32.039995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:32.039960    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca4c4487-27679"]
Oct 30 01:26:33.426008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:33.425971    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457e9435-2882-4e59-be1d-8bec75b53f1e" path="/var/lib/kubelet/pods/457e9435-2882-4e59-be1d-8bec75b53f1e/volumes"
Oct 30 01:26:34.512212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:34.512177    4425 scope.go:122] "RemoveContainer" containerID="f0fe3bcd9413c6be98a587aae745a6ce69b335d64a377c3267a6d898a7ae8434"
Oct 30 01:26:34.523278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:34.523234    4425 scope.go:122] "RemoveContainer" containerID="8b5b252cd103391d8e5e2370d6ef0c884ca69c4f6b553e61445def06552e7b7f"
Oct 30 01:26:36.209342 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.209297    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"]
Oct 30 01:26:36.210390 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.210363    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8915869a-a352-4b75-a761-7462858e6ce7" containerName="payload-processing"
Oct 30 01:26:36.210390 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.210389    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8915869a-a352-4b75-a761-7462858e6ce7" containerName="payload-processing"
Oct 30 01:26:36.210595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.210444    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9f921d86-8439-40e7-bb10-3a3f55b7f918" containerName="payload-processing"
Oct 30 01:26:36.210595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.210452    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f921d86-8439-40e7-bb10-3a3f55b7f918" containerName="payload-processing"
Oct 30 01:26:36.210705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.210632    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8915869a-a352-4b75-a761-7462858e6ce7" containerName="payload-processing"
Oct 30 01:26:36.210705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.210657    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f921d86-8439-40e7-bb10-3a3f55b7f918" containerName="payload-processing"
Oct 30 01:26:36.216621 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.216588    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.244621 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.244578    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"]
Oct 30 01:26:36.310963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.310929    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-1cb829-openshift-default-dockercfg-gh8kg\""
Oct 30 01:26:36.361488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.361439    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-envoy\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.361726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.361621    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pt54\" (UniqueName: \"kubernetes.io/projected/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-kube-api-access-9pt54\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.361806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.361726    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istiod-ca-cert\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.361806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.361780    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-workload-certs\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.361904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.361887    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-workload-socket\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.361946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.361924    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-data\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.361995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.361957    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-token\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.362100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.362072    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-podinfo\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.362186 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.362127    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-credential-socket\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.463850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.463746    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-workload-socket\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.463850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.463800    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-data\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.463850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.463833    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-token\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.464202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.463870    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-podinfo\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.464202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.463911    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-credential-socket\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.464202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.463942    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-envoy\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.464202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.464046    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pt54\" (UniqueName: \"kubernetes.io/projected/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-kube-api-access-9pt54\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.464202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.464115    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istiod-ca-cert\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.464202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.464197    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-workload-certs\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.464557 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.464283    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-workload-socket\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.464557 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.464453    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-credential-socket\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.464557 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.464482    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-data\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.464696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.464673    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-workload-certs\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.464975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.464955    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istiod-ca-cert\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.466472 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.466449    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-envoy\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.466615 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.466598    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-podinfo\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.485537 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.485497    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-token\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.485721 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.485594    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pt54\" (UniqueName: \"kubernetes.io/projected/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-kube-api-access-9pt54\") pod \"e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") " pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.534051 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.534013    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:36.865067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.865035    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"]
Oct 30 01:26:36.865671 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:26:36.865645    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cfcf3c_62bf_4b6d_9b78_abf5b6ae290c.slice/crio-e8a9423962edd0ab0ac1aa8e55f357732c3f76b0a9a56ad39e6d0cc204e17d1f WatchSource:0}: Error finding container e8a9423962edd0ab0ac1aa8e55f357732c3f76b0a9a56ad39e6d0cc204e17d1f: Status 404 returned error can't find the container with id e8a9423962edd0ab0ac1aa8e55f357732c3f76b0a9a56ad39e6d0cc204e17d1f
Oct 30 01:26:36.868495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.868455    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:26:36.868628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.868552    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:26:36.868628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:36.868601    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:26:37.039385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:37.039348    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl" event={"ID":"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c","Type":"ContainerStarted","Data":"2fa833472a79a8027ac56e751f311b2b8917468c23fee3912693671eee8656c9"}
Oct 30 01:26:37.039595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:37.039403    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl" event={"ID":"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c","Type":"ContainerStarted","Data":"e8a9423962edd0ab0ac1aa8e55f357732c3f76b0a9a56ad39e6d0cc204e17d1f"}
Oct 30 01:26:37.082062 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:37.081937    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl" podStartSLOduration=1.0819137620000001 podStartE2EDuration="1.081913762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:37.080393678 +0000 UTC m=+2705.830048988" watchObservedRunningTime="2026-10-30 01:26:37.081913762 +0000 UTC m=+2705.831569077"
Oct 30 01:26:37.534410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:37.534366    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:37.536648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:37.536615    4425 patch_prober.go:29] interesting pod/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 01:26:37.536782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:37.536675    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl" podUID="c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 01:26:37.573551 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:26:37.573503    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:26:38.535519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:38.535470    4425 patch_prober.go:29] interesting pod/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 01:26:38.535954 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:38.535533    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl" podUID="c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 01:26:39.534759 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:39.534716    4425 patch_prober.go:29] interesting pod/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 01:26:39.534997 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:39.534785    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl" podUID="c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 01:26:40.534746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:40.534707    4425 patch_prober.go:29] interesting pod/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 01:26:40.535214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:40.534769    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl" podUID="c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 01:26:40.973260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:40.972929    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"]
Oct 30 01:26:40.978832 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:40.978796    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"
Oct 30 01:26:40.980734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:40.980697    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"]
Oct 30 01:26:40.995232 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:40.992482    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:26:41.000978 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:41.000946    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:26:41.030739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:41.030694    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-683f64b3-5lxcg\" (UID: \"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"
Oct 30 01:26:41.030971 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:41.030928    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-683f64b3-5lxcg\" (UID: \"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"
Oct 30 01:26:41.031076 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:41.031061    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cksg2\" (UniqueName: \"kubernetes.io/projected/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-kube-api-access-cksg2\") pod \"maas-api-revoke-sub-683f64b3-5lxcg\" (UID: \"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"
Oct 30 01:26:41.132687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:41.132645    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-683f64b3-5lxcg\" (UID: \"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"
Oct 30 01:26:41.132915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:41.132804    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-683f64b3-5lxcg\" (UID: \"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"
Oct 30 01:26:41.132915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:41.132888    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cksg2\" (UniqueName: \"kubernetes.io/projected/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-kube-api-access-cksg2\") pod \"maas-api-revoke-sub-683f64b3-5lxcg\" (UID: \"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"
Oct 30 01:26:41.133438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:41.133403    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-683f64b3-5lxcg\" (UID: \"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"
Oct 30 01:26:41.135494 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:41.135469    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-683f64b3-5lxcg\" (UID: \"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"
Oct 30 01:26:41.176257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:41.176212    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cksg2\" (UniqueName: \"kubernetes.io/projected/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-kube-api-access-cksg2\") pod \"maas-api-revoke-sub-683f64b3-5lxcg\" (UID: \"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"
Oct 30 01:26:41.304507 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:41.304466    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"
Oct 30 01:26:41.535726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:41.535686    4425 patch_prober.go:29] interesting pod/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 01:26:41.536219 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:41.535747    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl" podUID="c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 01:26:41.639602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:41.639567    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"]
Oct 30 01:26:41.642225 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:26:41.642192    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3c6775_eb0a_4cdf_8bb4_9982eab056b8.slice/crio-17c9364e7057293d4b9ed9eac51e866db8f5e090c0348520204d70624e089104 WatchSource:0}: Error finding container 17c9364e7057293d4b9ed9eac51e866db8f5e090c0348520204d70624e089104: Status 404 returned error can't find the container with id 17c9364e7057293d4b9ed9eac51e866db8f5e090c0348520204d70624e089104
Oct 30 01:26:42.104297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:42.104259    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg" event={"ID":"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8","Type":"ContainerStarted","Data":"17c9364e7057293d4b9ed9eac51e866db8f5e090c0348520204d70624e089104"}
Oct 30 01:26:42.534629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:42.534593    4425 patch_prober.go:29] interesting pod/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 01:26:42.534848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:42.534650    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl" podUID="c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 01:26:43.115370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:43.115095    4425 generic.go:356] "Generic (PLEG): container finished" podID="6b3c6775-eb0a-4cdf-8bb4-9982eab056b8" containerID="9dffa0fc7eb747f9c977f70870d812f745adbb7d2309a90ea772f4d0a45f87cf" exitCode=0
Oct 30 01:26:43.115370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:43.115143    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg" event={"ID":"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8","Type":"ContainerDied","Data":"9dffa0fc7eb747f9c977f70870d812f745adbb7d2309a90ea772f4d0a45f87cf"}
Oct 30 01:26:43.540523 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:43.540480    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:43.540819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:43.540791    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:43.541697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:43.541677    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:26:44.300399 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:44.300364    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"
Oct 30 01:26:44.401684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:44.401583    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-kube-api-access-cksg2\" (UniqueName: \"kubernetes.io/projected/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-kube-api-access-cksg2\") pod \"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8\" (UID: \"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8\") "
Oct 30 01:26:44.401684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:44.401673    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-maas-api-openshift-ca\") pod \"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8\" (UID: \"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8\") "
Oct 30 01:26:44.402039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:44.401822    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-maas-api-xks-ca\") pod \"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8\" (UID: \"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8\") "
Oct 30 01:26:44.402129 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:44.402097    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-maas-api-openshift-ca" pod "6b3c6775-eb0a-4cdf-8bb4-9982eab056b8" (UID: "6b3c6775-eb0a-4cdf-8bb4-9982eab056b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:44.402480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:44.402453    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:26:44.404038 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:44.404007    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-maas-api-xks-ca" pod "6b3c6775-eb0a-4cdf-8bb4-9982eab056b8" (UID: "6b3c6775-eb0a-4cdf-8bb4-9982eab056b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:26:44.404137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:44.404018    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-kube-api-access-cksg2" pod "6b3c6775-eb0a-4cdf-8bb4-9982eab056b8" (UID: "6b3c6775-eb0a-4cdf-8bb4-9982eab056b8"). InnerVolumeSpecName "kube-api-access-cksg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:44.504097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:44.504050    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:26:44.504097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:44.504085    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cksg2\" (UniqueName: \"kubernetes.io/projected/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8-kube-api-access-cksg2\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:26:45.137552 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:45.137510    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg" event={"ID":"6b3c6775-eb0a-4cdf-8bb4-9982eab056b8","Type":"ContainerDied","Data":"17c9364e7057293d4b9ed9eac51e866db8f5e090c0348520204d70624e089104"}
Oct 30 01:26:45.137552 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:45.137552    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"
Oct 30 01:26:45.137862 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:45.137614    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17c9364e7057293d4b9ed9eac51e866db8f5e090c0348520204d70624e089104"
Oct 30 01:26:50.080638 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.080597    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"]
Oct 30 01:26:50.081496 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.081475    4425 cpu_manager.go:414] "containerMap: removing container" podUID="6b3c6775-eb0a-4cdf-8bb4-9982eab056b8" containerName="revoke-keys"
Oct 30 01:26:50.081572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.081497    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3c6775-eb0a-4cdf-8bb4-9982eab056b8" containerName="revoke-keys"
Oct 30 01:26:50.081630 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.081621    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3c6775-eb0a-4cdf-8bb4-9982eab056b8" containerName="revoke-keys"
Oct 30 01:26:50.086400 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.086372    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"
Oct 30 01:26:50.100350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.100306    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:26:50.105935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.105896    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"]
Oct 30 01:26:50.110235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.110206    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:26:50.183184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.183129    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c21980-39ac-404a-b60e-03bb753a335e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f19b72b-9jnj8\" (UID: \"43c21980-39ac-404a-b60e-03bb753a335e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"
Oct 30 01:26:50.183410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.183244    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx6jf\" (UniqueName: \"kubernetes.io/projected/43c21980-39ac-404a-b60e-03bb753a335e-kube-api-access-lx6jf\") pod \"maas-api-revoke-sub-8f19b72b-9jnj8\" (UID: \"43c21980-39ac-404a-b60e-03bb753a335e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"
Oct 30 01:26:50.183559 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.183537    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c21980-39ac-404a-b60e-03bb753a335e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f19b72b-9jnj8\" (UID: \"43c21980-39ac-404a-b60e-03bb753a335e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"
Oct 30 01:26:50.285285 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.285242    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c21980-39ac-404a-b60e-03bb753a335e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f19b72b-9jnj8\" (UID: \"43c21980-39ac-404a-b60e-03bb753a335e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"
Oct 30 01:26:50.285529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.285346    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c21980-39ac-404a-b60e-03bb753a335e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f19b72b-9jnj8\" (UID: \"43c21980-39ac-404a-b60e-03bb753a335e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"
Oct 30 01:26:50.285529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.285385    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx6jf\" (UniqueName: \"kubernetes.io/projected/43c21980-39ac-404a-b60e-03bb753a335e-kube-api-access-lx6jf\") pod \"maas-api-revoke-sub-8f19b72b-9jnj8\" (UID: \"43c21980-39ac-404a-b60e-03bb753a335e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"
Oct 30 01:26:50.286097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.286065    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c21980-39ac-404a-b60e-03bb753a335e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f19b72b-9jnj8\" (UID: \"43c21980-39ac-404a-b60e-03bb753a335e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"
Oct 30 01:26:50.287970 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.287942    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c21980-39ac-404a-b60e-03bb753a335e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f19b72b-9jnj8\" (UID: \"43c21980-39ac-404a-b60e-03bb753a335e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"
Oct 30 01:26:50.307695 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.307662    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx6jf\" (UniqueName: \"kubernetes.io/projected/43c21980-39ac-404a-b60e-03bb753a335e-kube-api-access-lx6jf\") pod \"maas-api-revoke-sub-8f19b72b-9jnj8\" (UID: \"43c21980-39ac-404a-b60e-03bb753a335e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"
Oct 30 01:26:50.406857 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.406755    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"
Oct 30 01:26:50.749891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:50.749856    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"]
Oct 30 01:26:50.751746 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:26:50.751716    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c21980_39ac_404a_b60e_03bb753a335e.slice/crio-53010b4673b9c028ceb3de51fc06461b39b3c5f8fb3dccdaa1965a037f9e59da WatchSource:0}: Error finding container 53010b4673b9c028ceb3de51fc06461b39b3c5f8fb3dccdaa1965a037f9e59da: Status 404 returned error can't find the container with id 53010b4673b9c028ceb3de51fc06461b39b3c5f8fb3dccdaa1965a037f9e59da
Oct 30 01:26:51.189018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:51.188980    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8" event={"ID":"43c21980-39ac-404a-b60e-03bb753a335e","Type":"ContainerStarted","Data":"6d96e22fbaf8a799b10348abdeeddafff1ccf38037d7201af95716d113b7f7ec"}
Oct 30 01:26:51.189606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:51.189042    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8" event={"ID":"43c21980-39ac-404a-b60e-03bb753a335e","Type":"ContainerStarted","Data":"53010b4673b9c028ceb3de51fc06461b39b3c5f8fb3dccdaa1965a037f9e59da"}
Oct 30 01:26:51.227630 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:51.227574    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8" podStartSLOduration=1.227556609 podStartE2EDuration="1.227556609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:51.223757473 +0000 UTC m=+2719.973412786" watchObservedRunningTime="2026-10-30 01:26:51.227556609 +0000 UTC m=+2719.977211917"
Oct 30 01:26:52.198615 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:52.198518    4425 generic.go:356] "Generic (PLEG): container finished" podID="43c21980-39ac-404a-b60e-03bb753a335e" containerID="6d96e22fbaf8a799b10348abdeeddafff1ccf38037d7201af95716d113b7f7ec" exitCode=0
Oct 30 01:26:52.199128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:52.198607    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8" event={"ID":"43c21980-39ac-404a-b60e-03bb753a335e","Type":"ContainerDied","Data":"6d96e22fbaf8a799b10348abdeeddafff1ccf38037d7201af95716d113b7f7ec"}
Oct 30 01:26:53.388832 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:53.388800    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"
Oct 30 01:26:53.429805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:53.429768    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c21980-39ac-404a-b60e-03bb753a335e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c21980-39ac-404a-b60e-03bb753a335e-maas-api-xks-ca\") pod \"43c21980-39ac-404a-b60e-03bb753a335e\" (UID: \"43c21980-39ac-404a-b60e-03bb753a335e\") "
Oct 30 01:26:53.431480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:53.429825    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c21980-39ac-404a-b60e-03bb753a335e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c21980-39ac-404a-b60e-03bb753a335e-maas-api-openshift-ca\") pod \"43c21980-39ac-404a-b60e-03bb753a335e\" (UID: \"43c21980-39ac-404a-b60e-03bb753a335e\") "
Oct 30 01:26:53.431480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:53.429938    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c21980-39ac-404a-b60e-03bb753a335e-kube-api-access-lx6jf\" (UniqueName: \"kubernetes.io/projected/43c21980-39ac-404a-b60e-03bb753a335e-kube-api-access-lx6jf\") pod \"43c21980-39ac-404a-b60e-03bb753a335e\" (UID: \"43c21980-39ac-404a-b60e-03bb753a335e\") "
Oct 30 01:26:53.431480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:53.430662    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c21980-39ac-404a-b60e-03bb753a335e-maas-api-openshift-ca" pod "43c21980-39ac-404a-b60e-03bb753a335e" (UID: "43c21980-39ac-404a-b60e-03bb753a335e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:26:53.431480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:53.431448    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c21980-39ac-404a-b60e-03bb753a335e-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:26:53.434809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:53.434762    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c21980-39ac-404a-b60e-03bb753a335e-maas-api-xks-ca" pod "43c21980-39ac-404a-b60e-03bb753a335e" (UID: "43c21980-39ac-404a-b60e-03bb753a335e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:26:53.436363 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:53.436328    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c21980-39ac-404a-b60e-03bb753a335e-kube-api-access-lx6jf" pod "43c21980-39ac-404a-b60e-03bb753a335e" (UID: "43c21980-39ac-404a-b60e-03bb753a335e"). InnerVolumeSpecName "kube-api-access-lx6jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:26:53.533396 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:53.533341    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c21980-39ac-404a-b60e-03bb753a335e-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:26:53.533396 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:53.533397    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx6jf\" (UniqueName: \"kubernetes.io/projected/43c21980-39ac-404a-b60e-03bb753a335e-kube-api-access-lx6jf\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:26:54.218605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:54.218565    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"
Oct 30 01:26:54.218605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:54.218584    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8" event={"ID":"43c21980-39ac-404a-b60e-03bb753a335e","Type":"ContainerDied","Data":"53010b4673b9c028ceb3de51fc06461b39b3c5f8fb3dccdaa1965a037f9e59da"}
Oct 30 01:26:54.218865 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:26:54.218651    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53010b4673b9c028ceb3de51fc06461b39b3c5f8fb3dccdaa1965a037f9e59da"
Oct 30 01:27:01.453011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:01.452960    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"]
Oct 30 01:27:02.316608 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:02.316561    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"]
Oct 30 01:27:06.454190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:06.454095    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5" podUID="a16443ab-4a60-4682-bcc3-7236f4ab7965" containerName="payload-processing" containerID="cri-o://0183f77930a8ce4e0beef3179c0aa0976efb1182a31467d2f030087af46452bd" gracePeriod=25
Oct 30 01:27:06.732664 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:06.732630    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:27:06.827952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:06.827913    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a16443ab-4a60-4682-bcc3-7236f4ab7965-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a16443ab-4a60-4682-bcc3-7236f4ab7965-routing-overlay-volume\") pod \"a16443ab-4a60-4682-bcc3-7236f4ab7965\" (UID: \"a16443ab-4a60-4682-bcc3-7236f4ab7965\") "
Oct 30 01:27:06.827952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:06.827956    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a16443ab-4a60-4682-bcc3-7236f4ab7965-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a16443ab-4a60-4682-bcc3-7236f4ab7965-plugins-config-volume\") pod \"a16443ab-4a60-4682-bcc3-7236f4ab7965\" (UID: \"a16443ab-4a60-4682-bcc3-7236f4ab7965\") "
Oct 30 01:27:06.828311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:06.827986    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16443ab-4a60-4682-bcc3-7236f4ab7965-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a16443ab-4a60-4682-bcc3-7236f4ab7965-provider-credentials\") pod \"a16443ab-4a60-4682-bcc3-7236f4ab7965\" (UID: \"a16443ab-4a60-4682-bcc3-7236f4ab7965\") "
Oct 30 01:27:06.828311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:06.828015    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16443ab-4a60-4682-bcc3-7236f4ab7965-kube-api-access-cbcqm\" (UniqueName: \"kubernetes.io/projected/a16443ab-4a60-4682-bcc3-7236f4ab7965-kube-api-access-cbcqm\") pod \"a16443ab-4a60-4682-bcc3-7236f4ab7965\" (UID: \"a16443ab-4a60-4682-bcc3-7236f4ab7965\") "
Oct 30 01:27:06.828508 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:06.828476    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16443ab-4a60-4682-bcc3-7236f4ab7965-routing-overlay-volume" pod "a16443ab-4a60-4682-bcc3-7236f4ab7965" (UID: "a16443ab-4a60-4682-bcc3-7236f4ab7965"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:06.828578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:06.828548    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16443ab-4a60-4682-bcc3-7236f4ab7965-plugins-config-volume" pod "a16443ab-4a60-4682-bcc3-7236f4ab7965" (UID: "a16443ab-4a60-4682-bcc3-7236f4ab7965"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:06.828884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:06.828867    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a16443ab-4a60-4682-bcc3-7236f4ab7965-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:06.828927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:06.828889    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a16443ab-4a60-4682-bcc3-7236f4ab7965-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:06.830548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:06.830521    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16443ab-4a60-4682-bcc3-7236f4ab7965-kube-api-access-cbcqm" pod "a16443ab-4a60-4682-bcc3-7236f4ab7965" (UID: "a16443ab-4a60-4682-bcc3-7236f4ab7965"). InnerVolumeSpecName "kube-api-access-cbcqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:06.830548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:06.830533    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16443ab-4a60-4682-bcc3-7236f4ab7965-provider-credentials" pod "a16443ab-4a60-4682-bcc3-7236f4ab7965" (UID: "a16443ab-4a60-4682-bcc3-7236f4ab7965"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:06.929923 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:06.929873    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbcqm\" (UniqueName: \"kubernetes.io/projected/a16443ab-4a60-4682-bcc3-7236f4ab7965-kube-api-access-cbcqm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:06.929923 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:06.929914    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a16443ab-4a60-4682-bcc3-7236f4ab7965-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:07.317725 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.317674    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv" podUID="b7f91a58-9c86-43e4-9dfd-6ece7b3d7066" containerName="payload-processing" containerID="cri-o://c33ca5a481c754265159f9c7a47aa29f7ac15583fb038e9140ad8e1fa9f9db0f" gracePeriod=25
Oct 30 01:27:07.332453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.332414    4425 generic.go:356] "Generic (PLEG): container finished" podID="a16443ab-4a60-4682-bcc3-7236f4ab7965" containerID="0183f77930a8ce4e0beef3179c0aa0976efb1182a31467d2f030087af46452bd" exitCode=0
Oct 30 01:27:07.332666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.332537    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"
Oct 30 01:27:07.332666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.332550    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5" event={"ID":"a16443ab-4a60-4682-bcc3-7236f4ab7965","Type":"ContainerDied","Data":"0183f77930a8ce4e0beef3179c0aa0976efb1182a31467d2f030087af46452bd"}
Oct 30 01:27:07.332666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.332614    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5" event={"ID":"a16443ab-4a60-4682-bcc3-7236f4ab7965","Type":"ContainerDied","Data":"e0835ba6a500da88f7d94cbe335250a3848e879deb5674ecdb57dd9c8e9f99b2"}
Oct 30 01:27:07.332666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.332637    4425 scope.go:122] "RemoveContainer" containerID="0183f77930a8ce4e0beef3179c0aa0976efb1182a31467d2f030087af46452bd"
Oct 30 01:27:07.346805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.346780    4425 scope.go:122] "RemoveContainer" containerID="0183f77930a8ce4e0beef3179c0aa0976efb1182a31467d2f030087af46452bd"
Oct 30 01:27:07.347401 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:07.347315    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0183f77930a8ce4e0beef3179c0aa0976efb1182a31467d2f030087af46452bd\": container with ID starting with 0183f77930a8ce4e0beef3179c0aa0976efb1182a31467d2f030087af46452bd not found: ID does not exist" containerID="0183f77930a8ce4e0beef3179c0aa0976efb1182a31467d2f030087af46452bd"
Oct 30 01:27:07.347401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.347362    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0183f77930a8ce4e0beef3179c0aa0976efb1182a31467d2f030087af46452bd"} err="failed to get container status \"0183f77930a8ce4e0beef3179c0aa0976efb1182a31467d2f030087af46452bd\": rpc error: code = NotFound desc = could not find container \"0183f77930a8ce4e0beef3179c0aa0976efb1182a31467d2f030087af46452bd\": container with ID starting with 0183f77930a8ce4e0beef3179c0aa0976efb1182a31467d2f030087af46452bd not found: ID does not exist"
Oct 30 01:27:07.439973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.439938    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"]
Oct 30 01:27:07.449528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.449494    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-5664pdlb5"]
Oct 30 01:27:07.564826 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:07.564757    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:27:07.605945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.605917    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:27:07.742982 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.742939    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-provider-credentials\") pod \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\" (UID: \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\") "
Oct 30 01:27:07.743230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.743084    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-plugins-config-volume\") pod \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\" (UID: \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\") "
Oct 30 01:27:07.743230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.743210    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-routing-overlay-volume\") pod \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\" (UID: \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\") "
Oct 30 01:27:07.743230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.743229    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-kube-api-access-t8k2s\" (UniqueName: \"kubernetes.io/projected/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-kube-api-access-t8k2s\") pod \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\" (UID: \"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066\") "
Oct 30 01:27:07.743599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.743564    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-routing-overlay-volume" pod "b7f91a58-9c86-43e4-9dfd-6ece7b3d7066" (UID: "b7f91a58-9c86-43e4-9dfd-6ece7b3d7066"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:07.743599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.743578    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-plugins-config-volume" pod "b7f91a58-9c86-43e4-9dfd-6ece7b3d7066" (UID: "b7f91a58-9c86-43e4-9dfd-6ece7b3d7066"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:07.743837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.743815    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:07.743906 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.743844    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:07.745728 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.745706    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-provider-credentials" pod "b7f91a58-9c86-43e4-9dfd-6ece7b3d7066" (UID: "b7f91a58-9c86-43e4-9dfd-6ece7b3d7066"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:07.745834 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.745808    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-kube-api-access-t8k2s" pod "b7f91a58-9c86-43e4-9dfd-6ece7b3d7066" (UID: "b7f91a58-9c86-43e4-9dfd-6ece7b3d7066"). InnerVolumeSpecName "kube-api-access-t8k2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:07.845734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.845608    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8k2s\" (UniqueName: \"kubernetes.io/projected/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-kube-api-access-t8k2s\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:07.845734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:07.845645    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:08.344551 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:08.344514    4425 generic.go:356] "Generic (PLEG): container finished" podID="b7f91a58-9c86-43e4-9dfd-6ece7b3d7066" containerID="c33ca5a481c754265159f9c7a47aa29f7ac15583fb038e9140ad8e1fa9f9db0f" exitCode=0
Oct 30 01:27:08.344795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:08.344669    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv" event={"ID":"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066","Type":"ContainerDied","Data":"c33ca5a481c754265159f9c7a47aa29f7ac15583fb038e9140ad8e1fa9f9db0f"}
Oct 30 01:27:08.344795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:08.344711    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv" event={"ID":"b7f91a58-9c86-43e4-9dfd-6ece7b3d7066","Type":"ContainerDied","Data":"4e2d3a8212ebd247dabb209bed707fddd657299c01e9d6dc14eabf4905667f9f"}
Oct 30 01:27:08.344795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:08.344741    4425 scope.go:122] "RemoveContainer" containerID="c33ca5a481c754265159f9c7a47aa29f7ac15583fb038e9140ad8e1fa9f9db0f"
Oct 30 01:27:08.344980 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:08.344841    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"
Oct 30 01:27:08.359395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:08.359367    4425 scope.go:122] "RemoveContainer" containerID="c33ca5a481c754265159f9c7a47aa29f7ac15583fb038e9140ad8e1fa9f9db0f"
Oct 30 01:27:08.359736 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:08.359714    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33ca5a481c754265159f9c7a47aa29f7ac15583fb038e9140ad8e1fa9f9db0f\": container with ID starting with c33ca5a481c754265159f9c7a47aa29f7ac15583fb038e9140ad8e1fa9f9db0f not found: ID does not exist" containerID="c33ca5a481c754265159f9c7a47aa29f7ac15583fb038e9140ad8e1fa9f9db0f"
Oct 30 01:27:08.359783 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:08.359747    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33ca5a481c754265159f9c7a47aa29f7ac15583fb038e9140ad8e1fa9f9db0f"} err="failed to get container status \"c33ca5a481c754265159f9c7a47aa29f7ac15583fb038e9140ad8e1fa9f9db0f\": rpc error: code = NotFound desc = could not find container \"c33ca5a481c754265159f9c7a47aa29f7ac15583fb038e9140ad8e1fa9f9db0f\": container with ID starting with c33ca5a481c754265159f9c7a47aa29f7ac15583fb038e9140ad8e1fa9f9db0f not found: ID does not exist"
Oct 30 01:27:08.393217 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:08.393169    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"]
Oct 30 01:27:08.407510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:08.407470    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-dbb8hgjpv"]
Oct 30 01:27:09.439187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.436411    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a16443ab-4a60-4682-bcc3-7236f4ab7965" path="/var/lib/kubelet/pods/a16443ab-4a60-4682-bcc3-7236f4ab7965/volumes"
Oct 30 01:27:09.439187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.437254    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f91a58-9c86-43e4-9dfd-6ece7b3d7066" path="/var/lib/kubelet/pods/b7f91a58-9c86-43e4-9dfd-6ece7b3d7066/volumes"
Oct 30 01:27:09.509924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.509885    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"]
Oct 30 01:27:09.510682 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.510665    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b7f91a58-9c86-43e4-9dfd-6ece7b3d7066" containerName="payload-processing"
Oct 30 01:27:09.510736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.510684    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f91a58-9c86-43e4-9dfd-6ece7b3d7066" containerName="payload-processing"
Oct 30 01:27:09.510736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.510697    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a16443ab-4a60-4682-bcc3-7236f4ab7965" containerName="payload-processing"
Oct 30 01:27:09.510736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.510703    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a16443ab-4a60-4682-bcc3-7236f4ab7965" containerName="payload-processing"
Oct 30 01:27:09.510736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.510726    4425 cpu_manager.go:414] "containerMap: removing container" podUID="43c21980-39ac-404a-b60e-03bb753a335e" containerName="revoke-keys"
Oct 30 01:27:09.510736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.510731    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c21980-39ac-404a-b60e-03bb753a335e" containerName="revoke-keys"
Oct 30 01:27:09.510898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.510888    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c21980-39ac-404a-b60e-03bb753a335e" containerName="revoke-keys"
Oct 30 01:27:09.510932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.510906    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f91a58-9c86-43e4-9dfd-6ece7b3d7066" containerName="payload-processing"
Oct 30 01:27:09.510932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.510913    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a16443ab-4a60-4682-bcc3-7236f4ab7965" containerName="payload-processing"
Oct 30 01:27:09.516123 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.516096    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"
Oct 30 01:27:09.536456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.536413    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"]
Oct 30 01:27:09.541351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.541320    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-c48628\""
Oct 30 01:27:09.541533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.541397    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-c48628-dockercfg-srsbk\""
Oct 30 01:27:09.621584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.621508    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"]
Oct 30 01:27:09.646671 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.646634    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"]
Oct 30 01:27:09.651277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.651242    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"
Oct 30 01:27:09.673782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.673745    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a09e9b-a4fa-4ef9-8d76-b75e98f68a74-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj\" (UID: \"23a09e9b-a4fa-4ef9-8d76-b75e98f68a74\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"
Oct 30 01:27:09.673972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.673932    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq4nj\" (UniqueName: \"kubernetes.io/projected/473e2e7d-4f24-449d-b508-cb9f05e78c8b-kube-api-access-xq4nj\") pod \"payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4\" (UID: \"473e2e7d-4f24-449d-b508-cb9f05e78c8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"
Oct 30 01:27:09.674108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.674081    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473e2e7d-4f24-449d-b508-cb9f05e78c8b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4\" (UID: \"473e2e7d-4f24-449d-b508-cb9f05e78c8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"
Oct 30 01:27:09.674220 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.674146    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qv2d\" (UniqueName: \"kubernetes.io/projected/23a09e9b-a4fa-4ef9-8d76-b75e98f68a74-kube-api-access-5qv2d\") pod \"payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj\" (UID: \"23a09e9b-a4fa-4ef9-8d76-b75e98f68a74\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"
Oct 30 01:27:09.775198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.775126    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qv2d\" (UniqueName: \"kubernetes.io/projected/23a09e9b-a4fa-4ef9-8d76-b75e98f68a74-kube-api-access-5qv2d\") pod \"payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj\" (UID: \"23a09e9b-a4fa-4ef9-8d76-b75e98f68a74\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"
Oct 30 01:27:09.775449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.775268    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a09e9b-a4fa-4ef9-8d76-b75e98f68a74-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj\" (UID: \"23a09e9b-a4fa-4ef9-8d76-b75e98f68a74\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"
Oct 30 01:27:09.775449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.775386    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq4nj\" (UniqueName: \"kubernetes.io/projected/473e2e7d-4f24-449d-b508-cb9f05e78c8b-kube-api-access-xq4nj\") pod \"payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4\" (UID: \"473e2e7d-4f24-449d-b508-cb9f05e78c8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"
Oct 30 01:27:09.775589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.775506    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473e2e7d-4f24-449d-b508-cb9f05e78c8b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4\" (UID: \"473e2e7d-4f24-449d-b508-cb9f05e78c8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"
Oct 30 01:27:09.776001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.775978    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a09e9b-a4fa-4ef9-8d76-b75e98f68a74-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj\" (UID: \"23a09e9b-a4fa-4ef9-8d76-b75e98f68a74\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"
Oct 30 01:27:09.776237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.776219    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473e2e7d-4f24-449d-b508-cb9f05e78c8b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4\" (UID: \"473e2e7d-4f24-449d-b508-cb9f05e78c8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"
Oct 30 01:27:09.806894 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.806855    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq4nj\" (UniqueName: \"kubernetes.io/projected/473e2e7d-4f24-449d-b508-cb9f05e78c8b-kube-api-access-xq4nj\") pod \"payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4\" (UID: \"473e2e7d-4f24-449d-b508-cb9f05e78c8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"
Oct 30 01:27:09.807656 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.807631    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qv2d\" (UniqueName: \"kubernetes.io/projected/23a09e9b-a4fa-4ef9-8d76-b75e98f68a74-kube-api-access-5qv2d\") pod \"payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj\" (UID: \"23a09e9b-a4fa-4ef9-8d76-b75e98f68a74\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"
Oct 30 01:27:09.832416 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.832377    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"
Oct 30 01:27:09.987417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:09.987370    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"
Oct 30 01:27:10.144347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:10.144310    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"]
Oct 30 01:27:10.146701 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:27:10.146673    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473e2e7d_4f24_449d_b508_cb9f05e78c8b.slice/crio-af5a0f0a361fcbf87a5e0b3139a1720c1efe0f66332095069b3916bdd5a62752 WatchSource:0}: Error finding container af5a0f0a361fcbf87a5e0b3139a1720c1efe0f66332095069b3916bdd5a62752: Status 404 returned error can't find the container with id af5a0f0a361fcbf87a5e0b3139a1720c1efe0f66332095069b3916bdd5a62752
Oct 30 01:27:10.297876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:10.297840    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"]
Oct 30 01:27:10.298043 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:27:10.298015    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a09e9b_a4fa_4ef9_8d76_b75e98f68a74.slice/crio-8ddd526f3aa01e562d88f0caaded8a6c8c0dcbd51e2a1f9f838db56610192745 WatchSource:0}: Error finding container 8ddd526f3aa01e562d88f0caaded8a6c8c0dcbd51e2a1f9f838db56610192745: Status 404 returned error can't find the container with id 8ddd526f3aa01e562d88f0caaded8a6c8c0dcbd51e2a1f9f838db56610192745
Oct 30 01:27:10.366967 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:10.366927    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj" event={"ID":"23a09e9b-a4fa-4ef9-8d76-b75e98f68a74","Type":"ContainerStarted","Data":"8ddd526f3aa01e562d88f0caaded8a6c8c0dcbd51e2a1f9f838db56610192745"}
Oct 30 01:27:10.369320 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:10.369287    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4" event={"ID":"473e2e7d-4f24-449d-b508-cb9f05e78c8b","Type":"ContainerStarted","Data":"87cde047e3cf906e7d56c3162af49301bc6cd0a2b1148018ba124e22bd998c2a"}
Oct 30 01:27:10.369471 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:10.369335    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4" event={"ID":"473e2e7d-4f24-449d-b508-cb9f05e78c8b","Type":"ContainerStarted","Data":"af5a0f0a361fcbf87a5e0b3139a1720c1efe0f66332095069b3916bdd5a62752"}
Oct 30 01:27:10.369471 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:10.369441    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"
Oct 30 01:27:10.412548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:10.412492    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4" podStartSLOduration=1.412476221 podStartE2EDuration="1.412476221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:10.411798181 +0000 UTC m=+2739.161453481" watchObservedRunningTime="2026-10-30 01:27:10.412476221 +0000 UTC m=+2739.162131637"
Oct 30 01:27:10.827780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:10.827739    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:27:11.379187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:11.379133    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj" event={"ID":"23a09e9b-a4fa-4ef9-8d76-b75e98f68a74","Type":"ContainerStarted","Data":"45936c75d450850f91b4c2b146e767ba5340965f4c92fc3e213c16158c44ce12"}
Oct 30 01:27:11.379411 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:11.379220    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"
Oct 30 01:27:11.418278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:11.418216    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj" podStartSLOduration=2.418200708 podStartE2EDuration="2.418200708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:11.414521203 +0000 UTC m=+2740.164176510" watchObservedRunningTime="2026-10-30 01:27:11.418200708 +0000 UTC m=+2740.167856017"
Oct 30 01:27:16.048894 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.048793    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"]
Oct 30 01:27:16.049449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.049091    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24" podUID="3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" containerName="istio-proxy" containerID="cri-o://e717335be880557407fe751ef248ff254376d4f1323e707e0d42a2135a255f66" gracePeriod=30
Oct 30 01:27:16.611752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.611705    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"]
Oct 30 01:27:16.616901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.616871    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:16.630432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.630404    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"kube-root-ca.crt\""
Oct 30 01:27:16.633456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.633427    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"openshift-service-ca.crt\""
Oct 30 01:27:16.633634 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.633498    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"routing-overlay\""
Oct 30 01:27:16.633767 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.633741    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2f820e\""
Oct 30 01:27:16.633864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.633787    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"agc-cred-4602157e7193\""
Oct 30 01:27:16.638761 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.638728    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"]
Oct 30 01:27:16.683406 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.683359    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51f680d6-14c5-4716-a142-f0cc9235f79e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8\" (UID: \"51f680d6-14c5-4716-a142-f0cc9235f79e\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:16.683609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.683548    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51f680d6-14c5-4716-a142-f0cc9235f79e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8\" (UID: \"51f680d6-14c5-4716-a142-f0cc9235f79e\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:16.683609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.683573    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-572ft\" (UniqueName: \"kubernetes.io/projected/51f680d6-14c5-4716-a142-f0cc9235f79e-kube-api-access-572ft\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8\" (UID: \"51f680d6-14c5-4716-a142-f0cc9235f79e\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:16.683716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.683610    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f680d6-14c5-4716-a142-f0cc9235f79e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8\" (UID: \"51f680d6-14c5-4716-a142-f0cc9235f79e\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:16.785270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.785224    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51f680d6-14c5-4716-a142-f0cc9235f79e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8\" (UID: \"51f680d6-14c5-4716-a142-f0cc9235f79e\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:16.785500 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.785366    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51f680d6-14c5-4716-a142-f0cc9235f79e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8\" (UID: \"51f680d6-14c5-4716-a142-f0cc9235f79e\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:16.785500 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.785385    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-572ft\" (UniqueName: \"kubernetes.io/projected/51f680d6-14c5-4716-a142-f0cc9235f79e-kube-api-access-572ft\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8\" (UID: \"51f680d6-14c5-4716-a142-f0cc9235f79e\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:16.785500 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.785413    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f680d6-14c5-4716-a142-f0cc9235f79e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8\" (UID: \"51f680d6-14c5-4716-a142-f0cc9235f79e\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:16.785982 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.785957    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51f680d6-14c5-4716-a142-f0cc9235f79e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8\" (UID: \"51f680d6-14c5-4716-a142-f0cc9235f79e\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:16.786303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.786284    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f680d6-14c5-4716-a142-f0cc9235f79e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8\" (UID: \"51f680d6-14c5-4716-a142-f0cc9235f79e\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:16.788074 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.788051    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51f680d6-14c5-4716-a142-f0cc9235f79e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8\" (UID: \"51f680d6-14c5-4716-a142-f0cc9235f79e\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:16.807376 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.807298    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-572ft\" (UniqueName: \"kubernetes.io/projected/51f680d6-14c5-4716-a142-f0cc9235f79e-kube-api-access-572ft\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8\" (UID: \"51f680d6-14c5-4716-a142-f0cc9235f79e\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:16.933976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:16.933879    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:17.303602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.303559    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"]
Oct 30 01:27:17.309638 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.309613    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"]
Oct 30 01:27:17.309749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.309678    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:17.311890 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:27:17.311863    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f680d6_14c5_4716_a142_f0cc9235f79e.slice/crio-31da0f8b31c6633d6e2850379e121d355b989df33cf1f6447c6b6a7050b81f3b WatchSource:0}: Error finding container 31da0f8b31c6633d6e2850379e121d355b989df33cf1f6447c6b6a7050b81f3b: Status 404 returned error can't find the container with id 31da0f8b31c6633d6e2850379e121d355b989df33cf1f6447c6b6a7050b81f3b
Oct 30 01:27:17.327175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.327133    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1afb3e\"/\"agc-cred-c1305eba1e78\""
Oct 30 01:27:17.333513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.333475    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"]
Oct 30 01:27:17.393227 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.393185    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9b1139d-7b79-4d77-b64b-677abcf63f0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx\" (UID: \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:17.393498 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.393284    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9b1139d-7b79-4d77-b64b-677abcf63f0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx\" (UID: \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:17.393576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.393544    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b1139d-7b79-4d77-b64b-677abcf63f0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx\" (UID: \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:17.393629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.393605    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfdv4\" (UniqueName: \"kubernetes.io/projected/a9b1139d-7b79-4d77-b64b-677abcf63f0d-kube-api-access-tfdv4\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx\" (UID: \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:17.432878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.432837    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8" event={"ID":"51f680d6-14c5-4716-a142-f0cc9235f79e","Type":"ContainerStarted","Data":"3a627ab8961e64c4bfa69fdf32fc5fc8da917213683b424f1542cd1be367df1d"}
Oct 30 01:27:17.433083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.432902    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8" event={"ID":"51f680d6-14c5-4716-a142-f0cc9235f79e","Type":"ContainerStarted","Data":"31da0f8b31c6633d6e2850379e121d355b989df33cf1f6447c6b6a7050b81f3b"}
Oct 30 01:27:17.433083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.432928    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:17.474800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.474721    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8" podStartSLOduration=1.4747014809999999 podStartE2EDuration="1.474701481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:17.470813936 +0000 UTC m=+2746.220469245" watchObservedRunningTime="2026-10-30 01:27:17.474701481 +0000 UTC m=+2746.224356789"
Oct 30 01:27:17.495594 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.495555    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9b1139d-7b79-4d77-b64b-677abcf63f0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx\" (UID: \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:17.495844 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.495671    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9b1139d-7b79-4d77-b64b-677abcf63f0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx\" (UID: \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:17.496402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.496370    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b1139d-7b79-4d77-b64b-677abcf63f0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx\" (UID: \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:17.496556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.496433    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfdv4\" (UniqueName: \"kubernetes.io/projected/a9b1139d-7b79-4d77-b64b-677abcf63f0d-kube-api-access-tfdv4\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx\" (UID: \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:17.496631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.496585    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9b1139d-7b79-4d77-b64b-677abcf63f0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx\" (UID: \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:17.497056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.497033    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b1139d-7b79-4d77-b64b-677abcf63f0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx\" (UID: \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:17.498577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.498560    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9b1139d-7b79-4d77-b64b-677abcf63f0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx\" (UID: \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:17.522135 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.522096    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfdv4\" (UniqueName: \"kubernetes.io/projected/a9b1139d-7b79-4d77-b64b-677abcf63f0d-kube-api-access-tfdv4\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx\" (UID: \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:17.646114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.646009    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:17.977118 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:17.977032    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"]
Oct 30 01:27:17.981363 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:27:17.981328    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b1139d_7b79_4d77_b64b_677abcf63f0d.slice/crio-e7b9fc4cbe06cd39457159e7cffdf437c8bf754a4fe1ebb51ce4e0871ce5a5df WatchSource:0}: Error finding container e7b9fc4cbe06cd39457159e7cffdf437c8bf754a4fe1ebb51ce4e0871ce5a5df: Status 404 returned error can't find the container with id e7b9fc4cbe06cd39457159e7cffdf437c8bf754a4fe1ebb51ce4e0871ce5a5df
Oct 30 01:27:18.444528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:18.444481    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx" event={"ID":"a9b1139d-7b79-4d77-b64b-677abcf63f0d","Type":"ContainerStarted","Data":"66086e5f7616a9a817842fbb111f82374b02b02a07a1c52ca3d2f18818e37c92"}
Oct 30 01:27:18.445022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:18.444555    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx" event={"ID":"a9b1139d-7b79-4d77-b64b-677abcf63f0d","Type":"ContainerStarted","Data":"e7b9fc4cbe06cd39457159e7cffdf437c8bf754a4fe1ebb51ce4e0871ce5a5df"}
Oct 30 01:27:18.445022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:18.444815    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:18.483467 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:18.483407    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx" podStartSLOduration=1.483390073 podStartE2EDuration="1.483390073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:18.482370205 +0000 UTC m=+2747.232025515" watchObservedRunningTime="2026-10-30 01:27:18.483390073 +0000 UTC m=+2747.233045467"
Oct 30 01:27:20.133469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.133430    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"]
Oct 30 01:27:20.138807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.138774    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"
Oct 30 01:27:20.153710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.153675    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c7bba9\""
Oct 30 01:27:20.154001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.153985    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"kube-root-ca.crt\""
Oct 30 01:27:20.154060 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.154018    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"openshift-service-ca.crt\""
Oct 30 01:27:20.163563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.163531    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"routing-overlay\""
Oct 30 01:27:20.164070 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.164041    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"agc-cred-812a725ec060\""
Oct 30 01:27:20.164070 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.164050    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"payload-processing-external-model-e2e--42876bb9-dockercfg-ktqp4\""
Oct 30 01:27:20.170197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.168624    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"]
Oct 30 01:27:20.258399 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.258340    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b53c06-1633-4a94-b567-13b404b5fb64-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf\" (UID: \"37b53c06-1633-4a94-b567-13b404b5fb64\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"
Oct 30 01:27:20.258645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.258585    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37b53c06-1633-4a94-b567-13b404b5fb64-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf\" (UID: \"37b53c06-1633-4a94-b567-13b404b5fb64\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"
Oct 30 01:27:20.258879 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.258853    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf\" (UID: \"37b53c06-1633-4a94-b567-13b404b5fb64\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"
Oct 30 01:27:20.258971 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.258900    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnk9q\" (UniqueName: \"kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-kube-api-access-dnk9q\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf\" (UID: \"37b53c06-1633-4a94-b567-13b404b5fb64\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"
Oct 30 01:27:20.359943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.359892    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf\" (UID: \"37b53c06-1633-4a94-b567-13b404b5fb64\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"
Oct 30 01:27:20.360168 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.359956    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnk9q\" (UniqueName: \"kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-kube-api-access-dnk9q\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf\" (UID: \"37b53c06-1633-4a94-b567-13b404b5fb64\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"
Oct 30 01:27:20.360168 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.360039    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b53c06-1633-4a94-b567-13b404b5fb64-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf\" (UID: \"37b53c06-1633-4a94-b567-13b404b5fb64\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"
Oct 30 01:27:20.360337 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.360152    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37b53c06-1633-4a94-b567-13b404b5fb64-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf\" (UID: \"37b53c06-1633-4a94-b567-13b404b5fb64\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"
Oct 30 01:27:20.360851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.360824    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37b53c06-1633-4a94-b567-13b404b5fb64-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf\" (UID: \"37b53c06-1633-4a94-b567-13b404b5fb64\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"
Oct 30 01:27:20.360961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.360906    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b53c06-1633-4a94-b567-13b404b5fb64-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf\" (UID: \"37b53c06-1633-4a94-b567-13b404b5fb64\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"
Oct 30 01:27:20.362793 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.362769    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf\" (UID: \"37b53c06-1633-4a94-b567-13b404b5fb64\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"
Oct 30 01:27:20.381315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.381272    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnk9q\" (UniqueName: \"kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-kube-api-access-dnk9q\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf\" (UID: \"37b53c06-1633-4a94-b567-13b404b5fb64\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"
Oct 30 01:27:20.454738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.454625    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"
Oct 30 01:27:20.772350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.772316    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"]
Oct 30 01:27:20.773933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.773899    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:27:20.773933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:20.773957    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24" podUID="3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:27:20.774254 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:27:20.774187    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b53c06_1633_4a94_b567_13b404b5fb64.slice/crio-78a2ceea8e55ffcb5e8e89a65a506e1d22b45ad73c08b2465ffa538556bdfb12 WatchSource:0}: Error finding container 78a2ceea8e55ffcb5e8e89a65a506e1d22b45ad73c08b2465ffa538556bdfb12: Status 404 returned error can't find the container with id 78a2ceea8e55ffcb5e8e89a65a506e1d22b45ad73c08b2465ffa538556bdfb12
Oct 30 01:27:21.355284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.355250    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 01:27:21.384813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.384777    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"
Oct 30 01:27:21.481421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.481381    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-kube-api-access-w4hts\" (UniqueName: \"kubernetes.io/projected/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-kube-api-access-w4hts\") pod \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") "
Oct 30 01:27:21.481904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.481875    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-data\") pod \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") "
Oct 30 01:27:21.483225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.483197    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-envoy\") pod \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") "
Oct 30 01:27:21.483432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.483265    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-workload-certs\") pod \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") "
Oct 30 01:27:21.483432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.483301    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-token\" (UniqueName: \"kubernetes.io/projected/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-token\") pod \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") "
Oct 30 01:27:21.483432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.483366    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-credential-socket\") pod \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") "
Oct 30 01:27:21.483432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.483399    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-podinfo\") pod \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") "
Oct 30 01:27:21.483647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.483446    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istiod-ca-cert\") pod \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") "
Oct 30 01:27:21.483647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.483486    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-workload-socket\") pod \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\" (UID: \"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79\") "
Oct 30 01:27:21.485318 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.485286    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-credential-socket" pod "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" (UID: "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:21.485523 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.485497    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-data" pod "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" (UID: "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:21.489563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.489531    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-workload-certs" pod "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" (UID: "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:21.490023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.490003    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istiod-ca-cert" pod "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" (UID: "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:21.493792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.492300    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-workload-socket" pod "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" (UID: "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:21.498387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.498258    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-token" pod "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" (UID: "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:21.498387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.498353    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-podinfo" pod "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" (UID: "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:27:21.499376 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.499348    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-kube-api-access-w4hts" pod "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" (UID: "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79"). InnerVolumeSpecName "kube-api-access-w4hts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:21.501806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.501774    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-envoy" pod "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" (UID: "3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:21.504466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.503449    4425 generic.go:356] "Generic (PLEG): container finished" podID="3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" containerID="e717335be880557407fe751ef248ff254376d4f1323e707e0d42a2135a255f66" exitCode=0
Oct 30 01:27:21.504466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.503626    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"
Oct 30 01:27:21.537019 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.536978    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"
Oct 30 01:27:21.537218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.537072    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"]
Oct 30 01:27:21.539342 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.539315    4425 cpu_manager.go:414] "containerMap: removing container" podUID="3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" containerName="istio-proxy"
Oct 30 01:27:21.539484 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.539346    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" containerName="istio-proxy"
Oct 30 01:27:21.539889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.539866    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" containerName="istio-proxy"
Oct 30 01:27:21.556098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.555722    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24" event={"ID":"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79","Type":"ContainerDied","Data":"e717335be880557407fe751ef248ff254376d4f1323e707e0d42a2135a255f66"}
Oct 30 01:27:21.556098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.555782    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"]
Oct 30 01:27:21.556098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.555838    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24" event={"ID":"3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79","Type":"ContainerDied","Data":"40a7f1e996f80e064f3f8c18ce94d9b0558e99a193ca1e44c64ff6b8f894620c"}
Oct 30 01:27:21.556098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.555857    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf" event={"ID":"37b53c06-1633-4a94-b567-13b404b5fb64","Type":"ContainerStarted","Data":"f5a0ebe83f3b534687a1c0c0951164d0be21040601988b9c37151fe2f31a0395"}
Oct 30 01:27:21.556098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.555894    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf" event={"ID":"37b53c06-1633-4a94-b567-13b404b5fb64","Type":"ContainerStarted","Data":"78a2ceea8e55ffcb5e8e89a65a506e1d22b45ad73c08b2465ffa538556bdfb12"}
Oct 30 01:27:21.556098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.555934    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:21.556098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.556095    4425 scope.go:122] "RemoveContainer" containerID="e717335be880557407fe751ef248ff254376d4f1323e707e0d42a2135a255f66"
Oct 30 01:27:21.569017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.568986    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"e2e-cfl-b02a01-provider-cred\""
Oct 30 01:27:21.577902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.577841    4425 scope.go:122] "RemoveContainer" containerID="e717335be880557407fe751ef248ff254376d4f1323e707e0d42a2135a255f66"
Oct 30 01:27:21.578315 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:21.578284    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e717335be880557407fe751ef248ff254376d4f1323e707e0d42a2135a255f66\": container with ID starting with e717335be880557407fe751ef248ff254376d4f1323e707e0d42a2135a255f66 not found: ID does not exist" containerID="e717335be880557407fe751ef248ff254376d4f1323e707e0d42a2135a255f66"
Oct 30 01:27:21.578436 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.578324    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e717335be880557407fe751ef248ff254376d4f1323e707e0d42a2135a255f66"} err="failed to get container status \"e717335be880557407fe751ef248ff254376d4f1323e707e0d42a2135a255f66\": rpc error: code = NotFound desc = could not find container \"e717335be880557407fe751ef248ff254376d4f1323e707e0d42a2135a255f66\": container with ID starting with e717335be880557407fe751ef248ff254376d4f1323e707e0d42a2135a255f66 not found: ID does not exist"
Oct 30 01:27:21.586120 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.586051    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf" podStartSLOduration=1.5860327939999999 podStartE2EDuration="1.586032794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:21.583613189 +0000 UTC m=+2750.333268499" watchObservedRunningTime="2026-10-30 01:27:21.586032794 +0000 UTC m=+2750.335688106"
Oct 30 01:27:21.586551 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.586530    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:21.586551 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.586552    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:21.586705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.586562    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:21.586705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.586571    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:21.586705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.586580    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:21.586705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.586589    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:21.586705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.586597    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:21.586705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.586605    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4hts\" (UniqueName: \"kubernetes.io/projected/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-kube-api-access-w4hts\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:21.586705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.586614    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:21.632677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.632441    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"]
Oct 30 01:27:21.639144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.639084    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-w7w24"]
Oct 30 01:27:21.688731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.688577    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:21.688731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.688663    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:21.689031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.688745    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s95zw\" (UniqueName: \"kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-kube-api-access-s95zw\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:21.689031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.688813    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:21.689031 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:21.689008    4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c7bba9/agc-cred-812a725ec060: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:21.689031 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:21.689025    4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:21.689271 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:21.689097    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-provider-credentials podName:37b53c06-1633-4a94-b567-13b404b5fb64 nodeName:}" failed. No retries permitted until 2026-10-30 01:27:22.189073927 +0000 UTC m=+2750.938729231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf" (UID: "37b53c06-1633-4a94-b567-13b404b5fb64") : secret "agc-cred-812a725ec060" not found
Oct 30 01:27:21.790139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.790100    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:21.790348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.790195    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:21.790348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.790251    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s95zw\" (UniqueName: \"kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-kube-api-access-s95zw\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:21.790348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.790302    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:21.790554 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:21.790392    4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c7bba9/agc-cred-812a725ec060: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:21.790554 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:21.790424    4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:21.790554 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:21.790501    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-provider-credentials podName:6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35 nodeName:}" failed. No retries permitted until 2026-10-30 01:27:22.290477564 +0000 UTC m=+2751.040132868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc" (UID: "6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35") : secret "agc-cred-812a725ec060" not found
Oct 30 01:27:21.790887 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.790866    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:21.791037 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.791018    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:21.812125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:21.812090    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s95zw\" (UniqueName: \"kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-kube-api-access-s95zw\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:22.196708 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:22.196673    4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c7bba9/agc-cred-812a725ec060: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:22.196708 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:22.196703    4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:22.196972 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:22.196760    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-provider-credentials podName:37b53c06-1633-4a94-b567-13b404b5fb64 nodeName:}" failed. No retries permitted until 2026-10-30 01:27:23.196744633 +0000 UTC m=+2751.946399921 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf" (UID: "37b53c06-1633-4a94-b567-13b404b5fb64") : secret "agc-cred-812a725ec060" not found
Oct 30 01:27:22.298603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.298555    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:22.298789 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:22.298695    4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c7bba9/agc-cred-812a725ec060: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:22.298789 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:22.298721    4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:22.298789 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:22.298779    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-provider-credentials podName:6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35 nodeName:}" failed. No retries permitted until 2026-10-30 01:27:23.29876392 +0000 UTC m=+2752.048419208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc" (UID: "6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35") : secret "agc-cred-812a725ec060" not found
Oct 30 01:27:22.306234 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.306196    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"]
Oct 30 01:27:22.344211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.344133    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"]
Oct 30 01:27:22.351501 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.351471    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:27:22.369881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.369843    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"]
Oct 30 01:27:22.390405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.390371    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"
Oct 30 01:27:22.502119 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.501992    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq\" (UID: \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:27:22.502119 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.502047    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq\" (UID: \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:27:22.502470 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.502387    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq\" (UID: \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:27:22.502539 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.502515    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlhtg\" (UniqueName: \"kubernetes.io/projected/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-kube-api-access-xlhtg\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq\" (UID: \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:27:22.603693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.603647    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq\" (UID: \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:27:22.603693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.603701    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq\" (UID: \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:27:22.603996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.603915    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq\" (UID: \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:27:22.604064 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.604035    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlhtg\" (UniqueName: \"kubernetes.io/projected/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-kube-api-access-xlhtg\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq\" (UID: \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:27:22.604587 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.604561    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq\" (UID: \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:27:22.604780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.604752    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq\" (UID: \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:27:22.606759 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.606696    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq\" (UID: \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:27:22.625843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.625809    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlhtg\" (UniqueName: \"kubernetes.io/projected/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-kube-api-access-xlhtg\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq\" (UID: \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:27:22.669432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.669396    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:27:22.980727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:22.980692    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"]
Oct 30 01:27:22.982080 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:27:22.982055    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf311ac5c_a3f8_4167_8a8f_ce2b03b66197.slice/crio-033ff0a368551206db35cdf02588f4a8ada8a8c3e100c03df5580202269e9250 WatchSource:0}: Error finding container 033ff0a368551206db35cdf02588f4a8ada8a8c3e100c03df5580202269e9250: Status 404 returned error can't find the container with id 033ff0a368551206db35cdf02588f4a8ada8a8c3e100c03df5580202269e9250
Oct 30 01:27:23.214698 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:23.214665    4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c7bba9/agc-cred-812a725ec060: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:23.214698 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:23.214691    4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:23.214985 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:23.214746    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-provider-credentials podName:37b53c06-1633-4a94-b567-13b404b5fb64 nodeName:}" failed. No retries permitted until 2026-10-30 01:27:25.214731148 +0000 UTC m=+2753.964386436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf" (UID: "37b53c06-1633-4a94-b567-13b404b5fb64") : secret "agc-cred-812a725ec060" not found
Oct 30 01:27:23.315873 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:23.315830    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:23.316064 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:23.315984    4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c7bba9/agc-cred-812a725ec060: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:23.316064 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:23.316011    4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:23.316220 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:23.316075    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-provider-credentials podName:6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35 nodeName:}" failed. No retries permitted until 2026-10-30 01:27:25.316058697 +0000 UTC m=+2754.065713984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc" (UID: "6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35") : secret "agc-cred-812a725ec060" not found
Oct 30 01:27:23.424971 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:23.424925    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79" path="/var/lib/kubelet/pods/3ba50e09-63b0-4d2e-b18e-f8fa1a30ef79/volumes"
Oct 30 01:27:23.545996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:23.545949    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq" event={"ID":"f311ac5c-a3f8-4167-8a8f-ce2b03b66197","Type":"ContainerStarted","Data":"b0e384391f6a27a487400c09e42a796b1e6dd9d6995bbbc388d15c3acb6ee69f"}
Oct 30 01:27:23.546180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:23.546168    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:27:23.546224 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:23.546206    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq" event={"ID":"f311ac5c-a3f8-4167-8a8f-ce2b03b66197","Type":"ContainerStarted","Data":"033ff0a368551206db35cdf02588f4a8ada8a8c3e100c03df5580202269e9250"}
Oct 30 01:27:23.585651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:23.585537    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq" podStartSLOduration=1.5855221959999999 podStartE2EDuration="1.585522196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:23.584548351 +0000 UTC m=+2752.334203664" watchObservedRunningTime="2026-10-30 01:27:23.585522196 +0000 UTC m=+2752.335177504"
Oct 30 01:27:25.243596 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:25.243554    4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c7bba9/agc-cred-812a725ec060: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:25.243596 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:25.243586    4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:25.244240 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:25.243637    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-provider-credentials podName:37b53c06-1633-4a94-b567-13b404b5fb64 nodeName:}" failed. No retries permitted until 2026-10-30 01:27:29.24362397 +0000 UTC m=+2757.993279258 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf" (UID: "37b53c06-1633-4a94-b567-13b404b5fb64") : secret "agc-cred-812a725ec060" not found
Oct 30 01:27:25.344828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:25.344774    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:25.345038 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:25.344918    4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c7bba9/agc-cred-812a725ec060: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:25.345038 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:25.344948    4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:25.345038 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:25.345007    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-provider-credentials podName:6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35 nodeName:}" failed. No retries permitted until 2026-10-30 01:27:29.344992184 +0000 UTC m=+2758.094647470 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc" (UID: "6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35") : secret "agc-cred-812a725ec060" not found
Oct 30 01:27:28.448350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:28.448312    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:28.546748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:28.546704    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf" podUID="37b53c06-1633-4a94-b567-13b404b5fb64" containerName="payload-processing" containerID="cri-o://f5a0ebe83f3b534687a1c0c0951164d0be21040601988b9c37151fe2f31a0395" gracePeriod=25
Oct 30 01:27:28.840352 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:28.840319    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"
Oct 30 01:27:28.896726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:28.896686    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-kube-api-access-dnk9q\" (UniqueName: \"kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-kube-api-access-dnk9q\") pod \"37b53c06-1633-4a94-b567-13b404b5fb64\" (UID: \"37b53c06-1633-4a94-b567-13b404b5fb64\") "
Oct 30 01:27:28.896931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:28.896782    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37b53c06-1633-4a94-b567-13b404b5fb64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b53c06-1633-4a94-b567-13b404b5fb64-plugins-config-volume\") pod \"37b53c06-1633-4a94-b567-13b404b5fb64\" (UID: \"37b53c06-1633-4a94-b567-13b404b5fb64\") "
Oct 30 01:27:28.896931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:28.896861    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-provider-credentials\" (UniqueName: \"kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-provider-credentials\") pod \"37b53c06-1633-4a94-b567-13b404b5fb64\" (UID: \"37b53c06-1633-4a94-b567-13b404b5fb64\") "
Oct 30 01:27:28.897088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:28.897003    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37b53c06-1633-4a94-b567-13b404b5fb64-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37b53c06-1633-4a94-b567-13b404b5fb64-routing-overlay-volume\") pod \"37b53c06-1633-4a94-b567-13b404b5fb64\" (UID: \"37b53c06-1633-4a94-b567-13b404b5fb64\") "
Oct 30 01:27:28.897480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:28.897416    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b53c06-1633-4a94-b567-13b404b5fb64-plugins-config-volume" pod "37b53c06-1633-4a94-b567-13b404b5fb64" (UID: "37b53c06-1633-4a94-b567-13b404b5fb64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:28.897885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:28.897853    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b53c06-1633-4a94-b567-13b404b5fb64-routing-overlay-volume" pod "37b53c06-1633-4a94-b567-13b404b5fb64" (UID: "37b53c06-1633-4a94-b567-13b404b5fb64"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:28.898463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:28.898433    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37b53c06-1633-4a94-b567-13b404b5fb64-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:28.898463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:28.898466    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37b53c06-1633-4a94-b567-13b404b5fb64-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:28.899582 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:28.899546    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-kube-api-access-dnk9q" pod "37b53c06-1633-4a94-b567-13b404b5fb64" (UID: "37b53c06-1633-4a94-b567-13b404b5fb64"). InnerVolumeSpecName "kube-api-access-dnk9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:28.899868 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:28.899838    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-provider-credentials" pod "37b53c06-1633-4a94-b567-13b404b5fb64" (UID: "37b53c06-1633-4a94-b567-13b404b5fb64"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:29.000112 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:29.000016    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnk9q\" (UniqueName: \"kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-kube-api-access-dnk9q\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:29.000112 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:29.000064    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37b53c06-1633-4a94-b567-13b404b5fb64-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:29.405508 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:29.405464    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:29.405734 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:29.405620    4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c7bba9/agc-cred-812a725ec060: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:29.405734 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:29.405648    4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc: secret "agc-cred-812a725ec060" not found
Oct 30 01:27:29.405734 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:29.405705    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-provider-credentials podName:6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35 nodeName:}" failed. No retries permitted until 2026-10-30 01:27:37.405691552 +0000 UTC m=+2766.155346840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc" (UID: "6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35") : secret "agc-cred-812a725ec060" not found
Oct 30 01:27:29.456753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:29.456716    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:29.555796 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:29.555756    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"]
Oct 30 01:27:29.599992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:29.599955    4425 generic.go:356] "Generic (PLEG): container finished" podID="37b53c06-1633-4a94-b567-13b404b5fb64" containerID="f5a0ebe83f3b534687a1c0c0951164d0be21040601988b9c37151fe2f31a0395" exitCode=0
Oct 30 01:27:29.600248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:29.600022    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"
Oct 30 01:27:29.600248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:29.600038    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf" event={"ID":"37b53c06-1633-4a94-b567-13b404b5fb64","Type":"ContainerDied","Data":"f5a0ebe83f3b534687a1c0c0951164d0be21040601988b9c37151fe2f31a0395"}
Oct 30 01:27:29.600248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:29.600093    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf" event={"ID":"37b53c06-1633-4a94-b567-13b404b5fb64","Type":"ContainerDied","Data":"78a2ceea8e55ffcb5e8e89a65a506e1d22b45ad73c08b2465ffa538556bdfb12"}
Oct 30 01:27:29.600248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:29.600117    4425 scope.go:122] "RemoveContainer" containerID="f5a0ebe83f3b534687a1c0c0951164d0be21040601988b9c37151fe2f31a0395"
Oct 30 01:27:29.615106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:29.615083    4425 scope.go:122] "RemoveContainer" containerID="f5a0ebe83f3b534687a1c0c0951164d0be21040601988b9c37151fe2f31a0395"
Oct 30 01:27:29.615460 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:29.615435    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a0ebe83f3b534687a1c0c0951164d0be21040601988b9c37151fe2f31a0395\": container with ID starting with f5a0ebe83f3b534687a1c0c0951164d0be21040601988b9c37151fe2f31a0395 not found: ID does not exist" containerID="f5a0ebe83f3b534687a1c0c0951164d0be21040601988b9c37151fe2f31a0395"
Oct 30 01:27:29.615560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:29.615469    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a0ebe83f3b534687a1c0c0951164d0be21040601988b9c37151fe2f31a0395"} err="failed to get container status \"f5a0ebe83f3b534687a1c0c0951164d0be21040601988b9c37151fe2f31a0395\": rpc error: code = NotFound desc = could not find container \"f5a0ebe83f3b534687a1c0c0951164d0be21040601988b9c37151fe2f31a0395\": container with ID starting with f5a0ebe83f3b534687a1c0c0951164d0be21040601988b9c37151fe2f31a0395 not found: ID does not exist"
Oct 30 01:27:29.647095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:29.647054    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"]
Oct 30 01:27:29.653512 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:29.653476    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-958dgpcvf"]
Oct 30 01:27:31.185586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.185549    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"]
Oct 30 01:27:31.186602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.186566    4425 cpu_manager.go:414] "containerMap: removing container" podUID="37b53c06-1633-4a94-b567-13b404b5fb64" containerName="payload-processing"
Oct 30 01:27:31.186602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.186598    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37b53c06-1633-4a94-b567-13b404b5fb64" containerName="payload-processing"
Oct 30 01:27:31.186838 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.186823    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="37b53c06-1633-4a94-b567-13b404b5fb64" containerName="payload-processing"
Oct 30 01:27:31.192122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.192095    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"
Oct 30 01:27:31.205410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.205381    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:27:31.215092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.215037    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:27:31.219940 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.219903    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"]
Oct 30 01:27:31.337903 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.337863    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7530d38e-q29fk\" (UID: \"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"
Oct 30 01:27:31.338085 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.338026    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7530d38e-q29fk\" (UID: \"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"
Oct 30 01:27:31.338085 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.338078    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bxgb\" (UniqueName: \"kubernetes.io/projected/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-kube-api-access-9bxgb\") pod \"maas-api-revoke-sub-7530d38e-q29fk\" (UID: \"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"
Oct 30 01:27:31.425526 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.425490    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37b53c06-1633-4a94-b567-13b404b5fb64" path="/var/lib/kubelet/pods/37b53c06-1633-4a94-b567-13b404b5fb64/volumes"
Oct 30 01:27:31.439111 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.439023    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7530d38e-q29fk\" (UID: \"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"
Oct 30 01:27:31.439333 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.439135    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7530d38e-q29fk\" (UID: \"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"
Oct 30 01:27:31.439333 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.439218    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bxgb\" (UniqueName: \"kubernetes.io/projected/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-kube-api-access-9bxgb\") pod \"maas-api-revoke-sub-7530d38e-q29fk\" (UID: \"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"
Oct 30 01:27:31.439811 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.439789    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7530d38e-q29fk\" (UID: \"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"
Oct 30 01:27:31.450748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.450709    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:27:31.461112 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.461075    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bxgb\" (UniqueName: \"kubernetes.io/projected/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-kube-api-access-9bxgb\") pod \"maas-api-revoke-sub-7530d38e-q29fk\" (UID: \"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"
Oct 30 01:27:31.462264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.462229    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7530d38e-q29fk\" (UID: \"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"
Oct 30 01:27:31.518462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.518430    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:27:31.527971 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.527920    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"
Oct 30 01:27:31.839073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:31.839035    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"]
Oct 30 01:27:31.839492 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:27:31.839468    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e582e9_cc6a_4396_ad0d_960cbe9b89f1.slice/crio-3803dfb5b9ce5835bb798b45d3a8d9dae614401ecd5090e5cfeb4d2a48b6e342 WatchSource:0}: Error finding container 3803dfb5b9ce5835bb798b45d3a8d9dae614401ecd5090e5cfeb4d2a48b6e342: Status 404 returned error can't find the container with id 3803dfb5b9ce5835bb798b45d3a8d9dae614401ecd5090e5cfeb4d2a48b6e342
Oct 30 01:27:32.305072 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:32.305028    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"]
Oct 30 01:27:32.708384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:32.708277    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk" event={"ID":"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1","Type":"ContainerStarted","Data":"e441f1828d45cc197dcac0f49c8da10bc94ab42397dfcdb9e2e8b32c964e7f48"}
Oct 30 01:27:32.710061 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:32.710009    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk" event={"ID":"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1","Type":"ContainerStarted","Data":"3803dfb5b9ce5835bb798b45d3a8d9dae614401ecd5090e5cfeb4d2a48b6e342"}
Oct 30 01:27:32.754111 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:32.754053    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk" podStartSLOduration=1.754033323 podStartE2EDuration="1.754033323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:32.750767518 +0000 UTC m=+2761.500422846" watchObservedRunningTime="2026-10-30 01:27:32.754033323 +0000 UTC m=+2761.503688631"
Oct 30 01:27:33.588223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.588177    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"]
Oct 30 01:27:33.593263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.593225    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:27:33.612863 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.612829    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"]
Oct 30 01:27:33.672301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.672259    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr\" (UID: \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:27:33.672516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.672333    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njb4x\" (UniqueName: \"kubernetes.io/projected/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-kube-api-access-njb4x\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr\" (UID: \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:27:33.672516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.672420    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr\" (UID: \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:27:33.672661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.672638    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr\" (UID: \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:27:33.718150 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.718114    4425 generic.go:356] "Generic (PLEG): container finished" podID="a2e582e9-cc6a-4396-ad0d-960cbe9b89f1" containerID="e441f1828d45cc197dcac0f49c8da10bc94ab42397dfcdb9e2e8b32c964e7f48" exitCode=0
Oct 30 01:27:33.718415 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.718187    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk" event={"ID":"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1","Type":"ContainerDied","Data":"e441f1828d45cc197dcac0f49c8da10bc94ab42397dfcdb9e2e8b32c964e7f48"}
Oct 30 01:27:33.773853 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.773812    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr\" (UID: \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:27:33.774049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.774025    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr\" (UID: \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:27:33.774104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.774066    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njb4x\" (UniqueName: \"kubernetes.io/projected/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-kube-api-access-njb4x\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr\" (UID: \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:27:33.774186 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.774102    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr\" (UID: \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:27:33.774740 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.774712    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr\" (UID: \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:27:33.774957 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.774938    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr\" (UID: \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:27:33.777018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.776991    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr\" (UID: \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:27:33.797961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.797922    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njb4x\" (UniqueName: \"kubernetes.io/projected/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-kube-api-access-njb4x\") pod \"payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr\" (UID: \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\") " pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:27:33.909654 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:33.909551    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:27:34.237237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.237192    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"]
Oct 30 01:27:34.237795 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:27:34.237765    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0a077a_c6fa_43a3_9028_a0d87178dc6f.slice/crio-e5d945606e735d36e72d7d72adfa951f98c93d4f54133d25bace47cd1ec76159 WatchSource:0}: Error finding container e5d945606e735d36e72d7d72adfa951f98c93d4f54133d25bace47cd1ec76159: Status 404 returned error can't find the container with id e5d945606e735d36e72d7d72adfa951f98c93d4f54133d25bace47cd1ec76159
Oct 30 01:27:34.556517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.556455    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt" podUID="2a78699a-583f-4438-a5f4-ce704e517df3" containerName="payload-processing" containerID="cri-o://c695b153c9c7f536d4f8d531cc5825bab1a4e110f944163256af91c72f3039f9" gracePeriod=25
Oct 30 01:27:34.557850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.557822    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:27:34.578172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.578128    4425 scope.go:122] "RemoveContainer" containerID="b9a497d26f8e46b550fcfa3123eb19496841d46ab2e7a63f79c4b205b6890e1a"
Oct 30 01:27:34.652088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.652041    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"]
Oct 30 01:27:34.652682 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:34.652648    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc" podUID="6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35"
Oct 30 01:27:34.676632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.676596    4425 scope.go:122] "RemoveContainer" containerID="3c602adc2609262976738c60536fa46aa4187507e8ef2f2c237c9702d657b667"
Oct 30 01:27:34.690041 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.690010    4425 scope.go:122] "RemoveContainer" containerID="2d7eb593420e07b4ca7546ea4278d94702b55e8ad0938d0826743f1d6507fbf3"
Oct 30 01:27:34.702638 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.702607    4425 scope.go:122] "RemoveContainer" containerID="fd15cc29229eaeb3d694ecf9f7efe71b5675c82f67e0acbd401e1145a5a655a6"
Oct 30 01:27:34.716045 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.716023    4425 scope.go:122] "RemoveContainer" containerID="687f2cc89e63d7babc30a1752307764af5e0133e3fe8984b7c45095c89d7bae6"
Oct 30 01:27:34.729986 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.729959    4425 scope.go:122] "RemoveContainer" containerID="66b01135e9e4ad433f3228cd37b3b68c3f92daa06586f1a017cb007ceb91ee9c"
Oct 30 01:27:34.732786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.732754    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr" event={"ID":"0e0a077a-c6fa-43a3-9028-a0d87178dc6f","Type":"ContainerStarted","Data":"246a2b1646f0d5b5b233ece406ffde9fea7aba1237cd23b1c73797de14f5e588"}
Oct 30 01:27:34.733014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.732818    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr" event={"ID":"0e0a077a-c6fa-43a3-9028-a0d87178dc6f","Type":"ContainerStarted","Data":"e5d945606e735d36e72d7d72adfa951f98c93d4f54133d25bace47cd1ec76159"}
Oct 30 01:27:34.733014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.732850    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:27:34.736339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.736308    4425 generic.go:356] "Generic (PLEG): container finished" podID="2a78699a-583f-4438-a5f4-ce704e517df3" containerID="c695b153c9c7f536d4f8d531cc5825bab1a4e110f944163256af91c72f3039f9" exitCode=0
Oct 30 01:27:34.736499 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.736382    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt" event={"ID":"2a78699a-583f-4438-a5f4-ce704e517df3","Type":"ContainerDied","Data":"c695b153c9c7f536d4f8d531cc5825bab1a4e110f944163256af91c72f3039f9"}
Oct 30 01:27:34.737747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.737720    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:34.749242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.749075    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:34.787917 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.785996    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr" podStartSLOduration=1.785974972 podStartE2EDuration="1.785974972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:34.779646619 +0000 UTC m=+2763.529301929" watchObservedRunningTime="2026-10-30 01:27:34.785974972 +0000 UTC m=+2763.535630278"
Oct 30 01:27:34.888698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.888653    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-plugins-config-volume\") pod \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") "
Oct 30 01:27:34.888869 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.888816    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-routing-overlay-volume\") pod \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") "
Oct 30 01:27:34.888966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.888949    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-kube-api-access-s95zw\" (UniqueName: \"kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-kube-api-access-s95zw\") pod \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\" (UID: \"6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35\") "
Oct 30 01:27:34.891203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.890696    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-routing-overlay-volume" pod "6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35" (UID: "6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:34.891203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.891054    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-plugins-config-volume" pod "6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35" (UID: "6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:34.892995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.892962    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:34.892995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.892993    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:34.905230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.896867    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-kube-api-access-s95zw" pod "6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35" (UID: "6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35"). InnerVolumeSpecName "kube-api-access-s95zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:34.905230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.899882    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:27:34.947110 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.947077    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"
Oct 30 01:27:34.994359 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.994327    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a78699a-583f-4438-a5f4-ce704e517df3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a78699a-583f-4438-a5f4-ce704e517df3-plugins-config-volume\") pod \"2a78699a-583f-4438-a5f4-ce704e517df3\" (UID: \"2a78699a-583f-4438-a5f4-ce704e517df3\") "
Oct 30 01:27:34.994693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.994670    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a78699a-583f-4438-a5f4-ce704e517df3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a78699a-583f-4438-a5f4-ce704e517df3-provider-credentials\") pod \"2a78699a-583f-4438-a5f4-ce704e517df3\" (UID: \"2a78699a-583f-4438-a5f4-ce704e517df3\") "
Oct 30 01:27:34.994837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.994808    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a78699a-583f-4438-a5f4-ce704e517df3-plugins-config-volume" pod "2a78699a-583f-4438-a5f4-ce704e517df3" (UID: "2a78699a-583f-4438-a5f4-ce704e517df3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:34.994997 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.994971    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a78699a-583f-4438-a5f4-ce704e517df3-kube-api-access-x245f\" (UniqueName: \"kubernetes.io/projected/2a78699a-583f-4438-a5f4-ce704e517df3-kube-api-access-x245f\") pod \"2a78699a-583f-4438-a5f4-ce704e517df3\" (UID: \"2a78699a-583f-4438-a5f4-ce704e517df3\") "
Oct 30 01:27:34.995195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.995026    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a78699a-583f-4438-a5f4-ce704e517df3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a78699a-583f-4438-a5f4-ce704e517df3-routing-overlay-volume\") pod \"2a78699a-583f-4438-a5f4-ce704e517df3\" (UID: \"2a78699a-583f-4438-a5f4-ce704e517df3\") "
Oct 30 01:27:34.995748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.995725    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s95zw\" (UniqueName: \"kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-kube-api-access-s95zw\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:34.995748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.995747    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a78699a-583f-4438-a5f4-ce704e517df3-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:34.996074 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.996050    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a78699a-583f-4438-a5f4-ce704e517df3-routing-overlay-volume" pod "2a78699a-583f-4438-a5f4-ce704e517df3" (UID: "2a78699a-583f-4438-a5f4-ce704e517df3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:34.997358 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.997333    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a78699a-583f-4438-a5f4-ce704e517df3-provider-credentials" pod "2a78699a-583f-4438-a5f4-ce704e517df3" (UID: "2a78699a-583f-4438-a5f4-ce704e517df3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:34.998191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:34.997961    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a78699a-583f-4438-a5f4-ce704e517df3-kube-api-access-x245f" pod "2a78699a-583f-4438-a5f4-ce704e517df3" (UID: "2a78699a-583f-4438-a5f4-ce704e517df3"). InnerVolumeSpecName "kube-api-access-x245f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:35.096622 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.096525    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-kube-api-access-9bxgb\" (UniqueName: \"kubernetes.io/projected/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-kube-api-access-9bxgb\") pod \"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1\" (UID: \"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1\") "
Oct 30 01:27:35.096838 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.096815    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-maas-api-xks-ca\") pod \"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1\" (UID: \"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1\") "
Oct 30 01:27:35.096914 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.096891    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-maas-api-openshift-ca\") pod \"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1\" (UID: \"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1\") "
Oct 30 01:27:35.097263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.097235    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-maas-api-openshift-ca" pod "a2e582e9-cc6a-4396-ad0d-960cbe9b89f1" (UID: "a2e582e9-cc6a-4396-ad0d-960cbe9b89f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:35.097476 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.097462    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:35.097528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.097479    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a78699a-583f-4438-a5f4-ce704e517df3-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:35.097528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.097491    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x245f\" (UniqueName: \"kubernetes.io/projected/2a78699a-583f-4438-a5f4-ce704e517df3-kube-api-access-x245f\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:35.097528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.097502    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a78699a-583f-4438-a5f4-ce704e517df3-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:35.098857 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.098834    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-kube-api-access-9bxgb" pod "a2e582e9-cc6a-4396-ad0d-960cbe9b89f1" (UID: "a2e582e9-cc6a-4396-ad0d-960cbe9b89f1"). InnerVolumeSpecName "kube-api-access-9bxgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:35.098919 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.098890    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-maas-api-xks-ca" pod "a2e582e9-cc6a-4396-ad0d-960cbe9b89f1" (UID: "a2e582e9-cc6a-4396-ad0d-960cbe9b89f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:27:35.199065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.199019    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:35.199065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.199057    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bxgb\" (UniqueName: \"kubernetes.io/projected/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1-kube-api-access-9bxgb\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:35.747709 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.747614    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"
Oct 30 01:27:35.748173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.747622    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt" event={"ID":"2a78699a-583f-4438-a5f4-ce704e517df3","Type":"ContainerDied","Data":"20a3a734cc638d8252762afae17043cc916654d1f6472aa0148496f2699174ed"}
Oct 30 01:27:35.748173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.747792    4425 scope.go:122] "RemoveContainer" containerID="c695b153c9c7f536d4f8d531cc5825bab1a4e110f944163256af91c72f3039f9"
Oct 30 01:27:35.749514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.749490    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"
Oct 30 01:27:35.749657 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.749513    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk" event={"ID":"a2e582e9-cc6a-4396-ad0d-960cbe9b89f1","Type":"ContainerDied","Data":"3803dfb5b9ce5835bb798b45d3a8d9dae614401ecd5090e5cfeb4d2a48b6e342"}
Oct 30 01:27:35.749657 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.749562    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3803dfb5b9ce5835bb798b45d3a8d9dae614401ecd5090e5cfeb4d2a48b6e342"
Oct 30 01:27:35.749938 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.749918    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"
Oct 30 01:27:35.823543 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.823500    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"]
Oct 30 01:27:35.828658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.828615    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-66b968tnc"]
Oct 30 01:27:35.856398 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.856352    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"]
Oct 30 01:27:35.860729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.860698    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6mmwzt"]
Oct 30 01:27:35.910688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:35.910643    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:37.305779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.305730    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8" podUID="51f680d6-14c5-4716-a142-f0cc9235f79e" containerName="payload-processing" containerID="cri-o://3a627ab8961e64c4bfa69fdf32fc5fc8da917213683b424f1542cd1be367df1d" gracePeriod=25
Oct 30 01:27:37.426203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.426133    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a78699a-583f-4438-a5f4-ce704e517df3" path="/var/lib/kubelet/pods/2a78699a-583f-4438-a5f4-ce704e517df3/volumes"
Oct 30 01:27:37.426690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.426675    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35" path="/var/lib/kubelet/pods/6689c6d8-e7b5-4840-ad0d-c8c7bbcc9c35/volumes"
Oct 30 01:27:37.560092 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:37.560005    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:27:37.613692 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.613659    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:37.736483 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.736430    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51f680d6-14c5-4716-a142-f0cc9235f79e-kube-api-access-572ft\" (UniqueName: \"kubernetes.io/projected/51f680d6-14c5-4716-a142-f0cc9235f79e-kube-api-access-572ft\") pod \"51f680d6-14c5-4716-a142-f0cc9235f79e\" (UID: \"51f680d6-14c5-4716-a142-f0cc9235f79e\") "
Oct 30 01:27:37.736661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.736553    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51f680d6-14c5-4716-a142-f0cc9235f79e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51f680d6-14c5-4716-a142-f0cc9235f79e-routing-overlay-volume\") pod \"51f680d6-14c5-4716-a142-f0cc9235f79e\" (UID: \"51f680d6-14c5-4716-a142-f0cc9235f79e\") "
Oct 30 01:27:37.736661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.736615    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51f680d6-14c5-4716-a142-f0cc9235f79e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/51f680d6-14c5-4716-a142-f0cc9235f79e-provider-credentials\") pod \"51f680d6-14c5-4716-a142-f0cc9235f79e\" (UID: \"51f680d6-14c5-4716-a142-f0cc9235f79e\") "
Oct 30 01:27:37.736661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.736651    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51f680d6-14c5-4716-a142-f0cc9235f79e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f680d6-14c5-4716-a142-f0cc9235f79e-plugins-config-volume\") pod \"51f680d6-14c5-4716-a142-f0cc9235f79e\" (UID: \"51f680d6-14c5-4716-a142-f0cc9235f79e\") "
Oct 30 01:27:37.737068 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.737036    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f680d6-14c5-4716-a142-f0cc9235f79e-routing-overlay-volume" pod "51f680d6-14c5-4716-a142-f0cc9235f79e" (UID: "51f680d6-14c5-4716-a142-f0cc9235f79e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:37.737244 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.737097    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f680d6-14c5-4716-a142-f0cc9235f79e-plugins-config-volume" pod "51f680d6-14c5-4716-a142-f0cc9235f79e" (UID: "51f680d6-14c5-4716-a142-f0cc9235f79e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:37.737335 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.737318    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51f680d6-14c5-4716-a142-f0cc9235f79e-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:37.737382 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.737341    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/51f680d6-14c5-4716-a142-f0cc9235f79e-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:37.739502 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.739481    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f680d6-14c5-4716-a142-f0cc9235f79e-kube-api-access-572ft" pod "51f680d6-14c5-4716-a142-f0cc9235f79e" (UID: "51f680d6-14c5-4716-a142-f0cc9235f79e"). InnerVolumeSpecName "kube-api-access-572ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:37.739587 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.739515    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f680d6-14c5-4716-a142-f0cc9235f79e-provider-credentials" pod "51f680d6-14c5-4716-a142-f0cc9235f79e" (UID: "51f680d6-14c5-4716-a142-f0cc9235f79e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:37.769595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.769557    4425 generic.go:356] "Generic (PLEG): container finished" podID="51f680d6-14c5-4716-a142-f0cc9235f79e" containerID="3a627ab8961e64c4bfa69fdf32fc5fc8da917213683b424f1542cd1be367df1d" exitCode=0
Oct 30 01:27:37.769825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.769604    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"
Oct 30 01:27:37.769825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.769642    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8" event={"ID":"51f680d6-14c5-4716-a142-f0cc9235f79e","Type":"ContainerDied","Data":"3a627ab8961e64c4bfa69fdf32fc5fc8da917213683b424f1542cd1be367df1d"}
Oct 30 01:27:37.769825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.769706    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8" event={"ID":"51f680d6-14c5-4716-a142-f0cc9235f79e","Type":"ContainerDied","Data":"31da0f8b31c6633d6e2850379e121d355b989df33cf1f6447c6b6a7050b81f3b"}
Oct 30 01:27:37.769825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.769738    4425 scope.go:122] "RemoveContainer" containerID="3a627ab8961e64c4bfa69fdf32fc5fc8da917213683b424f1542cd1be367df1d"
Oct 30 01:27:37.781963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.781939    4425 scope.go:122] "RemoveContainer" containerID="3a627ab8961e64c4bfa69fdf32fc5fc8da917213683b424f1542cd1be367df1d"
Oct 30 01:27:37.782361 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:37.782330    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a627ab8961e64c4bfa69fdf32fc5fc8da917213683b424f1542cd1be367df1d\": container with ID starting with 3a627ab8961e64c4bfa69fdf32fc5fc8da917213683b424f1542cd1be367df1d not found: ID does not exist" containerID="3a627ab8961e64c4bfa69fdf32fc5fc8da917213683b424f1542cd1be367df1d"
Oct 30 01:27:37.782445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.782376    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a627ab8961e64c4bfa69fdf32fc5fc8da917213683b424f1542cd1be367df1d"} err="failed to get container status \"3a627ab8961e64c4bfa69fdf32fc5fc8da917213683b424f1542cd1be367df1d\": rpc error: code = NotFound desc = could not find container \"3a627ab8961e64c4bfa69fdf32fc5fc8da917213683b424f1542cd1be367df1d\": container with ID starting with 3a627ab8961e64c4bfa69fdf32fc5fc8da917213683b424f1542cd1be367df1d not found: ID does not exist"
Oct 30 01:27:37.817247 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.817125    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"]
Oct 30 01:27:37.823973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.823937    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-6fc8qcmn8"]
Oct 30 01:27:37.838418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.838376    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-572ft\" (UniqueName: \"kubernetes.io/projected/51f680d6-14c5-4716-a142-f0cc9235f79e-kube-api-access-572ft\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:37.838418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:37.838414    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/51f680d6-14c5-4716-a142-f0cc9235f79e-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:39.425170 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:39.425106    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51f680d6-14c5-4716-a142-f0cc9235f79e" path="/var/lib/kubelet/pods/51f680d6-14c5-4716-a142-f0cc9235f79e/volumes"
Oct 30 01:27:41.393414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.393363    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"]
Oct 30 01:27:41.394168 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.394139    4425 cpu_manager.go:414] "containerMap: removing container" podUID="2a78699a-583f-4438-a5f4-ce704e517df3" containerName="payload-processing"
Oct 30 01:27:41.394239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.394170    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a78699a-583f-4438-a5f4-ce704e517df3" containerName="payload-processing"
Oct 30 01:27:41.394239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.394182    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a2e582e9-cc6a-4396-ad0d-960cbe9b89f1" containerName="revoke-keys"
Oct 30 01:27:41.394239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.394188    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e582e9-cc6a-4396-ad0d-960cbe9b89f1" containerName="revoke-keys"
Oct 30 01:27:41.394239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.394208    4425 cpu_manager.go:414] "containerMap: removing container" podUID="51f680d6-14c5-4716-a142-f0cc9235f79e" containerName="payload-processing"
Oct 30 01:27:41.394239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.394213    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51f680d6-14c5-4716-a142-f0cc9235f79e" containerName="payload-processing"
Oct 30 01:27:41.394401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.394372    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a78699a-583f-4438-a5f4-ce704e517df3" containerName="payload-processing"
Oct 30 01:27:41.394401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.394380    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e582e9-cc6a-4396-ad0d-960cbe9b89f1" containerName="revoke-keys"
Oct 30 01:27:41.394401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.394387    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="51f680d6-14c5-4716-a142-f0cc9235f79e" containerName="payload-processing"
Oct 30 01:27:41.399798 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.399773    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:27:41.424063 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.424024    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"openshift-service-ca.crt\""
Oct 30 01:27:41.429423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.429377    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a47b9a\""
Oct 30 01:27:41.429627 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.429524    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"agc-cred-224c6e3b278f\""
Oct 30 01:27:41.429697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.429688    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"payload-processing-external-model-e2e--beface97-dockercfg-5f8dx\""
Oct 30 01:27:41.429751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.429740    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"routing-overlay\""
Oct 30 01:27:41.429847 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.429832    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"kube-root-ca.crt\""
Oct 30 01:27:41.432061 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.432034    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"]
Oct 30 01:27:41.490366 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.490310    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz9fm\" (UniqueName: \"kubernetes.io/projected/0169b30d-41bf-40bf-9e5d-f97e9a533494-kube-api-access-kz9fm\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8\" (UID: \"0169b30d-41bf-40bf-9e5d-f97e9a533494\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:27:41.490600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.490505    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0169b30d-41bf-40bf-9e5d-f97e9a533494-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8\" (UID: \"0169b30d-41bf-40bf-9e5d-f97e9a533494\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:27:41.490600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.490556    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0169b30d-41bf-40bf-9e5d-f97e9a533494-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8\" (UID: \"0169b30d-41bf-40bf-9e5d-f97e9a533494\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:27:41.490818 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.490800    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0169b30d-41bf-40bf-9e5d-f97e9a533494-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8\" (UID: \"0169b30d-41bf-40bf-9e5d-f97e9a533494\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:27:41.591934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.591893    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0169b30d-41bf-40bf-9e5d-f97e9a533494-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8\" (UID: \"0169b30d-41bf-40bf-9e5d-f97e9a533494\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:27:41.591934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.591940    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0169b30d-41bf-40bf-9e5d-f97e9a533494-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8\" (UID: \"0169b30d-41bf-40bf-9e5d-f97e9a533494\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:27:41.592268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.592099    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0169b30d-41bf-40bf-9e5d-f97e9a533494-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8\" (UID: \"0169b30d-41bf-40bf-9e5d-f97e9a533494\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:27:41.592268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.592145    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz9fm\" (UniqueName: \"kubernetes.io/projected/0169b30d-41bf-40bf-9e5d-f97e9a533494-kube-api-access-kz9fm\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8\" (UID: \"0169b30d-41bf-40bf-9e5d-f97e9a533494\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:27:41.592884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.592848    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0169b30d-41bf-40bf-9e5d-f97e9a533494-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8\" (UID: \"0169b30d-41bf-40bf-9e5d-f97e9a533494\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:27:41.593015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.592907    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0169b30d-41bf-40bf-9e5d-f97e9a533494-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8\" (UID: \"0169b30d-41bf-40bf-9e5d-f97e9a533494\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:27:41.594687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.594664    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0169b30d-41bf-40bf-9e5d-f97e9a533494-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8\" (UID: \"0169b30d-41bf-40bf-9e5d-f97e9a533494\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:27:41.613609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.613574    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz9fm\" (UniqueName: \"kubernetes.io/projected/0169b30d-41bf-40bf-9e5d-f97e9a533494-kube-api-access-kz9fm\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8\" (UID: \"0169b30d-41bf-40bf-9e5d-f97e9a533494\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:27:41.716411 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:41.716319    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:27:42.031653 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:42.031614    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"]
Oct 30 01:27:42.033013 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:27:42.032984    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0169b30d_41bf_40bf_9e5d_f97e9a533494.slice/crio-6ff62fda6a77cdbe727f11ca183ebdc0a7006816ab2445e981f2c62e54a822f5 WatchSource:0}: Error finding container 6ff62fda6a77cdbe727f11ca183ebdc0a7006816ab2445e981f2c62e54a822f5: Status 404 returned error can't find the container with id 6ff62fda6a77cdbe727f11ca183ebdc0a7006816ab2445e981f2c62e54a822f5
Oct 30 01:27:42.823615 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:42.823579    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8" event={"ID":"0169b30d-41bf-40bf-9e5d-f97e9a533494","Type":"ContainerStarted","Data":"c261fa0974e867beaea78a73a42269a28c6bd3491e26ea5c32f8f6cc971c5ed0"}
Oct 30 01:27:42.824062 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:42.823633    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8" event={"ID":"0169b30d-41bf-40bf-9e5d-f97e9a533494","Type":"ContainerStarted","Data":"6ff62fda6a77cdbe727f11ca183ebdc0a7006816ab2445e981f2c62e54a822f5"}
Oct 30 01:27:42.824062 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:42.823668    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:27:42.864568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:42.864499    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8" podStartSLOduration=1.864480247 podStartE2EDuration="1.864480247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:42.862077042 +0000 UTC m=+2771.611732352" watchObservedRunningTime="2026-10-30 01:27:42.864480247 +0000 UTC m=+2771.614135603"
Oct 30 01:27:45.051513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.051470    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"]
Oct 30 01:27:45.056780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.056750    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"
Oct 30 01:27:45.081516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.081386    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:27:45.081516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.081466    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:27:45.086652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.086613    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"]
Oct 30 01:27:45.153296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.153257    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0994f410-4586-44ef-a171-2e3b946e60f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8988efd-zw842\" (UID: \"0994f410-4586-44ef-a171-2e3b946e60f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"
Oct 30 01:27:45.153511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.153336    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj9cz\" (UniqueName: \"kubernetes.io/projected/0994f410-4586-44ef-a171-2e3b946e60f0-kube-api-access-vj9cz\") pod \"maas-api-revoke-sub-c8988efd-zw842\" (UID: \"0994f410-4586-44ef-a171-2e3b946e60f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"
Oct 30 01:27:45.153511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.153438    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0994f410-4586-44ef-a171-2e3b946e60f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8988efd-zw842\" (UID: \"0994f410-4586-44ef-a171-2e3b946e60f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"
Oct 30 01:27:45.255208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.255125    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0994f410-4586-44ef-a171-2e3b946e60f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8988efd-zw842\" (UID: \"0994f410-4586-44ef-a171-2e3b946e60f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"
Oct 30 01:27:45.255208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.255211    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj9cz\" (UniqueName: \"kubernetes.io/projected/0994f410-4586-44ef-a171-2e3b946e60f0-kube-api-access-vj9cz\") pod \"maas-api-revoke-sub-c8988efd-zw842\" (UID: \"0994f410-4586-44ef-a171-2e3b946e60f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"
Oct 30 01:27:45.255510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.255241    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0994f410-4586-44ef-a171-2e3b946e60f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8988efd-zw842\" (UID: \"0994f410-4586-44ef-a171-2e3b946e60f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"
Oct 30 01:27:45.256046 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.256017    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0994f410-4586-44ef-a171-2e3b946e60f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8988efd-zw842\" (UID: \"0994f410-4586-44ef-a171-2e3b946e60f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"
Oct 30 01:27:45.258198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.258171    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0994f410-4586-44ef-a171-2e3b946e60f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8988efd-zw842\" (UID: \"0994f410-4586-44ef-a171-2e3b946e60f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"
Oct 30 01:27:45.277114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.277072    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj9cz\" (UniqueName: \"kubernetes.io/projected/0994f410-4586-44ef-a171-2e3b946e60f0-kube-api-access-vj9cz\") pod \"maas-api-revoke-sub-c8988efd-zw842\" (UID: \"0994f410-4586-44ef-a171-2e3b946e60f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"
Oct 30 01:27:45.375673 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.375573    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"
Oct 30 01:27:45.697092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.697054    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"]
Oct 30 01:27:45.697890 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:27:45.697860    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0994f410_4586_44ef_a171_2e3b946e60f0.slice/crio-0d10803a809156f7f8011b02cc3bc56effdf250f0ec19808f886e9f3ca0c8050 WatchSource:0}: Error finding container 0d10803a809156f7f8011b02cc3bc56effdf250f0ec19808f886e9f3ca0c8050: Status 404 returned error can't find the container with id 0d10803a809156f7f8011b02cc3bc56effdf250f0ec19808f886e9f3ca0c8050
Oct 30 01:27:45.755565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.755532    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:27:45.829342 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.829298    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"]
Oct 30 01:27:45.855677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:45.855643    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842" event={"ID":"0994f410-4586-44ef-a171-2e3b946e60f0","Type":"ContainerStarted","Data":"0d10803a809156f7f8011b02cc3bc56effdf250f0ec19808f886e9f3ca0c8050"}
Oct 30 01:27:46.864858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:46.864822    4425 generic.go:356] "Generic (PLEG): container finished" podID="0994f410-4586-44ef-a171-2e3b946e60f0" containerID="29bf32c6a30de007662724758e19aca303181b8fedacc9c6dc766c3c982a18b2" exitCode=0
Oct 30 01:27:46.865339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:46.864888    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842" event={"ID":"0994f410-4586-44ef-a171-2e3b946e60f0","Type":"ContainerDied","Data":"29bf32c6a30de007662724758e19aca303181b8fedacc9c6dc766c3c982a18b2"}
Oct 30 01:27:48.034287 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:48.034248    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"
Oct 30 01:27:48.099048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:48.099005    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0994f410-4586-44ef-a171-2e3b946e60f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0994f410-4586-44ef-a171-2e3b946e60f0-maas-api-xks-ca\") pod \"0994f410-4586-44ef-a171-2e3b946e60f0\" (UID: \"0994f410-4586-44ef-a171-2e3b946e60f0\") "
Oct 30 01:27:48.099315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:48.099291    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0994f410-4586-44ef-a171-2e3b946e60f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0994f410-4586-44ef-a171-2e3b946e60f0-maas-api-openshift-ca\") pod \"0994f410-4586-44ef-a171-2e3b946e60f0\" (UID: \"0994f410-4586-44ef-a171-2e3b946e60f0\") "
Oct 30 01:27:48.099379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:48.099334    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0994f410-4586-44ef-a171-2e3b946e60f0-kube-api-access-vj9cz\" (UniqueName: \"kubernetes.io/projected/0994f410-4586-44ef-a171-2e3b946e60f0-kube-api-access-vj9cz\") pod \"0994f410-4586-44ef-a171-2e3b946e60f0\" (UID: \"0994f410-4586-44ef-a171-2e3b946e60f0\") "
Oct 30 01:27:48.099765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:48.099736    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0994f410-4586-44ef-a171-2e3b946e60f0-maas-api-openshift-ca" pod "0994f410-4586-44ef-a171-2e3b946e60f0" (UID: "0994f410-4586-44ef-a171-2e3b946e60f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:48.099969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:48.099946    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0994f410-4586-44ef-a171-2e3b946e60f0-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:48.101661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:48.101626    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0994f410-4586-44ef-a171-2e3b946e60f0-maas-api-xks-ca" pod "0994f410-4586-44ef-a171-2e3b946e60f0" (UID: "0994f410-4586-44ef-a171-2e3b946e60f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:27:48.101772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:48.101698    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0994f410-4586-44ef-a171-2e3b946e60f0-kube-api-access-vj9cz" pod "0994f410-4586-44ef-a171-2e3b946e60f0" (UID: "0994f410-4586-44ef-a171-2e3b946e60f0"). InnerVolumeSpecName "kube-api-access-vj9cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:48.201626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:48.201529    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0994f410-4586-44ef-a171-2e3b946e60f0-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:48.201626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:48.201565    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj9cz\" (UniqueName: \"kubernetes.io/projected/0994f410-4586-44ef-a171-2e3b946e60f0-kube-api-access-vj9cz\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:48.882962 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:48.882925    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"
Oct 30 01:27:48.882962 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:48.882951    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842" event={"ID":"0994f410-4586-44ef-a171-2e3b946e60f0","Type":"ContainerDied","Data":"0d10803a809156f7f8011b02cc3bc56effdf250f0ec19808f886e9f3ca0c8050"}
Oct 30 01:27:48.883304 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:48.883004    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d10803a809156f7f8011b02cc3bc56effdf250f0ec19808f886e9f3ca0c8050"
Oct 30 01:27:50.830046 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:50.829994    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx" podUID="a9b1139d-7b79-4d77-b64b-677abcf63f0d" containerName="payload-processing" containerID="cri-o://66086e5f7616a9a817842fbb111f82374b02b02a07a1c52ca3d2f18818e37c92" gracePeriod=25
Oct 30 01:27:51.128129 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.128098    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:51.243237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.243197    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b1139d-7b79-4d77-b64b-677abcf63f0d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9b1139d-7b79-4d77-b64b-677abcf63f0d-provider-credentials\") pod \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\" (UID: \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\") "
Oct 30 01:27:51.243237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.243236    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b1139d-7b79-4d77-b64b-677abcf63f0d-kube-api-access-tfdv4\" (UniqueName: \"kubernetes.io/projected/a9b1139d-7b79-4d77-b64b-677abcf63f0d-kube-api-access-tfdv4\") pod \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\" (UID: \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\") "
Oct 30 01:27:51.243498 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.243399    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b1139d-7b79-4d77-b64b-677abcf63f0d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9b1139d-7b79-4d77-b64b-677abcf63f0d-routing-overlay-volume\") pod \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\" (UID: \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\") "
Oct 30 01:27:51.243498 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.243488    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b1139d-7b79-4d77-b64b-677abcf63f0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b1139d-7b79-4d77-b64b-677abcf63f0d-plugins-config-volume\") pod \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\" (UID: \"a9b1139d-7b79-4d77-b64b-677abcf63f0d\") "
Oct 30 01:27:51.243875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.243839    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b1139d-7b79-4d77-b64b-677abcf63f0d-routing-overlay-volume" pod "a9b1139d-7b79-4d77-b64b-677abcf63f0d" (UID: "a9b1139d-7b79-4d77-b64b-677abcf63f0d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:51.244002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.243904    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b1139d-7b79-4d77-b64b-677abcf63f0d-plugins-config-volume" pod "a9b1139d-7b79-4d77-b64b-677abcf63f0d" (UID: "a9b1139d-7b79-4d77-b64b-677abcf63f0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:51.244167 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.244127    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9b1139d-7b79-4d77-b64b-677abcf63f0d-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:51.244242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.244172    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9b1139d-7b79-4d77-b64b-677abcf63f0d-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:51.245716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.245690    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b1139d-7b79-4d77-b64b-677abcf63f0d-kube-api-access-tfdv4" pod "a9b1139d-7b79-4d77-b64b-677abcf63f0d" (UID: "a9b1139d-7b79-4d77-b64b-677abcf63f0d"). InnerVolumeSpecName "kube-api-access-tfdv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:51.245821 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.245732    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b1139d-7b79-4d77-b64b-677abcf63f0d-provider-credentials" pod "a9b1139d-7b79-4d77-b64b-677abcf63f0d" (UID: "a9b1139d-7b79-4d77-b64b-677abcf63f0d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:51.345494 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.345383    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9b1139d-7b79-4d77-b64b-677abcf63f0d-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:51.345494 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.345421    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfdv4\" (UniqueName: \"kubernetes.io/projected/a9b1139d-7b79-4d77-b64b-677abcf63f0d-kube-api-access-tfdv4\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:51.909879 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.909839    4425 generic.go:356] "Generic (PLEG): container finished" podID="a9b1139d-7b79-4d77-b64b-677abcf63f0d" containerID="66086e5f7616a9a817842fbb111f82374b02b02a07a1c52ca3d2f18818e37c92" exitCode=0
Oct 30 01:27:51.910389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.909894    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"
Oct 30 01:27:51.910389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.909896    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx" event={"ID":"a9b1139d-7b79-4d77-b64b-677abcf63f0d","Type":"ContainerDied","Data":"66086e5f7616a9a817842fbb111f82374b02b02a07a1c52ca3d2f18818e37c92"}
Oct 30 01:27:51.910389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.910022    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx" event={"ID":"a9b1139d-7b79-4d77-b64b-677abcf63f0d","Type":"ContainerDied","Data":"e7b9fc4cbe06cd39457159e7cffdf437c8bf754a4fe1ebb51ce4e0871ce5a5df"}
Oct 30 01:27:51.910389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.910047    4425 scope.go:122] "RemoveContainer" containerID="66086e5f7616a9a817842fbb111f82374b02b02a07a1c52ca3d2f18818e37c92"
Oct 30 01:27:51.921446 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.921419    4425 scope.go:122] "RemoveContainer" containerID="66086e5f7616a9a817842fbb111f82374b02b02a07a1c52ca3d2f18818e37c92"
Oct 30 01:27:51.921805 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:51.921780    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66086e5f7616a9a817842fbb111f82374b02b02a07a1c52ca3d2f18818e37c92\": container with ID starting with 66086e5f7616a9a817842fbb111f82374b02b02a07a1c52ca3d2f18818e37c92 not found: ID does not exist" containerID="66086e5f7616a9a817842fbb111f82374b02b02a07a1c52ca3d2f18818e37c92"
Oct 30 01:27:51.921844 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.921815    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66086e5f7616a9a817842fbb111f82374b02b02a07a1c52ca3d2f18818e37c92"} err="failed to get container status \"66086e5f7616a9a817842fbb111f82374b02b02a07a1c52ca3d2f18818e37c92\": rpc error: code = NotFound desc = could not find container \"66086e5f7616a9a817842fbb111f82374b02b02a07a1c52ca3d2f18818e37c92\": container with ID starting with 66086e5f7616a9a817842fbb111f82374b02b02a07a1c52ca3d2f18818e37c92 not found: ID does not exist"
Oct 30 01:27:51.957138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.957095    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"]
Oct 30 01:27:51.966267 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:51.966230    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-78b7b6ntx"]
Oct 30 01:27:52.329386 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:52.329345    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"]
Oct 30 01:27:53.153552 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.153511    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"]
Oct 30 01:27:53.154637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.154609    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a9b1139d-7b79-4d77-b64b-677abcf63f0d" containerName="payload-processing"
Oct 30 01:27:53.154637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.154635    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b1139d-7b79-4d77-b64b-677abcf63f0d" containerName="payload-processing"
Oct 30 01:27:53.154855 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.154668    4425 cpu_manager.go:414] "containerMap: removing container" podUID="0994f410-4586-44ef-a171-2e3b946e60f0" containerName="revoke-keys"
Oct 30 01:27:53.154855 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.154675    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0994f410-4586-44ef-a171-2e3b946e60f0" containerName="revoke-keys"
Oct 30 01:27:53.154961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.154878    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b1139d-7b79-4d77-b64b-677abcf63f0d" containerName="payload-processing"
Oct 30 01:27:53.154961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.154899    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0994f410-4586-44ef-a171-2e3b946e60f0" containerName="revoke-keys"
Oct 30 01:27:53.160241 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.160215    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:27:53.185893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.185858    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-c48628-serving-cert\""
Oct 30 01:27:53.190965 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.190925    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"]
Oct 30 01:27:53.274043 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.273998    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/692eafe6-0180-4aff-b472-c9d94ce846ea-metrics-certs\") pod \"maas-api-e2e-extmod-b-c48628-648b654d8-k6b88\" (UID: \"692eafe6-0180-4aff-b472-c9d94ce846ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:27:53.274322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.274145    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7c75\" (UniqueName: \"kubernetes.io/projected/692eafe6-0180-4aff-b472-c9d94ce846ea-kube-api-access-b7c75\") pod \"maas-api-e2e-extmod-b-c48628-648b654d8-k6b88\" (UID: \"692eafe6-0180-4aff-b472-c9d94ce846ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:27:53.274322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.274299    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/692eafe6-0180-4aff-b472-c9d94ce846ea-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c48628-648b654d8-k6b88\" (UID: \"692eafe6-0180-4aff-b472-c9d94ce846ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:27:53.274471 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.274403    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/692eafe6-0180-4aff-b472-c9d94ce846ea-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c48628-648b654d8-k6b88\" (UID: \"692eafe6-0180-4aff-b472-c9d94ce846ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:27:53.375617 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.375563    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/692eafe6-0180-4aff-b472-c9d94ce846ea-metrics-certs\") pod \"maas-api-e2e-extmod-b-c48628-648b654d8-k6b88\" (UID: \"692eafe6-0180-4aff-b472-c9d94ce846ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:27:53.375848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.375706    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7c75\" (UniqueName: \"kubernetes.io/projected/692eafe6-0180-4aff-b472-c9d94ce846ea-kube-api-access-b7c75\") pod \"maas-api-e2e-extmod-b-c48628-648b654d8-k6b88\" (UID: \"692eafe6-0180-4aff-b472-c9d94ce846ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:27:53.375848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.375773    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/692eafe6-0180-4aff-b472-c9d94ce846ea-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c48628-648b654d8-k6b88\" (UID: \"692eafe6-0180-4aff-b472-c9d94ce846ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:27:53.376685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.376651    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/692eafe6-0180-4aff-b472-c9d94ce846ea-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c48628-648b654d8-k6b88\" (UID: \"692eafe6-0180-4aff-b472-c9d94ce846ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:27:53.376847 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.376758    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/692eafe6-0180-4aff-b472-c9d94ce846ea-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c48628-648b654d8-k6b88\" (UID: \"692eafe6-0180-4aff-b472-c9d94ce846ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:27:53.376944 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:53.376920    4425 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-serving-cert: secret "maas-api-e2e-extmod-b-c48628-serving-cert" not found
Oct 30 01:27:53.377258 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:53.377233    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/692eafe6-0180-4aff-b472-c9d94ce846ea-maas-api-tls podName:692eafe6-0180-4aff-b472-c9d94ce846ea nodeName:}" failed. No retries permitted until 2026-10-30 01:27:53.877147411 +0000 UTC m=+2782.626802717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/692eafe6-0180-4aff-b472-c9d94ce846ea-maas-api-tls") pod "maas-api-e2e-extmod-b-c48628-648b654d8-k6b88" (UID: "692eafe6-0180-4aff-b472-c9d94ce846ea") : secret "maas-api-e2e-extmod-b-c48628-serving-cert" not found
Oct 30 01:27:53.378772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.378745    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/692eafe6-0180-4aff-b472-c9d94ce846ea-metrics-certs\") pod \"maas-api-e2e-extmod-b-c48628-648b654d8-k6b88\" (UID: \"692eafe6-0180-4aff-b472-c9d94ce846ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:27:53.400780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.400747    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7c75\" (UniqueName: \"kubernetes.io/projected/692eafe6-0180-4aff-b472-c9d94ce846ea-kube-api-access-b7c75\") pod \"maas-api-e2e-extmod-b-c48628-648b654d8-k6b88\" (UID: \"692eafe6-0180-4aff-b472-c9d94ce846ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:27:53.432241 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.432109    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b1139d-7b79-4d77-b64b-677abcf63f0d" path="/var/lib/kubelet/pods/a9b1139d-7b79-4d77-b64b-677abcf63f0d/volumes"
Oct 30 01:27:53.838344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.838308    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:27:53.883859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.883819    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/692eafe6-0180-4aff-b472-c9d94ce846ea-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c48628-648b654d8-k6b88\" (UID: \"692eafe6-0180-4aff-b472-c9d94ce846ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:27:53.886596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:53.886565    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/692eafe6-0180-4aff-b472-c9d94ce846ea-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c48628-648b654d8-k6b88\" (UID: \"692eafe6-0180-4aff-b472-c9d94ce846ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:27:54.076001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:54.075961    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:27:54.443570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:54.443223    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"]
Oct 30 01:27:54.444195 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:27:54.444125    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692eafe6_0180_4aff_b472_c9d94ce846ea.slice/crio-49244357d0f629330a1867e2236686c9abc05fa93606e90891222b34a7a7b596 WatchSource:0}: Error finding container 49244357d0f629330a1867e2236686c9abc05fa93606e90891222b34a7a7b596: Status 404 returned error can't find the container with id 49244357d0f629330a1867e2236686c9abc05fa93606e90891222b34a7a7b596
Oct 30 01:27:54.948077 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:54.948035    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88" event={"ID":"692eafe6-0180-4aff-b472-c9d94ce846ea","Type":"ContainerStarted","Data":"49244357d0f629330a1867e2236686c9abc05fa93606e90891222b34a7a7b596"}
Oct 30 01:27:55.614194 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.614113    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"]
Oct 30 01:27:55.619325 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.619298    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:27:55.638663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.638627    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"]
Oct 30 01:27:55.714624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.714579    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f9afbfc-e35c-412b-932a-2c971b0f98a2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq\" (UID: \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:27:55.714624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.714627    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9afbfc-e35c-412b-932a-2c971b0f98a2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq\" (UID: \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:27:55.714924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.714803    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx87v\" (UniqueName: \"kubernetes.io/projected/5f9afbfc-e35c-412b-932a-2c971b0f98a2-kube-api-access-wx87v\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq\" (UID: \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:27:55.715012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.714982    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f9afbfc-e35c-412b-932a-2c971b0f98a2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq\" (UID: \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:27:55.720869 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.720837    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-499482\"/\"agc-cred-703041db0cf0\""
Oct 30 01:27:55.721017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.720899    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-499482\"/\"payload-processing-external-model-e2e--b4b1dd99-dockercfg-8rkcz\""
Oct 30 01:27:55.816673 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.816634    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f9afbfc-e35c-412b-932a-2c971b0f98a2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq\" (UID: \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:27:55.816673 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.816681    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9afbfc-e35c-412b-932a-2c971b0f98a2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq\" (UID: \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:27:55.816926 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.816756    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx87v\" (UniqueName: \"kubernetes.io/projected/5f9afbfc-e35c-412b-932a-2c971b0f98a2-kube-api-access-wx87v\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq\" (UID: \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:27:55.816926 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.816866    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f9afbfc-e35c-412b-932a-2c971b0f98a2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq\" (UID: \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:27:55.817542 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.817521    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f9afbfc-e35c-412b-932a-2c971b0f98a2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq\" (UID: \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:27:55.817593 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.817534    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9afbfc-e35c-412b-932a-2c971b0f98a2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq\" (UID: \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:27:55.819543 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.819519    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f9afbfc-e35c-412b-932a-2c971b0f98a2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq\" (UID: \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:27:55.838103 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.838062    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx87v\" (UniqueName: \"kubernetes.io/projected/5f9afbfc-e35c-412b-932a-2c971b0f98a2-kube-api-access-wx87v\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq\" (UID: \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:27:55.935271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.935139    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:27:55.958168 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.958121    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88" event={"ID":"692eafe6-0180-4aff-b472-c9d94ce846ea","Type":"ContainerStarted","Data":"1a4a9fc0742bec53726fbb7169564e181d028e30ee1eda1a9a9706d0dc00921d"}
Oct 30 01:27:55.958372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.958214    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:27:55.997728 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:55.997392    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88" podStartSLOduration=2.528220819 podStartE2EDuration="2.997369191s" totalImagesPullingTime="469.148372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:55.995605798 +0000 UTC m=+2784.745261106" watchObservedRunningTime="2026-10-30 01:27:55.997369191 +0000 UTC m=+2784.747024501"
Oct 30 01:27:56.250407 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:56.250376    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"]
Oct 30 01:27:56.252205 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:27:56.252169    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9afbfc_e35c_412b_932a_2c971b0f98a2.slice/crio-02c331c0b2a31228f92bb6605e5cdb0b84001926a26f0a1c6391efc8e4c3a0f5 WatchSource:0}: Error finding container 02c331c0b2a31228f92bb6605e5cdb0b84001926a26f0a1c6391efc8e4c3a0f5: Status 404 returned error can't find the container with id 02c331c0b2a31228f92bb6605e5cdb0b84001926a26f0a1c6391efc8e4c3a0f5
Oct 30 01:27:56.979002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:56.978968    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq" event={"ID":"5f9afbfc-e35c-412b-932a-2c971b0f98a2","Type":"ContainerStarted","Data":"423fd41f1de09305e7149916b2388c96621660413c976994b35ff188fb144a12"}
Oct 30 01:27:56.979463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:56.979020    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq" event={"ID":"5f9afbfc-e35c-412b-932a-2c971b0f98a2","Type":"ContainerStarted","Data":"02c331c0b2a31228f92bb6605e5cdb0b84001926a26f0a1c6391efc8e4c3a0f5"}
Oct 30 01:27:56.979463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:56.979057    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:27:57.017795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.017706    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq" podStartSLOduration=2.017684518 podStartE2EDuration="2.017684518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:57.014998203 +0000 UTC m=+2785.764653525" watchObservedRunningTime="2026-10-30 01:27:57.017684518 +0000 UTC m=+2785.767339828"
Oct 30 01:27:57.330643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.330597    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk" podUID="adf5721c-1a9f-4360-b164-ad125d052f17" containerName="payload-processing" containerID="cri-o://499b145c7eb88d4a1668284f948a75755110b4e4889d2a6934a621f1f24a2e7f" gracePeriod=25
Oct 30 01:27:57.613025 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.612994    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:27:57.656810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.654746    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf5721c-1a9f-4360-b164-ad125d052f17-provider-credentials\" (UniqueName: \"kubernetes.io/projected/adf5721c-1a9f-4360-b164-ad125d052f17-provider-credentials\") pod \"adf5721c-1a9f-4360-b164-ad125d052f17\" (UID: \"adf5721c-1a9f-4360-b164-ad125d052f17\") "
Oct 30 01:27:57.656810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.654827    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adf5721c-1a9f-4360-b164-ad125d052f17-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adf5721c-1a9f-4360-b164-ad125d052f17-routing-overlay-volume\") pod \"adf5721c-1a9f-4360-b164-ad125d052f17\" (UID: \"adf5721c-1a9f-4360-b164-ad125d052f17\") "
Oct 30 01:27:57.656810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.654944    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf5721c-1a9f-4360-b164-ad125d052f17-kube-api-access-dcj6s\" (UniqueName: \"kubernetes.io/projected/adf5721c-1a9f-4360-b164-ad125d052f17-kube-api-access-dcj6s\") pod \"adf5721c-1a9f-4360-b164-ad125d052f17\" (UID: \"adf5721c-1a9f-4360-b164-ad125d052f17\") "
Oct 30 01:27:57.656810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.655080    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adf5721c-1a9f-4360-b164-ad125d052f17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf5721c-1a9f-4360-b164-ad125d052f17-plugins-config-volume\") pod \"adf5721c-1a9f-4360-b164-ad125d052f17\" (UID: \"adf5721c-1a9f-4360-b164-ad125d052f17\") "
Oct 30 01:27:57.657192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.657040    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf5721c-1a9f-4360-b164-ad125d052f17-routing-overlay-volume" pod "adf5721c-1a9f-4360-b164-ad125d052f17" (UID: "adf5721c-1a9f-4360-b164-ad125d052f17"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:57.659294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.657515    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf5721c-1a9f-4360-b164-ad125d052f17-plugins-config-volume" pod "adf5721c-1a9f-4360-b164-ad125d052f17" (UID: "adf5721c-1a9f-4360-b164-ad125d052f17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:57.659294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.657844    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf5721c-1a9f-4360-b164-ad125d052f17-provider-credentials" pod "adf5721c-1a9f-4360-b164-ad125d052f17" (UID: "adf5721c-1a9f-4360-b164-ad125d052f17"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:57.677697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.677649    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf5721c-1a9f-4360-b164-ad125d052f17-kube-api-access-dcj6s" pod "adf5721c-1a9f-4360-b164-ad125d052f17" (UID: "adf5721c-1a9f-4360-b164-ad125d052f17"). InnerVolumeSpecName "kube-api-access-dcj6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:57.757800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.757755    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcj6s\" (UniqueName: \"kubernetes.io/projected/adf5721c-1a9f-4360-b164-ad125d052f17-kube-api-access-dcj6s\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:57.757800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.757794    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adf5721c-1a9f-4360-b164-ad125d052f17-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:57.757800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.757804    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/adf5721c-1a9f-4360-b164-ad125d052f17-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:57.757800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.757812    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adf5721c-1a9f-4360-b164-ad125d052f17-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:27:57.984984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.984894    4425 generic.go:356] "Generic (PLEG): container finished" podID="adf5721c-1a9f-4360-b164-ad125d052f17" containerID="499b145c7eb88d4a1668284f948a75755110b4e4889d2a6934a621f1f24a2e7f" exitCode=0
Oct 30 01:27:57.984984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.984967    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"
Oct 30 01:27:57.984984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.984967    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk" event={"ID":"adf5721c-1a9f-4360-b164-ad125d052f17","Type":"ContainerDied","Data":"499b145c7eb88d4a1668284f948a75755110b4e4889d2a6934a621f1f24a2e7f"}
Oct 30 01:27:57.985584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.985055    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk" event={"ID":"adf5721c-1a9f-4360-b164-ad125d052f17","Type":"ContainerDied","Data":"e7a0b822ba5dcc81b04f6057bac02922b1805d0e51597b0a72abef28ca7ef80d"}
Oct 30 01:27:57.985584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.985080    4425 scope.go:122] "RemoveContainer" containerID="499b145c7eb88d4a1668284f948a75755110b4e4889d2a6934a621f1f24a2e7f"
Oct 30 01:27:57.996288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.996263    4425 scope.go:122] "RemoveContainer" containerID="499b145c7eb88d4a1668284f948a75755110b4e4889d2a6934a621f1f24a2e7f"
Oct 30 01:27:57.996662 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:27:57.996641    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"499b145c7eb88d4a1668284f948a75755110b4e4889d2a6934a621f1f24a2e7f\": container with ID starting with 499b145c7eb88d4a1668284f948a75755110b4e4889d2a6934a621f1f24a2e7f not found: ID does not exist" containerID="499b145c7eb88d4a1668284f948a75755110b4e4889d2a6934a621f1f24a2e7f"
Oct 30 01:27:57.996733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:57.996672    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"499b145c7eb88d4a1668284f948a75755110b4e4889d2a6934a621f1f24a2e7f"} err="failed to get container status \"499b145c7eb88d4a1668284f948a75755110b4e4889d2a6934a621f1f24a2e7f\": rpc error: code = NotFound desc = could not find container \"499b145c7eb88d4a1668284f948a75755110b4e4889d2a6934a621f1f24a2e7f\": container with ID starting with 499b145c7eb88d4a1668284f948a75755110b4e4889d2a6934a621f1f24a2e7f not found: ID does not exist"
Oct 30 01:27:58.032362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:58.032326    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"]
Oct 30 01:27:58.041920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:58.041835    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-7cf9rpghk"]
Oct 30 01:27:59.425488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:27:59.425452    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adf5721c-1a9f-4360-b164-ad125d052f17" path="/var/lib/kubelet/pods/adf5721c-1a9f-4360-b164-ad125d052f17/volumes"
Oct 30 01:28:01.981606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:01.981575    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:28:06.766609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.766568    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"]
Oct 30 01:28:06.767535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.767511    4425 cpu_manager.go:414] "containerMap: removing container" podUID="adf5721c-1a9f-4360-b164-ad125d052f17" containerName="payload-processing"
Oct 30 01:28:06.767535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.767536    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adf5721c-1a9f-4360-b164-ad125d052f17" containerName="payload-processing"
Oct 30 01:28:06.767698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.767686    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="adf5721c-1a9f-4360-b164-ad125d052f17" containerName="payload-processing"
Oct 30 01:28:06.771984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.771964    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:28:06.791678 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.791644    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"]
Oct 30 01:28:06.796666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.796634    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-499482\"/\"agc-cred-0d94db6fc286\""
Oct 30 01:28:06.882149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.882100    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t\" (UID: \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:28:06.882396 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.882208    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t\" (UID: \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:28:06.882470 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.882447    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t\" (UID: \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:28:06.882514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.882490    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k45hl\" (UniqueName: \"kubernetes.io/projected/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-kube-api-access-k45hl\") pod \"payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t\" (UID: \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:28:06.984242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.984196    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t\" (UID: \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:28:06.984242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.984247    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t\" (UID: \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:28:06.984489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.984434    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t\" (UID: \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:28:06.984489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.984464    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k45hl\" (UniqueName: \"kubernetes.io/projected/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-kube-api-access-k45hl\") pod \"payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t\" (UID: \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:28:06.984932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.984890    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t\" (UID: \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:28:06.985170 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.985139    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t\" (UID: \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:28:06.987108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:06.987084    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t\" (UID: \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:28:07.005902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.005861    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k45hl\" (UniqueName: \"kubernetes.io/projected/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-kube-api-access-k45hl\") pod \"payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t\" (UID: \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:28:07.075802 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.075714    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"]
Oct 30 01:28:07.081912 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.081882    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"
Oct 30 01:28:07.088428 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.088395    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:28:07.095910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.095879    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:28:07.096631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.096602    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:28:07.101926 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.101671    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"]
Oct 30 01:28:07.188648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.188044    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq8h7\" (UniqueName: \"kubernetes.io/projected/788a53b3-4edf-4be5-89d4-9ae57c3b817f-kube-api-access-vq8h7\") pod \"maas-api-revoke-sub-81e9e8ce-g5gh9\" (UID: \"788a53b3-4edf-4be5-89d4-9ae57c3b817f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"
Oct 30 01:28:07.188648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.188180    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788a53b3-4edf-4be5-89d4-9ae57c3b817f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81e9e8ce-g5gh9\" (UID: \"788a53b3-4edf-4be5-89d4-9ae57c3b817f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"
Oct 30 01:28:07.188648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.188240    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788a53b3-4edf-4be5-89d4-9ae57c3b817f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81e9e8ce-g5gh9\" (UID: \"788a53b3-4edf-4be5-89d4-9ae57c3b817f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"
Oct 30 01:28:07.290009 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.289969    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq8h7\" (UniqueName: \"kubernetes.io/projected/788a53b3-4edf-4be5-89d4-9ae57c3b817f-kube-api-access-vq8h7\") pod \"maas-api-revoke-sub-81e9e8ce-g5gh9\" (UID: \"788a53b3-4edf-4be5-89d4-9ae57c3b817f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"
Oct 30 01:28:07.290262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.290063    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788a53b3-4edf-4be5-89d4-9ae57c3b817f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81e9e8ce-g5gh9\" (UID: \"788a53b3-4edf-4be5-89d4-9ae57c3b817f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"
Oct 30 01:28:07.290262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.290112    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788a53b3-4edf-4be5-89d4-9ae57c3b817f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81e9e8ce-g5gh9\" (UID: \"788a53b3-4edf-4be5-89d4-9ae57c3b817f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"
Oct 30 01:28:07.290855 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.290832    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788a53b3-4edf-4be5-89d4-9ae57c3b817f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81e9e8ce-g5gh9\" (UID: \"788a53b3-4edf-4be5-89d4-9ae57c3b817f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"
Oct 30 01:28:07.292636 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.292576    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788a53b3-4edf-4be5-89d4-9ae57c3b817f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81e9e8ce-g5gh9\" (UID: \"788a53b3-4edf-4be5-89d4-9ae57c3b817f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"
Oct 30 01:28:07.318897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.318860    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq8h7\" (UniqueName: \"kubernetes.io/projected/788a53b3-4edf-4be5-89d4-9ae57c3b817f-kube-api-access-vq8h7\") pod \"maas-api-revoke-sub-81e9e8ce-g5gh9\" (UID: \"788a53b3-4edf-4be5-89d4-9ae57c3b817f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"
Oct 30 01:28:07.405921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.405832    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"
Oct 30 01:28:07.414489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.414463    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"]
Oct 30 01:28:07.566497 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:28:07.566455    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:28:07.723297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:07.723268    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"]
Oct 30 01:28:07.724578 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:28:07.724553    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788a53b3_4edf_4be5_89d4_9ae57c3b817f.slice/crio-f1a601bb83914610524283df21db09ef0560a094119cc85c253e057e852d4d5b WatchSource:0}: Error finding container f1a601bb83914610524283df21db09ef0560a094119cc85c253e057e852d4d5b: Status 404 returned error can't find the container with id f1a601bb83914610524283df21db09ef0560a094119cc85c253e057e852d4d5b
Oct 30 01:28:08.067616 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:08.067577    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9" event={"ID":"788a53b3-4edf-4be5-89d4-9ae57c3b817f","Type":"ContainerStarted","Data":"f1a601bb83914610524283df21db09ef0560a094119cc85c253e057e852d4d5b"}
Oct 30 01:28:08.069401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:08.069363    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t" event={"ID":"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26","Type":"ContainerStarted","Data":"628beffd62d1cdbcc3e63594d886e53ca7366dd39ed6909f9f7ff8f262ebbace"}
Oct 30 01:28:08.069575 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:08.069427    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t" event={"ID":"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26","Type":"ContainerStarted","Data":"9e07350728cf34249c49ef525d10ba30f3c39a1056944326773cbc3dce07adb4"}
Oct 30 01:28:08.069575 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:08.069458    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:28:08.108614 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:08.108552    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t" podStartSLOduration=2.108533514 podStartE2EDuration="2.108533514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:08.105372722 +0000 UTC m=+2796.855028021" watchObservedRunningTime="2026-10-30 01:28:08.108533514 +0000 UTC m=+2796.858188880"
Oct 30 01:28:08.998361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:08.998278    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:28:09.081374 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:09.081333    4425 generic.go:356] "Generic (PLEG): container finished" podID="788a53b3-4edf-4be5-89d4-9ae57c3b817f" containerID="c17a5276789e039011da8532fc6d8cc065b6203a47c85c7dfa8803976accdfbd" exitCode=0
Oct 30 01:28:09.081861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:09.081380    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9" event={"ID":"788a53b3-4edf-4be5-89d4-9ae57c3b817f","Type":"ContainerDied","Data":"c17a5276789e039011da8532fc6d8cc065b6203a47c85c7dfa8803976accdfbd"}
Oct 30 01:28:09.216546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:09.216506    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"]
Oct 30 01:28:10.260313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:10.260280    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"
Oct 30 01:28:10.339564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:10.339525    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788a53b3-4edf-4be5-89d4-9ae57c3b817f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788a53b3-4edf-4be5-89d4-9ae57c3b817f-maas-api-openshift-ca\") pod \"788a53b3-4edf-4be5-89d4-9ae57c3b817f\" (UID: \"788a53b3-4edf-4be5-89d4-9ae57c3b817f\") "
Oct 30 01:28:10.339789 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:10.339669    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788a53b3-4edf-4be5-89d4-9ae57c3b817f-kube-api-access-vq8h7\" (UniqueName: \"kubernetes.io/projected/788a53b3-4edf-4be5-89d4-9ae57c3b817f-kube-api-access-vq8h7\") pod \"788a53b3-4edf-4be5-89d4-9ae57c3b817f\" (UID: \"788a53b3-4edf-4be5-89d4-9ae57c3b817f\") "
Oct 30 01:28:10.339789 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:10.339711    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/788a53b3-4edf-4be5-89d4-9ae57c3b817f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788a53b3-4edf-4be5-89d4-9ae57c3b817f-maas-api-xks-ca\") pod \"788a53b3-4edf-4be5-89d4-9ae57c3b817f\" (UID: \"788a53b3-4edf-4be5-89d4-9ae57c3b817f\") "
Oct 30 01:28:10.340021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:10.339932    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788a53b3-4edf-4be5-89d4-9ae57c3b817f-maas-api-openshift-ca" pod "788a53b3-4edf-4be5-89d4-9ae57c3b817f" (UID: "788a53b3-4edf-4be5-89d4-9ae57c3b817f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:10.340251 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:10.340232    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/788a53b3-4edf-4be5-89d4-9ae57c3b817f-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:28:10.342205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:10.342174    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788a53b3-4edf-4be5-89d4-9ae57c3b817f-kube-api-access-vq8h7" pod "788a53b3-4edf-4be5-89d4-9ae57c3b817f" (UID: "788a53b3-4edf-4be5-89d4-9ae57c3b817f"). InnerVolumeSpecName "kube-api-access-vq8h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:10.342205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:10.342188    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/788a53b3-4edf-4be5-89d4-9ae57c3b817f-maas-api-xks-ca" pod "788a53b3-4edf-4be5-89d4-9ae57c3b817f" (UID: "788a53b3-4edf-4be5-89d4-9ae57c3b817f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:28:10.441661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:10.441618    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq8h7\" (UniqueName: \"kubernetes.io/projected/788a53b3-4edf-4be5-89d4-9ae57c3b817f-kube-api-access-vq8h7\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:28:10.441661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:10.441654    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/788a53b3-4edf-4be5-89d4-9ae57c3b817f-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:28:11.099039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:11.099007    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"
Oct 30 01:28:11.099272 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:11.099039    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9" event={"ID":"788a53b3-4edf-4be5-89d4-9ae57c3b817f","Type":"ContainerDied","Data":"f1a601bb83914610524283df21db09ef0560a094119cc85c253e057e852d4d5b"}
Oct 30 01:28:11.099272 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:11.099098    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1a601bb83914610524283df21db09ef0560a094119cc85c253e057e852d4d5b"
Oct 30 01:28:14.217346 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:14.217290    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr" podUID="0e0a077a-c6fa-43a3-9028-a0d87178dc6f" containerName="payload-processing" containerID="cri-o://246a2b1646f0d5b5b233ece406ffde9fea7aba1237cd23b1c73797de14f5e588" gracePeriod=25
Oct 30 01:28:14.510966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:14.510938    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:28:14.699031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:14.698991    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-plugins-config-volume\") pod \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\" (UID: \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\") "
Oct 30 01:28:14.699031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:14.699035    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-routing-overlay-volume\") pod \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\" (UID: \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\") "
Oct 30 01:28:14.699361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:14.699058    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-kube-api-access-njb4x\" (UniqueName: \"kubernetes.io/projected/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-kube-api-access-njb4x\") pod \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\" (UID: \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\") "
Oct 30 01:28:14.699361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:14.699170    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-provider-credentials\") pod \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\" (UID: \"0e0a077a-c6fa-43a3-9028-a0d87178dc6f\") "
Oct 30 01:28:14.699520 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:14.699495    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-routing-overlay-volume" pod "0e0a077a-c6fa-43a3-9028-a0d87178dc6f" (UID: "0e0a077a-c6fa-43a3-9028-a0d87178dc6f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:14.699620 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:14.699592    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-plugins-config-volume" pod "0e0a077a-c6fa-43a3-9028-a0d87178dc6f" (UID: "0e0a077a-c6fa-43a3-9028-a0d87178dc6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:14.699880 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:14.699861    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:28:14.699947 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:14.699886    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:28:14.701566 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:14.701537    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-kube-api-access-njb4x" pod "0e0a077a-c6fa-43a3-9028-a0d87178dc6f" (UID: "0e0a077a-c6fa-43a3-9028-a0d87178dc6f"). InnerVolumeSpecName "kube-api-access-njb4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:14.701685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:14.701594    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-provider-credentials" pod "0e0a077a-c6fa-43a3-9028-a0d87178dc6f" (UID: "0e0a077a-c6fa-43a3-9028-a0d87178dc6f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:14.801797 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:14.801740    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njb4x\" (UniqueName: \"kubernetes.io/projected/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-kube-api-access-njb4x\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:28:14.801797 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:14.801785    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e0a077a-c6fa-43a3-9028-a0d87178dc6f-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:28:15.132659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:15.132568    4425 generic.go:356] "Generic (PLEG): container finished" podID="0e0a077a-c6fa-43a3-9028-a0d87178dc6f" containerID="246a2b1646f0d5b5b233ece406ffde9fea7aba1237cd23b1c73797de14f5e588" exitCode=0
Oct 30 01:28:15.132659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:15.132617    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"
Oct 30 01:28:15.132897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:15.132651    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr" event={"ID":"0e0a077a-c6fa-43a3-9028-a0d87178dc6f","Type":"ContainerDied","Data":"246a2b1646f0d5b5b233ece406ffde9fea7aba1237cd23b1c73797de14f5e588"}
Oct 30 01:28:15.132897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:15.132722    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr" event={"ID":"0e0a077a-c6fa-43a3-9028-a0d87178dc6f","Type":"ContainerDied","Data":"e5d945606e735d36e72d7d72adfa951f98c93d4f54133d25bace47cd1ec76159"}
Oct 30 01:28:15.132897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:15.132749    4425 scope.go:122] "RemoveContainer" containerID="246a2b1646f0d5b5b233ece406ffde9fea7aba1237cd23b1c73797de14f5e588"
Oct 30 01:28:15.146018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:15.145992    4425 scope.go:122] "RemoveContainer" containerID="246a2b1646f0d5b5b233ece406ffde9fea7aba1237cd23b1c73797de14f5e588"
Oct 30 01:28:15.146479 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:28:15.146448    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246a2b1646f0d5b5b233ece406ffde9fea7aba1237cd23b1c73797de14f5e588\": container with ID starting with 246a2b1646f0d5b5b233ece406ffde9fea7aba1237cd23b1c73797de14f5e588 not found: ID does not exist" containerID="246a2b1646f0d5b5b233ece406ffde9fea7aba1237cd23b1c73797de14f5e588"
Oct 30 01:28:15.146560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:15.146488    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246a2b1646f0d5b5b233ece406ffde9fea7aba1237cd23b1c73797de14f5e588"} err="failed to get container status \"246a2b1646f0d5b5b233ece406ffde9fea7aba1237cd23b1c73797de14f5e588\": rpc error: code = NotFound desc = could not find container \"246a2b1646f0d5b5b233ece406ffde9fea7aba1237cd23b1c73797de14f5e588\": container with ID starting with 246a2b1646f0d5b5b233ece406ffde9fea7aba1237cd23b1c73797de14f5e588 not found: ID does not exist"
Oct 30 01:28:15.188593 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:15.188548    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"]
Oct 30 01:28:15.200036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:15.199989    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1afb3e/payload-processing-external-model-e2e-extmod-b-1afb3e-58c6rngkr"]
Oct 30 01:28:15.424740 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:15.424653    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e0a077a-c6fa-43a3-9028-a0d87178dc6f" path="/var/lib/kubelet/pods/0e0a077a-c6fa-43a3-9028-a0d87178dc6f/volumes"
Oct 30 01:28:19.085956 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:19.085926    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:28:19.159640 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:19.159590    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"]
Oct 30 01:28:21.484900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:21.484850    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:28:24.160558 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:24.160510    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq" podUID="5f9afbfc-e35c-412b-932a-2c971b0f98a2" containerName="payload-processing" containerID="cri-o://423fd41f1de09305e7149916b2388c96621660413c976994b35ff188fb144a12" gracePeriod=25
Oct 30 01:28:24.463550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:24.463521    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:28:24.540100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:24.540058    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9afbfc-e35c-412b-932a-2c971b0f98a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9afbfc-e35c-412b-932a-2c971b0f98a2-plugins-config-volume\") pod \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\" (UID: \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\") "
Oct 30 01:28:24.540384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:24.540183    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9afbfc-e35c-412b-932a-2c971b0f98a2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f9afbfc-e35c-412b-932a-2c971b0f98a2-routing-overlay-volume\") pod \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\" (UID: \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\") "
Oct 30 01:28:24.540384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:24.540253    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9afbfc-e35c-412b-932a-2c971b0f98a2-kube-api-access-wx87v\" (UniqueName: \"kubernetes.io/projected/5f9afbfc-e35c-412b-932a-2c971b0f98a2-kube-api-access-wx87v\") pod \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\" (UID: \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\") "
Oct 30 01:28:24.540532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:24.540416    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9afbfc-e35c-412b-932a-2c971b0f98a2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f9afbfc-e35c-412b-932a-2c971b0f98a2-provider-credentials\") pod \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\" (UID: \"5f9afbfc-e35c-412b-932a-2c971b0f98a2\") "
Oct 30 01:28:24.540712 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:24.540633    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9afbfc-e35c-412b-932a-2c971b0f98a2-routing-overlay-volume" pod "5f9afbfc-e35c-412b-932a-2c971b0f98a2" (UID: "5f9afbfc-e35c-412b-932a-2c971b0f98a2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:24.540712 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:24.540669    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9afbfc-e35c-412b-932a-2c971b0f98a2-plugins-config-volume" pod "5f9afbfc-e35c-412b-932a-2c971b0f98a2" (UID: "5f9afbfc-e35c-412b-932a-2c971b0f98a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:28:24.541122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:24.541083    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f9afbfc-e35c-412b-932a-2c971b0f98a2-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:28:24.541122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:24.541109    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f9afbfc-e35c-412b-932a-2c971b0f98a2-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:28:24.543239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:24.543212    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9afbfc-e35c-412b-932a-2c971b0f98a2-kube-api-access-wx87v" pod "5f9afbfc-e35c-412b-932a-2c971b0f98a2" (UID: "5f9afbfc-e35c-412b-932a-2c971b0f98a2"). InnerVolumeSpecName "kube-api-access-wx87v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:24.543332 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:24.543261    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9afbfc-e35c-412b-932a-2c971b0f98a2-provider-credentials" pod "5f9afbfc-e35c-412b-932a-2c971b0f98a2" (UID: "5f9afbfc-e35c-412b-932a-2c971b0f98a2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:28:24.642726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:24.642687    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f9afbfc-e35c-412b-932a-2c971b0f98a2-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:28:24.642726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:24.642718    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx87v\" (UniqueName: \"kubernetes.io/projected/5f9afbfc-e35c-412b-932a-2c971b0f98a2-kube-api-access-wx87v\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:28:25.219750 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:25.219712    4425 generic.go:356] "Generic (PLEG): container finished" podID="5f9afbfc-e35c-412b-932a-2c971b0f98a2" containerID="423fd41f1de09305e7149916b2388c96621660413c976994b35ff188fb144a12" exitCode=0
Oct 30 01:28:25.220332 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:25.219766    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"
Oct 30 01:28:25.220332 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:25.219797    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq" event={"ID":"5f9afbfc-e35c-412b-932a-2c971b0f98a2","Type":"ContainerDied","Data":"423fd41f1de09305e7149916b2388c96621660413c976994b35ff188fb144a12"}
Oct 30 01:28:25.220332 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:25.219867    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq" event={"ID":"5f9afbfc-e35c-412b-932a-2c971b0f98a2","Type":"ContainerDied","Data":"02c331c0b2a31228f92bb6605e5cdb0b84001926a26f0a1c6391efc8e4c3a0f5"}
Oct 30 01:28:25.220332 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:25.219898    4425 scope.go:122] "RemoveContainer" containerID="423fd41f1de09305e7149916b2388c96621660413c976994b35ff188fb144a12"
Oct 30 01:28:25.236386 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:25.236356    4425 scope.go:122] "RemoveContainer" containerID="423fd41f1de09305e7149916b2388c96621660413c976994b35ff188fb144a12"
Oct 30 01:28:25.236760 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:28:25.236735    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"423fd41f1de09305e7149916b2388c96621660413c976994b35ff188fb144a12\": container with ID starting with 423fd41f1de09305e7149916b2388c96621660413c976994b35ff188fb144a12 not found: ID does not exist" containerID="423fd41f1de09305e7149916b2388c96621660413c976994b35ff188fb144a12"
Oct 30 01:28:25.236850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:25.236769    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423fd41f1de09305e7149916b2388c96621660413c976994b35ff188fb144a12"} err="failed to get container status \"423fd41f1de09305e7149916b2388c96621660413c976994b35ff188fb144a12\": rpc error: code = NotFound desc = could not find container \"423fd41f1de09305e7149916b2388c96621660413c976994b35ff188fb144a12\": container with ID starting with 423fd41f1de09305e7149916b2388c96621660413c976994b35ff188fb144a12 not found: ID does not exist"
Oct 30 01:28:25.273113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:25.273062    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"]
Oct 30 01:28:25.279543 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:25.279496    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dwzbkq"]
Oct 30 01:28:25.425653 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:25.425611    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f9afbfc-e35c-412b-932a-2c971b0f98a2" path="/var/lib/kubelet/pods/5f9afbfc-e35c-412b-932a-2c971b0f98a2/volumes"
Oct 30 01:28:37.558371 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:28:37.558313    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:28:56.789012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.788913    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"]
Oct 30 01:28:56.789933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.789909    4425 cpu_manager.go:414] "containerMap: removing container" podUID="0e0a077a-c6fa-43a3-9028-a0d87178dc6f" containerName="payload-processing"
Oct 30 01:28:56.789982 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.789934    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0a077a-c6fa-43a3-9028-a0d87178dc6f" containerName="payload-processing"
Oct 30 01:28:56.789982 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.789971    4425 cpu_manager.go:414] "containerMap: removing container" podUID="5f9afbfc-e35c-412b-932a-2c971b0f98a2" containerName="payload-processing"
Oct 30 01:28:56.789982 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.789977    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f9afbfc-e35c-412b-932a-2c971b0f98a2" containerName="payload-processing"
Oct 30 01:28:56.790087 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.789994    4425 cpu_manager.go:414] "containerMap: removing container" podUID="788a53b3-4edf-4be5-89d4-9ae57c3b817f" containerName="revoke-keys"
Oct 30 01:28:56.790087 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.789999    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="788a53b3-4edf-4be5-89d4-9ae57c3b817f" containerName="revoke-keys"
Oct 30 01:28:56.790173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.790123    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0a077a-c6fa-43a3-9028-a0d87178dc6f" containerName="payload-processing"
Oct 30 01:28:56.790173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.790138    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f9afbfc-e35c-412b-932a-2c971b0f98a2" containerName="payload-processing"
Oct 30 01:28:56.790173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.790147    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="788a53b3-4edf-4be5-89d4-9ae57c3b817f" containerName="revoke-keys"
Oct 30 01:28:56.796218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.796191    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:28:56.814198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.813701    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"openshift-service-ca.crt\""
Oct 30 01:28:56.814198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.814135    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"]
Oct 30 01:28:56.822858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.822718    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"routing-overlay\""
Oct 30 01:28:56.824493 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.824079    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2f820e\""
Oct 30 01:28:56.824732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.824249    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"payload-processing-external-model-e2e--59030ca4-dockercfg-fhkjd\""
Oct 30 01:28:56.824862 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.824330    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"kube-root-ca.crt\""
Oct 30 01:28:56.824980 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.824418    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"agc-cred-be73cbbeb589\""
Oct 30 01:28:56.912362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.912316    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/429b7f89-db64-4879-a251-ba7cb81274d5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7\" (UID: \"429b7f89-db64-4879-a251-ba7cb81274d5\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:28:56.912362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.912370    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/429b7f89-db64-4879-a251-ba7cb81274d5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7\" (UID: \"429b7f89-db64-4879-a251-ba7cb81274d5\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:28:56.912695 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.912529    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2ggw\" (UniqueName: \"kubernetes.io/projected/429b7f89-db64-4879-a251-ba7cb81274d5-kube-api-access-s2ggw\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7\" (UID: \"429b7f89-db64-4879-a251-ba7cb81274d5\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:28:56.912695 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:56.912667    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/429b7f89-db64-4879-a251-ba7cb81274d5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7\" (UID: \"429b7f89-db64-4879-a251-ba7cb81274d5\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:28:57.013750 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.013709    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/429b7f89-db64-4879-a251-ba7cb81274d5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7\" (UID: \"429b7f89-db64-4879-a251-ba7cb81274d5\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:28:57.013750 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.013755    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/429b7f89-db64-4879-a251-ba7cb81274d5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7\" (UID: \"429b7f89-db64-4879-a251-ba7cb81274d5\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:28:57.014071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.013806    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2ggw\" (UniqueName: \"kubernetes.io/projected/429b7f89-db64-4879-a251-ba7cb81274d5-kube-api-access-s2ggw\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7\" (UID: \"429b7f89-db64-4879-a251-ba7cb81274d5\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:28:57.014071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.013894    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/429b7f89-db64-4879-a251-ba7cb81274d5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7\" (UID: \"429b7f89-db64-4879-a251-ba7cb81274d5\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:28:57.014968 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.014736    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/429b7f89-db64-4879-a251-ba7cb81274d5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7\" (UID: \"429b7f89-db64-4879-a251-ba7cb81274d5\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:28:57.014968 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.014851    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/429b7f89-db64-4879-a251-ba7cb81274d5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7\" (UID: \"429b7f89-db64-4879-a251-ba7cb81274d5\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:28:57.016728 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.016706    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/429b7f89-db64-4879-a251-ba7cb81274d5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7\" (UID: \"429b7f89-db64-4879-a251-ba7cb81274d5\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:28:57.036237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.036203    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2ggw\" (UniqueName: \"kubernetes.io/projected/429b7f89-db64-4879-a251-ba7cb81274d5-kube-api-access-s2ggw\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7\" (UID: \"429b7f89-db64-4879-a251-ba7cb81274d5\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:28:57.112632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.112528    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:28:57.427516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.427484    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"]
Oct 30 01:28:57.428621 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:28:57.428592    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429b7f89_db64_4879_a251_ba7cb81274d5.slice/crio-bca5759f96d1b15f57a7185d28f4381fae51bfc2d13ead2637a857bcf5312da5 WatchSource:0}: Error finding container bca5759f96d1b15f57a7185d28f4381fae51bfc2d13ead2637a857bcf5312da5: Status 404 returned error can't find the container with id bca5759f96d1b15f57a7185d28f4381fae51bfc2d13ead2637a857bcf5312da5
Oct 30 01:28:57.483990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.483946    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7" event={"ID":"429b7f89-db64-4879-a251-ba7cb81274d5","Type":"ContainerStarted","Data":"bca5759f96d1b15f57a7185d28f4381fae51bfc2d13ead2637a857bcf5312da5"}
Oct 30 01:28:57.527831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.527789    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"]
Oct 30 01:28:57.534266 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.533520    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:28:57.549786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.549754    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2f820e\"/\"agc-cred-df8a30b89c3f\""
Oct 30 01:28:57.580789 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.580737    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"]
Oct 30 01:28:57.622277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.622237    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f9e186-d092-44a7-8d83-2c1f9e96388a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql\" (UID: \"37f9e186-d092-44a7-8d83-2c1f9e96388a\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:28:57.622511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.622310    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k9hf\" (UniqueName: \"kubernetes.io/projected/37f9e186-d092-44a7-8d83-2c1f9e96388a-kube-api-access-5k9hf\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql\" (UID: \"37f9e186-d092-44a7-8d83-2c1f9e96388a\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:28:57.622511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.622417    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37f9e186-d092-44a7-8d83-2c1f9e96388a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql\" (UID: \"37f9e186-d092-44a7-8d83-2c1f9e96388a\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:28:57.622732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.622707    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37f9e186-d092-44a7-8d83-2c1f9e96388a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql\" (UID: \"37f9e186-d092-44a7-8d83-2c1f9e96388a\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:28:57.724707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.724592    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f9e186-d092-44a7-8d83-2c1f9e96388a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql\" (UID: \"37f9e186-d092-44a7-8d83-2c1f9e96388a\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:28:57.724707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.724665    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k9hf\" (UniqueName: \"kubernetes.io/projected/37f9e186-d092-44a7-8d83-2c1f9e96388a-kube-api-access-5k9hf\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql\" (UID: \"37f9e186-d092-44a7-8d83-2c1f9e96388a\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:28:57.724707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.724694    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37f9e186-d092-44a7-8d83-2c1f9e96388a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql\" (UID: \"37f9e186-d092-44a7-8d83-2c1f9e96388a\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:28:57.725043 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.724835    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37f9e186-d092-44a7-8d83-2c1f9e96388a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql\" (UID: \"37f9e186-d092-44a7-8d83-2c1f9e96388a\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:28:57.725504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.725478    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37f9e186-d092-44a7-8d83-2c1f9e96388a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql\" (UID: \"37f9e186-d092-44a7-8d83-2c1f9e96388a\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:28:57.725635 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.725563    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f9e186-d092-44a7-8d83-2c1f9e96388a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql\" (UID: \"37f9e186-d092-44a7-8d83-2c1f9e96388a\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:28:57.727592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.727563    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37f9e186-d092-44a7-8d83-2c1f9e96388a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql\" (UID: \"37f9e186-d092-44a7-8d83-2c1f9e96388a\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:28:57.746175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.746119    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k9hf\" (UniqueName: \"kubernetes.io/projected/37f9e186-d092-44a7-8d83-2c1f9e96388a-kube-api-access-5k9hf\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql\" (UID: \"37f9e186-d092-44a7-8d83-2c1f9e96388a\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:28:57.880217 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:57.880140    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:28:58.209982 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:58.208856    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"]
Oct 30 01:28:58.209982 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:28:58.209900    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f9e186_d092_44a7_8d83_2c1f9e96388a.slice/crio-a0af1369bc8bdc6333d6363ce1ab232ad9438603196014c9a43df9750c2c0354 WatchSource:0}: Error finding container a0af1369bc8bdc6333d6363ce1ab232ad9438603196014c9a43df9750c2c0354: Status 404 returned error can't find the container with id a0af1369bc8bdc6333d6363ce1ab232ad9438603196014c9a43df9750c2c0354
Oct 30 01:28:58.493841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:58.493729    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql" event={"ID":"37f9e186-d092-44a7-8d83-2c1f9e96388a","Type":"ContainerStarted","Data":"ccc2e5a4e3158806b9ed6916f12d325c56f538b67703874b3c2b5d3db439c995"}
Oct 30 01:28:58.493841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:58.493804    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:28:58.493841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:58.493831    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql" event={"ID":"37f9e186-d092-44a7-8d83-2c1f9e96388a","Type":"ContainerStarted","Data":"a0af1369bc8bdc6333d6363ce1ab232ad9438603196014c9a43df9750c2c0354"}
Oct 30 01:28:58.495735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:58.495703    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7" event={"ID":"429b7f89-db64-4879-a251-ba7cb81274d5","Type":"ContainerStarted","Data":"763f46f1f1dbfe6c6b73d9b27399f9a368bd76864906f670ea9848cffe696051"}
Oct 30 01:28:58.495888 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:58.495812    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:28:58.534056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:58.533991    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql" podStartSLOduration=1.5339753699999998 podStartE2EDuration="1.53397537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:58.530192755 +0000 UTC m=+2847.279848056" watchObservedRunningTime="2026-10-30 01:28:58.53397537 +0000 UTC m=+2847.283630727"
Oct 30 01:28:58.570225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:28:58.570168    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7" podStartSLOduration=2.570135223 podStartE2EDuration="2.570135223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:58.566271866 +0000 UTC m=+2847.315927172" watchObservedRunningTime="2026-10-30 01:28:58.570135223 +0000 UTC m=+2847.319790531"
Oct 30 01:29:00.128062 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:00.128026    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"]
Oct 30 01:29:00.133441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:00.133412    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"
Oct 30 01:29:00.156981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:00.156945    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"]
Oct 30 01:29:00.161747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:00.161712    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:29:00.161996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:00.161849    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:29:00.266766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:00.266725    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fz5j\" (UniqueName: \"kubernetes.io/projected/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-kube-api-access-8fz5j\") pod \"maas-api-revoke-sub-49f5225c-6p52f\" (UID: \"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"
Oct 30 01:29:00.266976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:00.266864    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49f5225c-6p52f\" (UID: \"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"
Oct 30 01:29:00.266976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:00.266936    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49f5225c-6p52f\" (UID: \"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"
Oct 30 01:29:00.368487 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:00.368442    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49f5225c-6p52f\" (UID: \"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"
Oct 30 01:29:00.368731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:00.368535    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49f5225c-6p52f\" (UID: \"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"
Oct 30 01:29:00.368731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:00.368705    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fz5j\" (UniqueName: \"kubernetes.io/projected/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-kube-api-access-8fz5j\") pod \"maas-api-revoke-sub-49f5225c-6p52f\" (UID: \"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"
Oct 30 01:29:00.369295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:00.369265    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49f5225c-6p52f\" (UID: \"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"
Oct 30 01:29:00.371031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:00.371009    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49f5225c-6p52f\" (UID: \"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"
Oct 30 01:29:00.390291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:00.390201    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fz5j\" (UniqueName: \"kubernetes.io/projected/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-kube-api-access-8fz5j\") pod \"maas-api-revoke-sub-49f5225c-6p52f\" (UID: \"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"
Oct 30 01:29:00.449694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:00.449650    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"
Oct 30 01:29:00.758889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:00.758853    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"]
Oct 30 01:29:00.759756 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:29:00.759724    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97d4fc2_afb0_4a9b_a05a_b4b6737e0caf.slice/crio-bde350e6ddefb88d4ba44f85e5a5869f1610304fbb200e11535afe3c40e48ee4 WatchSource:0}: Error finding container bde350e6ddefb88d4ba44f85e5a5869f1610304fbb200e11535afe3c40e48ee4: Status 404 returned error can't find the container with id bde350e6ddefb88d4ba44f85e5a5869f1610304fbb200e11535afe3c40e48ee4
Oct 30 01:29:01.527244 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:01.527189    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f" event={"ID":"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf","Type":"ContainerStarted","Data":"d8b61d977f8c04cb05dc1c611680434459a29d5107c488de6b5fdb8b629e91c4"}
Oct 30 01:29:01.527682 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:01.527267    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f" event={"ID":"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf","Type":"ContainerStarted","Data":"bde350e6ddefb88d4ba44f85e5a5869f1610304fbb200e11535afe3c40e48ee4"}
Oct 30 01:29:01.565342 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:01.565260    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f" podStartSLOduration=1.5652375360000002 podStartE2EDuration="1.565237536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:01.562112175 +0000 UTC m=+2850.311767466" watchObservedRunningTime="2026-10-30 01:29:01.565237536 +0000 UTC m=+2850.314892846"
Oct 30 01:29:02.536931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:02.536882    4425 generic.go:356] "Generic (PLEG): container finished" podID="f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf" containerID="d8b61d977f8c04cb05dc1c611680434459a29d5107c488de6b5fdb8b629e91c4" exitCode=0
Oct 30 01:29:02.537561 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:02.536966    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f" event={"ID":"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf","Type":"ContainerDied","Data":"d8b61d977f8c04cb05dc1c611680434459a29d5107c488de6b5fdb8b629e91c4"}
Oct 30 01:29:03.722904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:03.722872    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"
Oct 30 01:29:03.816743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:03.816637    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-maas-api-xks-ca\") pod \"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf\" (UID: \"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf\") "
Oct 30 01:29:03.816743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:03.816701    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-kube-api-access-8fz5j\" (UniqueName: \"kubernetes.io/projected/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-kube-api-access-8fz5j\") pod \"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf\" (UID: \"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf\") "
Oct 30 01:29:03.817036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:03.816859    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-maas-api-openshift-ca\") pod \"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf\" (UID: \"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf\") "
Oct 30 01:29:03.817376 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:03.817340    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-maas-api-openshift-ca" pod "f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf" (UID: "f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:03.817668 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:03.817646    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:03.819079 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:03.819054    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-maas-api-xks-ca" pod "f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf" (UID: "f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:03.819256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:03.819215    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-kube-api-access-8fz5j" pod "f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf" (UID: "f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf"). InnerVolumeSpecName "kube-api-access-8fz5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:03.918991 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:03.918943    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fz5j\" (UniqueName: \"kubernetes.io/projected/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-kube-api-access-8fz5j\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:03.918991 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:03.918983    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:04.554550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:04.554516    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"
Oct 30 01:29:04.554550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:04.554542    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f" event={"ID":"f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf","Type":"ContainerDied","Data":"bde350e6ddefb88d4ba44f85e5a5869f1610304fbb200e11535afe3c40e48ee4"}
Oct 30 01:29:04.554796 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:04.554599    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bde350e6ddefb88d4ba44f85e5a5869f1610304fbb200e11535afe3c40e48ee4"
Oct 30 01:29:07.559214 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:29:07.559170    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:29:09.507008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:09.506969    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:29:09.507566 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:09.507054    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:29:09.621570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:09.621527    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"]
Oct 30 01:29:11.405352 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:11.405310    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"]
Oct 30 01:29:14.622739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:14.622672    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7" podUID="429b7f89-db64-4879-a251-ba7cb81274d5" containerName="payload-processing" containerID="cri-o://763f46f1f1dbfe6c6b73d9b27399f9a368bd76864906f670ea9848cffe696051" gracePeriod=25
Oct 30 01:29:14.792609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:14.792563    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:29:14.911022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:14.910994    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:29:14.977131 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:14.977078    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429b7f89-db64-4879-a251-ba7cb81274d5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/429b7f89-db64-4879-a251-ba7cb81274d5-provider-credentials\") pod \"429b7f89-db64-4879-a251-ba7cb81274d5\" (UID: \"429b7f89-db64-4879-a251-ba7cb81274d5\") "
Oct 30 01:29:14.977359 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:14.977204    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/429b7f89-db64-4879-a251-ba7cb81274d5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/429b7f89-db64-4879-a251-ba7cb81274d5-routing-overlay-volume\") pod \"429b7f89-db64-4879-a251-ba7cb81274d5\" (UID: \"429b7f89-db64-4879-a251-ba7cb81274d5\") "
Oct 30 01:29:14.977359 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:14.977305    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429b7f89-db64-4879-a251-ba7cb81274d5-kube-api-access-s2ggw\" (UniqueName: \"kubernetes.io/projected/429b7f89-db64-4879-a251-ba7cb81274d5-kube-api-access-s2ggw\") pod \"429b7f89-db64-4879-a251-ba7cb81274d5\" (UID: \"429b7f89-db64-4879-a251-ba7cb81274d5\") "
Oct 30 01:29:14.977495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:14.977404    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/429b7f89-db64-4879-a251-ba7cb81274d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/429b7f89-db64-4879-a251-ba7cb81274d5-plugins-config-volume\") pod \"429b7f89-db64-4879-a251-ba7cb81274d5\" (UID: \"429b7f89-db64-4879-a251-ba7cb81274d5\") "
Oct 30 01:29:14.977670 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:14.977634    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429b7f89-db64-4879-a251-ba7cb81274d5-routing-overlay-volume" pod "429b7f89-db64-4879-a251-ba7cb81274d5" (UID: "429b7f89-db64-4879-a251-ba7cb81274d5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:14.978067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:14.978035    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429b7f89-db64-4879-a251-ba7cb81274d5-plugins-config-volume" pod "429b7f89-db64-4879-a251-ba7cb81274d5" (UID: "429b7f89-db64-4879-a251-ba7cb81274d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:14.978404 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:14.978365    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/429b7f89-db64-4879-a251-ba7cb81274d5-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:14.978404 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:14.978396    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/429b7f89-db64-4879-a251-ba7cb81274d5-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:14.980114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:14.980085    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429b7f89-db64-4879-a251-ba7cb81274d5-kube-api-access-s2ggw" pod "429b7f89-db64-4879-a251-ba7cb81274d5" (UID: "429b7f89-db64-4879-a251-ba7cb81274d5"). InnerVolumeSpecName "kube-api-access-s2ggw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:14.980659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:14.980628    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429b7f89-db64-4879-a251-ba7cb81274d5-provider-credentials" pod "429b7f89-db64-4879-a251-ba7cb81274d5" (UID: "429b7f89-db64-4879-a251-ba7cb81274d5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:15.079301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:15.079266    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2ggw\" (UniqueName: \"kubernetes.io/projected/429b7f89-db64-4879-a251-ba7cb81274d5-kube-api-access-s2ggw\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:15.079301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:15.079296    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/429b7f89-db64-4879-a251-ba7cb81274d5-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:15.653735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:15.653695    4425 generic.go:356] "Generic (PLEG): container finished" podID="429b7f89-db64-4879-a251-ba7cb81274d5" containerID="763f46f1f1dbfe6c6b73d9b27399f9a368bd76864906f670ea9848cffe696051" exitCode=0
Oct 30 01:29:15.654188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:15.653760    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"
Oct 30 01:29:15.654188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:15.653775    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7" event={"ID":"429b7f89-db64-4879-a251-ba7cb81274d5","Type":"ContainerDied","Data":"763f46f1f1dbfe6c6b73d9b27399f9a368bd76864906f670ea9848cffe696051"}
Oct 30 01:29:15.654188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:15.653835    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7" event={"ID":"429b7f89-db64-4879-a251-ba7cb81274d5","Type":"ContainerDied","Data":"bca5759f96d1b15f57a7185d28f4381fae51bfc2d13ead2637a857bcf5312da5"}
Oct 30 01:29:15.654188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:15.653864    4425 scope.go:122] "RemoveContainer" containerID="763f46f1f1dbfe6c6b73d9b27399f9a368bd76864906f670ea9848cffe696051"
Oct 30 01:29:15.668700 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:15.668673    4425 scope.go:122] "RemoveContainer" containerID="763f46f1f1dbfe6c6b73d9b27399f9a368bd76864906f670ea9848cffe696051"
Oct 30 01:29:15.669124 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:29:15.669072    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"763f46f1f1dbfe6c6b73d9b27399f9a368bd76864906f670ea9848cffe696051\": container with ID starting with 763f46f1f1dbfe6c6b73d9b27399f9a368bd76864906f670ea9848cffe696051 not found: ID does not exist" containerID="763f46f1f1dbfe6c6b73d9b27399f9a368bd76864906f670ea9848cffe696051"
Oct 30 01:29:15.669351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:15.669116    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"763f46f1f1dbfe6c6b73d9b27399f9a368bd76864906f670ea9848cffe696051"} err="failed to get container status \"763f46f1f1dbfe6c6b73d9b27399f9a368bd76864906f670ea9848cffe696051\": rpc error: code = NotFound desc = could not find container \"763f46f1f1dbfe6c6b73d9b27399f9a368bd76864906f670ea9848cffe696051\": container with ID starting with 763f46f1f1dbfe6c6b73d9b27399f9a368bd76864906f670ea9848cffe696051 not found: ID does not exist"
Oct 30 01:29:15.701292 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:15.701197    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"]
Oct 30 01:29:15.710262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:15.710230    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7klkz7"]
Oct 30 01:29:16.406514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:16.406462    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq" podUID="f311ac5c-a3f8-4167-8a8f-ce2b03b66197" containerName="payload-processing" containerID="cri-o://b0e384391f6a27a487400c09e42a796b1e6dd9d6995bbbc388d15c3acb6ee69f" gracePeriod=25
Oct 30 01:29:16.671496 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:16.671398    4425 generic.go:356] "Generic (PLEG): container finished" podID="f311ac5c-a3f8-4167-8a8f-ce2b03b66197" containerID="b0e384391f6a27a487400c09e42a796b1e6dd9d6995bbbc388d15c3acb6ee69f" exitCode=0
Oct 30 01:29:16.671496 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:16.671465    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq" event={"ID":"f311ac5c-a3f8-4167-8a8f-ce2b03b66197","Type":"ContainerDied","Data":"b0e384391f6a27a487400c09e42a796b1e6dd9d6995bbbc388d15c3acb6ee69f"}
Oct 30 01:29:16.730407 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:16.730375    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:29:16.807981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:16.807939    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-provider-credentials\") pod \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\" (UID: \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\") "
Oct 30 01:29:16.808217 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:16.808118    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-routing-overlay-volume\") pod \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\" (UID: \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\") "
Oct 30 01:29:16.808318 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:16.808267    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-kube-api-access-xlhtg\" (UniqueName: \"kubernetes.io/projected/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-kube-api-access-xlhtg\") pod \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\" (UID: \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\") "
Oct 30 01:29:16.808318 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:16.808302    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-plugins-config-volume\") pod \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\" (UID: \"f311ac5c-a3f8-4167-8a8f-ce2b03b66197\") "
Oct 30 01:29:16.808656 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:16.808545    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-routing-overlay-volume" pod "f311ac5c-a3f8-4167-8a8f-ce2b03b66197" (UID: "f311ac5c-a3f8-4167-8a8f-ce2b03b66197"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:16.808833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:16.808785    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-plugins-config-volume" pod "f311ac5c-a3f8-4167-8a8f-ce2b03b66197" (UID: "f311ac5c-a3f8-4167-8a8f-ce2b03b66197"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:16.809011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:16.808990    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:16.809097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:16.809016    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:16.811076 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:16.811048    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-kube-api-access-xlhtg" pod "f311ac5c-a3f8-4167-8a8f-ce2b03b66197" (UID: "f311ac5c-a3f8-4167-8a8f-ce2b03b66197"). InnerVolumeSpecName "kube-api-access-xlhtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:16.811243 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:16.811092    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-provider-credentials" pod "f311ac5c-a3f8-4167-8a8f-ce2b03b66197" (UID: "f311ac5c-a3f8-4167-8a8f-ce2b03b66197"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:16.910403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:16.910357    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlhtg\" (UniqueName: \"kubernetes.io/projected/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-kube-api-access-xlhtg\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:16.910403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:16.910392    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f311ac5c-a3f8-4167-8a8f-ce2b03b66197-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:17.424172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:17.424131    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="429b7f89-db64-4879-a251-ba7cb81274d5" path="/var/lib/kubelet/pods/429b7f89-db64-4879-a251-ba7cb81274d5/volumes"
Oct 30 01:29:17.685525 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:17.685411    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq" event={"ID":"f311ac5c-a3f8-4167-8a8f-ce2b03b66197","Type":"ContainerDied","Data":"033ff0a368551206db35cdf02588f4a8ada8a8c3e100c03df5580202269e9250"}
Oct 30 01:29:17.685525 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:17.685400    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"
Oct 30 01:29:17.685525 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:17.685490    4425 scope.go:122] "RemoveContainer" containerID="b0e384391f6a27a487400c09e42a796b1e6dd9d6995bbbc388d15c3acb6ee69f"
Oct 30 01:29:17.732799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:17.732765    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"]
Oct 30 01:29:17.742215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:17.742148    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-5b95wrswq"]
Oct 30 01:29:19.425348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.425313    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f311ac5c-a3f8-4167-8a8f-ce2b03b66197" path="/var/lib/kubelet/pods/f311ac5c-a3f8-4167-8a8f-ce2b03b66197/volumes"
Oct 30 01:29:19.544409 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.544369    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"]
Oct 30 01:29:19.545242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.545208    4425 cpu_manager.go:414] "containerMap: removing container" podUID="f311ac5c-a3f8-4167-8a8f-ce2b03b66197" containerName="payload-processing"
Oct 30 01:29:19.545242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.545233    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f311ac5c-a3f8-4167-8a8f-ce2b03b66197" containerName="payload-processing"
Oct 30 01:29:19.545499 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.545255    4425 cpu_manager.go:414] "containerMap: removing container" podUID="f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf" containerName="revoke-keys"
Oct 30 01:29:19.545499 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.545261    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf" containerName="revoke-keys"
Oct 30 01:29:19.545499 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.545286    4425 cpu_manager.go:414] "containerMap: removing container" podUID="429b7f89-db64-4879-a251-ba7cb81274d5" containerName="payload-processing"
Oct 30 01:29:19.545499 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.545293    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429b7f89-db64-4879-a251-ba7cb81274d5" containerName="payload-processing"
Oct 30 01:29:19.545499 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.545474    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f311ac5c-a3f8-4167-8a8f-ce2b03b66197" containerName="payload-processing"
Oct 30 01:29:19.545499 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.545493    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf" containerName="revoke-keys"
Oct 30 01:29:19.545734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.545512    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="429b7f89-db64-4879-a251-ba7cb81274d5" containerName="payload-processing"
Oct 30 01:29:19.551036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.551011    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"
Oct 30 01:29:19.589406 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.589370    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:29:19.589406 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.589375    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:29:19.597453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.597408    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"]
Oct 30 01:29:19.646431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.646389    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d61281a-grqfz\" (UID: \"04e0e257-3b3c-414f-a8ac-d4fef17ef3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"
Oct 30 01:29:19.646431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.646430    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d61281a-grqfz\" (UID: \"04e0e257-3b3c-414f-a8ac-d4fef17ef3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"
Oct 30 01:29:19.646727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.646588    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znpm6\" (UniqueName: \"kubernetes.io/projected/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-kube-api-access-znpm6\") pod \"maas-api-revoke-sub-8d61281a-grqfz\" (UID: \"04e0e257-3b3c-414f-a8ac-d4fef17ef3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"
Oct 30 01:29:19.748026 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.747927    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d61281a-grqfz\" (UID: \"04e0e257-3b3c-414f-a8ac-d4fef17ef3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"
Oct 30 01:29:19.748026 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.747972    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d61281a-grqfz\" (UID: \"04e0e257-3b3c-414f-a8ac-d4fef17ef3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"
Oct 30 01:29:19.748026 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.748024    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znpm6\" (UniqueName: \"kubernetes.io/projected/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-kube-api-access-znpm6\") pod \"maas-api-revoke-sub-8d61281a-grqfz\" (UID: \"04e0e257-3b3c-414f-a8ac-d4fef17ef3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"
Oct 30 01:29:19.748814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.748783    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d61281a-grqfz\" (UID: \"04e0e257-3b3c-414f-a8ac-d4fef17ef3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"
Oct 30 01:29:19.750632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.750599    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d61281a-grqfz\" (UID: \"04e0e257-3b3c-414f-a8ac-d4fef17ef3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"
Oct 30 01:29:19.773768 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.773731    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znpm6\" (UniqueName: \"kubernetes.io/projected/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-kube-api-access-znpm6\") pod \"maas-api-revoke-sub-8d61281a-grqfz\" (UID: \"04e0e257-3b3c-414f-a8ac-d4fef17ef3be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"
Oct 30 01:29:19.867405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:19.867363    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"
Oct 30 01:29:20.175941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:20.175906    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"]
Oct 30 01:29:20.177290 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:29:20.177247    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e0e257_3b3c_414f_a8ac_d4fef17ef3be.slice/crio-e4f7519d6cb11b31df14858a182dbedd28e63a221302accc5fd89d1a02b3c913 WatchSource:0}: Error finding container e4f7519d6cb11b31df14858a182dbedd28e63a221302accc5fd89d1a02b3c913: Status 404 returned error can't find the container with id e4f7519d6cb11b31df14858a182dbedd28e63a221302accc5fd89d1a02b3c913
Oct 30 01:29:20.712626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:20.712572    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz" event={"ID":"04e0e257-3b3c-414f-a8ac-d4fef17ef3be","Type":"ContainerStarted","Data":"49beb32709139df625d2098b9215fa84c21149242137633743feb1f386ade5dc"}
Oct 30 01:29:20.713052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:20.712642    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz" event={"ID":"04e0e257-3b3c-414f-a8ac-d4fef17ef3be","Type":"ContainerStarted","Data":"e4f7519d6cb11b31df14858a182dbedd28e63a221302accc5fd89d1a02b3c913"}
Oct 30 01:29:20.749973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:20.749917    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz" podStartSLOduration=1.749902514 podStartE2EDuration="1.749902514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:20.74893126 +0000 UTC m=+2869.498586570" watchObservedRunningTime="2026-10-30 01:29:20.749902514 +0000 UTC m=+2869.499557822"
Oct 30 01:29:21.722137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:21.722103    4425 generic.go:356] "Generic (PLEG): container finished" podID="04e0e257-3b3c-414f-a8ac-d4fef17ef3be" containerID="49beb32709139df625d2098b9215fa84c21149242137633743feb1f386ade5dc" exitCode=0
Oct 30 01:29:21.722602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:21.722185    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz" event={"ID":"04e0e257-3b3c-414f-a8ac-d4fef17ef3be","Type":"ContainerDied","Data":"49beb32709139df625d2098b9215fa84c21149242137633743feb1f386ade5dc"}
Oct 30 01:29:22.914497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:22.914465    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"
Oct 30 01:29:22.993124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:22.992997    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-maas-api-xks-ca\") pod \"04e0e257-3b3c-414f-a8ac-d4fef17ef3be\" (UID: \"04e0e257-3b3c-414f-a8ac-d4fef17ef3be\") "
Oct 30 01:29:22.993124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:22.993111    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-maas-api-openshift-ca\") pod \"04e0e257-3b3c-414f-a8ac-d4fef17ef3be\" (UID: \"04e0e257-3b3c-414f-a8ac-d4fef17ef3be\") "
Oct 30 01:29:22.993465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:22.993199    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-kube-api-access-znpm6\" (UniqueName: \"kubernetes.io/projected/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-kube-api-access-znpm6\") pod \"04e0e257-3b3c-414f-a8ac-d4fef17ef3be\" (UID: \"04e0e257-3b3c-414f-a8ac-d4fef17ef3be\") "
Oct 30 01:29:22.994089 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:22.993641    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-maas-api-openshift-ca" pod "04e0e257-3b3c-414f-a8ac-d4fef17ef3be" (UID: "04e0e257-3b3c-414f-a8ac-d4fef17ef3be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:22.994089 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:22.994051    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:22.995646 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:22.995612    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-maas-api-xks-ca" pod "04e0e257-3b3c-414f-a8ac-d4fef17ef3be" (UID: "04e0e257-3b3c-414f-a8ac-d4fef17ef3be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:22.995790 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:22.995760    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-kube-api-access-znpm6" pod "04e0e257-3b3c-414f-a8ac-d4fef17ef3be" (UID: "04e0e257-3b3c-414f-a8ac-d4fef17ef3be"). InnerVolumeSpecName "kube-api-access-znpm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:23.095922 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:23.095875    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:23.095922 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:23.095915    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znpm6\" (UniqueName: \"kubernetes.io/projected/04e0e257-3b3c-414f-a8ac-d4fef17ef3be-kube-api-access-znpm6\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:23.738890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:23.738788    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"
Oct 30 01:29:23.738890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:23.738821    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz" event={"ID":"04e0e257-3b3c-414f-a8ac-d4fef17ef3be","Type":"ContainerDied","Data":"e4f7519d6cb11b31df14858a182dbedd28e63a221302accc5fd89d1a02b3c913"}
Oct 30 01:29:23.738890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:23.738870    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4f7519d6cb11b31df14858a182dbedd28e63a221302accc5fd89d1a02b3c913"
Oct 30 01:29:29.446523 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.446479    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"]
Oct 30 01:29:29.447329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.447311    4425 cpu_manager.go:414] "containerMap: removing container" podUID="04e0e257-3b3c-414f-a8ac-d4fef17ef3be" containerName="revoke-keys"
Oct 30 01:29:29.447329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.447330    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e0e257-3b3c-414f-a8ac-d4fef17ef3be" containerName="revoke-keys"
Oct 30 01:29:29.447460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.447451    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e0e257-3b3c-414f-a8ac-d4fef17ef3be" containerName="revoke-keys"
Oct 30 01:29:29.451628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.451609    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:29:29.477644 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.477605    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 01:29:29.477644 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.477604    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 01:29:29.477644 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.477604    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 01:29:29.477995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.477614    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0e3c0a7229d2\""
Oct 30 01:29:29.477995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.477617    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 01:29:29.477995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.477699    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-drckg\""
Oct 30 01:29:29.483541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.483499    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"]
Oct 30 01:29:29.592482 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.592439    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88e7d729-60a5-46d5-b177-fe3217157f61-provider-credentials\") pod \"payload-processing-external-model-574d87dd89-s7xlz\" (UID: \"88e7d729-60a5-46d5-b177-fe3217157f61\") " pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:29:29.592713 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.592648    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88e7d729-60a5-46d5-b177-fe3217157f61-routing-overlay-volume\") pod \"payload-processing-external-model-574d87dd89-s7xlz\" (UID: \"88e7d729-60a5-46d5-b177-fe3217157f61\") " pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:29:29.592713 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.592693    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl7g9\" (UniqueName: \"kubernetes.io/projected/88e7d729-60a5-46d5-b177-fe3217157f61-kube-api-access-vl7g9\") pod \"payload-processing-external-model-574d87dd89-s7xlz\" (UID: \"88e7d729-60a5-46d5-b177-fe3217157f61\") " pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:29:29.592855 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.592772    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e7d729-60a5-46d5-b177-fe3217157f61-plugins-config-volume\") pod \"payload-processing-external-model-574d87dd89-s7xlz\" (UID: \"88e7d729-60a5-46d5-b177-fe3217157f61\") " pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:29:29.694295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.694248    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88e7d729-60a5-46d5-b177-fe3217157f61-provider-credentials\") pod \"payload-processing-external-model-574d87dd89-s7xlz\" (UID: \"88e7d729-60a5-46d5-b177-fe3217157f61\") " pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:29:29.694527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.694374    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88e7d729-60a5-46d5-b177-fe3217157f61-routing-overlay-volume\") pod \"payload-processing-external-model-574d87dd89-s7xlz\" (UID: \"88e7d729-60a5-46d5-b177-fe3217157f61\") " pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:29:29.694527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.694400    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl7g9\" (UniqueName: \"kubernetes.io/projected/88e7d729-60a5-46d5-b177-fe3217157f61-kube-api-access-vl7g9\") pod \"payload-processing-external-model-574d87dd89-s7xlz\" (UID: \"88e7d729-60a5-46d5-b177-fe3217157f61\") " pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:29:29.694527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.694451    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e7d729-60a5-46d5-b177-fe3217157f61-plugins-config-volume\") pod \"payload-processing-external-model-574d87dd89-s7xlz\" (UID: \"88e7d729-60a5-46d5-b177-fe3217157f61\") " pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:29:29.695366 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.695339    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88e7d729-60a5-46d5-b177-fe3217157f61-routing-overlay-volume\") pod \"payload-processing-external-model-574d87dd89-s7xlz\" (UID: \"88e7d729-60a5-46d5-b177-fe3217157f61\") " pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:29:29.695485 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.695363    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e7d729-60a5-46d5-b177-fe3217157f61-plugins-config-volume\") pod \"payload-processing-external-model-574d87dd89-s7xlz\" (UID: \"88e7d729-60a5-46d5-b177-fe3217157f61\") " pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:29:29.697446 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.697396    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88e7d729-60a5-46d5-b177-fe3217157f61-provider-credentials\") pod \"payload-processing-external-model-574d87dd89-s7xlz\" (UID: \"88e7d729-60a5-46d5-b177-fe3217157f61\") " pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:29:29.716342 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.716311    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl7g9\" (UniqueName: \"kubernetes.io/projected/88e7d729-60a5-46d5-b177-fe3217157f61-kube-api-access-vl7g9\") pod \"payload-processing-external-model-574d87dd89-s7xlz\" (UID: \"88e7d729-60a5-46d5-b177-fe3217157f61\") " pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:29:29.767133 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:29.767093    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:29:30.080062 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:30.080031    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"]
Oct 30 01:29:30.081621 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:29:30.081587    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e7d729_60a5_46d5_b177_fe3217157f61.slice/crio-2c2896c2e486f535d213c2130736fc2f8a769fba7e3227fab7e0f70c47c7aec8 WatchSource:0}: Error finding container 2c2896c2e486f535d213c2130736fc2f8a769fba7e3227fab7e0f70c47c7aec8: Status 404 returned error can't find the container with id 2c2896c2e486f535d213c2130736fc2f8a769fba7e3227fab7e0f70c47c7aec8
Oct 30 01:29:30.817412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:30.817368    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz" event={"ID":"88e7d729-60a5-46d5-b177-fe3217157f61","Type":"ContainerStarted","Data":"c0d0230ff0b0ea27331da6dff70708ea324a34b3f9c7f4e737a86e9b3f7cd553"}
Oct 30 01:29:30.817915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:30.817432    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz" event={"ID":"88e7d729-60a5-46d5-b177-fe3217157f61","Type":"ContainerStarted","Data":"2c2896c2e486f535d213c2130736fc2f8a769fba7e3227fab7e0f70c47c7aec8"}
Oct 30 01:29:30.817915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:30.817473    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:29:30.858719 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:30.858655    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz" podStartSLOduration=1.8586363989999999 podStartE2EDuration="1.858636399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:30.854238059 +0000 UTC m=+2879.603893369" watchObservedRunningTime="2026-10-30 01:29:30.858636399 +0000 UTC m=+2879.608291708"
Oct 30 01:29:32.002367 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:32.002330    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"]
Oct 30 01:29:37.003611 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.003558    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8" podUID="0169b30d-41bf-40bf-9e5d-f97e9a533494" containerName="payload-processing" containerID="cri-o://c261fa0974e867beaea78a73a42269a28c6bd3491e26ea5c32f8f6cc971c5ed0" gracePeriod=25
Oct 30 01:29:37.312497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.312464    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:29:37.424224 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.424146    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0169b30d-41bf-40bf-9e5d-f97e9a533494-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0169b30d-41bf-40bf-9e5d-f97e9a533494-routing-overlay-volume\") pod \"0169b30d-41bf-40bf-9e5d-f97e9a533494\" (UID: \"0169b30d-41bf-40bf-9e5d-f97e9a533494\") "
Oct 30 01:29:37.424471 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.424284    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0169b30d-41bf-40bf-9e5d-f97e9a533494-kube-api-access-kz9fm\" (UniqueName: \"kubernetes.io/projected/0169b30d-41bf-40bf-9e5d-f97e9a533494-kube-api-access-kz9fm\") pod \"0169b30d-41bf-40bf-9e5d-f97e9a533494\" (UID: \"0169b30d-41bf-40bf-9e5d-f97e9a533494\") "
Oct 30 01:29:37.424471 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.424426    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0169b30d-41bf-40bf-9e5d-f97e9a533494-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0169b30d-41bf-40bf-9e5d-f97e9a533494-provider-credentials\") pod \"0169b30d-41bf-40bf-9e5d-f97e9a533494\" (UID: \"0169b30d-41bf-40bf-9e5d-f97e9a533494\") "
Oct 30 01:29:37.424614 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.424578    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0169b30d-41bf-40bf-9e5d-f97e9a533494-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0169b30d-41bf-40bf-9e5d-f97e9a533494-plugins-config-volume\") pod \"0169b30d-41bf-40bf-9e5d-f97e9a533494\" (UID: \"0169b30d-41bf-40bf-9e5d-f97e9a533494\") "
Oct 30 01:29:37.424701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.424674    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0169b30d-41bf-40bf-9e5d-f97e9a533494-routing-overlay-volume" pod "0169b30d-41bf-40bf-9e5d-f97e9a533494" (UID: "0169b30d-41bf-40bf-9e5d-f97e9a533494"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:37.425101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.425071    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0169b30d-41bf-40bf-9e5d-f97e9a533494-plugins-config-volume" pod "0169b30d-41bf-40bf-9e5d-f97e9a533494" (UID: "0169b30d-41bf-40bf-9e5d-f97e9a533494"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:37.425278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.425195    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0169b30d-41bf-40bf-9e5d-f97e9a533494-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:37.425278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.425212    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0169b30d-41bf-40bf-9e5d-f97e9a533494-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:37.426984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.426957    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0169b30d-41bf-40bf-9e5d-f97e9a533494-provider-credentials" pod "0169b30d-41bf-40bf-9e5d-f97e9a533494" (UID: "0169b30d-41bf-40bf-9e5d-f97e9a533494"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:37.427096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.426996    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0169b30d-41bf-40bf-9e5d-f97e9a533494-kube-api-access-kz9fm" pod "0169b30d-41bf-40bf-9e5d-f97e9a533494" (UID: "0169b30d-41bf-40bf-9e5d-f97e9a533494"). InnerVolumeSpecName "kube-api-access-kz9fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:37.526885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.526789    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz9fm\" (UniqueName: \"kubernetes.io/projected/0169b30d-41bf-40bf-9e5d-f97e9a533494-kube-api-access-kz9fm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:37.526885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.526826    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0169b30d-41bf-40bf-9e5d-f97e9a533494-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:37.557756 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:29:37.557719    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:29:37.874793 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.874686    4425 generic.go:356] "Generic (PLEG): container finished" podID="0169b30d-41bf-40bf-9e5d-f97e9a533494" containerID="c261fa0974e867beaea78a73a42269a28c6bd3491e26ea5c32f8f6cc971c5ed0" exitCode=0
Oct 30 01:29:37.874793 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.874764    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"
Oct 30 01:29:37.875013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.874766    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8" event={"ID":"0169b30d-41bf-40bf-9e5d-f97e9a533494","Type":"ContainerDied","Data":"c261fa0974e867beaea78a73a42269a28c6bd3491e26ea5c32f8f6cc971c5ed0"}
Oct 30 01:29:37.875013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.874897    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8" event={"ID":"0169b30d-41bf-40bf-9e5d-f97e9a533494","Type":"ContainerDied","Data":"6ff62fda6a77cdbe727f11ca183ebdc0a7006816ab2445e981f2c62e54a822f5"}
Oct 30 01:29:37.875013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.874927    4425 scope.go:122] "RemoveContainer" containerID="c261fa0974e867beaea78a73a42269a28c6bd3491e26ea5c32f8f6cc971c5ed0"
Oct 30 01:29:37.887230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.887073    4425 scope.go:122] "RemoveContainer" containerID="c261fa0974e867beaea78a73a42269a28c6bd3491e26ea5c32f8f6cc971c5ed0"
Oct 30 01:29:37.887565 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:29:37.887530    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c261fa0974e867beaea78a73a42269a28c6bd3491e26ea5c32f8f6cc971c5ed0\": container with ID starting with c261fa0974e867beaea78a73a42269a28c6bd3491e26ea5c32f8f6cc971c5ed0 not found: ID does not exist" containerID="c261fa0974e867beaea78a73a42269a28c6bd3491e26ea5c32f8f6cc971c5ed0"
Oct 30 01:29:37.887673 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.887578    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c261fa0974e867beaea78a73a42269a28c6bd3491e26ea5c32f8f6cc971c5ed0"} err="failed to get container status \"c261fa0974e867beaea78a73a42269a28c6bd3491e26ea5c32f8f6cc971c5ed0\": rpc error: code = NotFound desc = could not find container \"c261fa0974e867beaea78a73a42269a28c6bd3491e26ea5c32f8f6cc971c5ed0\": container with ID starting with c261fa0974e867beaea78a73a42269a28c6bd3491e26ea5c32f8f6cc971c5ed0 not found: ID does not exist"
Oct 30 01:29:37.923638 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.923589    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"]
Oct 30 01:29:37.934174 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:37.934119    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-757b6cjq8"]
Oct 30 01:29:39.425313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:39.425275    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0169b30d-41bf-40bf-9e5d-f97e9a533494" path="/var/lib/kubelet/pods/0169b30d-41bf-40bf-9e5d-f97e9a533494/volumes"
Oct 30 01:29:41.828749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:41.828720    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:29:45.156684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.156642    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"]
Oct 30 01:29:45.157773 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.157743    4425 cpu_manager.go:414] "containerMap: removing container" podUID="0169b30d-41bf-40bf-9e5d-f97e9a533494" containerName="payload-processing"
Oct 30 01:29:45.157773 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.157766    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0169b30d-41bf-40bf-9e5d-f97e9a533494" containerName="payload-processing"
Oct 30 01:29:45.157963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.157946    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0169b30d-41bf-40bf-9e5d-f97e9a533494" containerName="payload-processing"
Oct 30 01:29:45.163330 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.163304    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"
Oct 30 01:29:45.182771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.182737    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"]
Oct 30 01:29:45.186831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.186793    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-06d7d794cf59\""
Oct 30 01:29:45.340423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.340383    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq9jn\" (UniqueName: \"kubernetes.io/projected/bc600f07-5918-43a2-89a5-96388018a0f5-kube-api-access-dq9jn\") pod \"payload-processing-external-model-79c9fb4bbb-rdx2h\" (UID: \"bc600f07-5918-43a2-89a5-96388018a0f5\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"
Oct 30 01:29:45.340629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.340594    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc600f07-5918-43a2-89a5-96388018a0f5-provider-credentials\") pod \"payload-processing-external-model-79c9fb4bbb-rdx2h\" (UID: \"bc600f07-5918-43a2-89a5-96388018a0f5\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"
Oct 30 01:29:45.340767 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.340738    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc600f07-5918-43a2-89a5-96388018a0f5-routing-overlay-volume\") pod \"payload-processing-external-model-79c9fb4bbb-rdx2h\" (UID: \"bc600f07-5918-43a2-89a5-96388018a0f5\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"
Oct 30 01:29:45.340815 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.340800    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc600f07-5918-43a2-89a5-96388018a0f5-plugins-config-volume\") pod \"payload-processing-external-model-79c9fb4bbb-rdx2h\" (UID: \"bc600f07-5918-43a2-89a5-96388018a0f5\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"
Oct 30 01:29:45.442463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.442343    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc600f07-5918-43a2-89a5-96388018a0f5-provider-credentials\") pod \"payload-processing-external-model-79c9fb4bbb-rdx2h\" (UID: \"bc600f07-5918-43a2-89a5-96388018a0f5\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"
Oct 30 01:29:45.442674 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.442562    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc600f07-5918-43a2-89a5-96388018a0f5-routing-overlay-volume\") pod \"payload-processing-external-model-79c9fb4bbb-rdx2h\" (UID: \"bc600f07-5918-43a2-89a5-96388018a0f5\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"
Oct 30 01:29:45.442827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.442800    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc600f07-5918-43a2-89a5-96388018a0f5-plugins-config-volume\") pod \"payload-processing-external-model-79c9fb4bbb-rdx2h\" (UID: \"bc600f07-5918-43a2-89a5-96388018a0f5\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"
Oct 30 01:29:45.442998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.442971    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq9jn\" (UniqueName: \"kubernetes.io/projected/bc600f07-5918-43a2-89a5-96388018a0f5-kube-api-access-dq9jn\") pod \"payload-processing-external-model-79c9fb4bbb-rdx2h\" (UID: \"bc600f07-5918-43a2-89a5-96388018a0f5\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"
Oct 30 01:29:45.443780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.443752    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc600f07-5918-43a2-89a5-96388018a0f5-routing-overlay-volume\") pod \"payload-processing-external-model-79c9fb4bbb-rdx2h\" (UID: \"bc600f07-5918-43a2-89a5-96388018a0f5\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"
Oct 30 01:29:45.443885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.443782    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc600f07-5918-43a2-89a5-96388018a0f5-plugins-config-volume\") pod \"payload-processing-external-model-79c9fb4bbb-rdx2h\" (UID: \"bc600f07-5918-43a2-89a5-96388018a0f5\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"
Oct 30 01:29:45.445590 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.445568    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc600f07-5918-43a2-89a5-96388018a0f5-provider-credentials\") pod \"payload-processing-external-model-79c9fb4bbb-rdx2h\" (UID: \"bc600f07-5918-43a2-89a5-96388018a0f5\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"
Oct 30 01:29:45.464061 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.464025    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq9jn\" (UniqueName: \"kubernetes.io/projected/bc600f07-5918-43a2-89a5-96388018a0f5-kube-api-access-dq9jn\") pod \"payload-processing-external-model-79c9fb4bbb-rdx2h\" (UID: \"bc600f07-5918-43a2-89a5-96388018a0f5\") " pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"
Oct 30 01:29:45.479900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.479866    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"
Oct 30 01:29:45.793541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.793499    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"]
Oct 30 01:29:45.794735 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:29:45.794699    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc600f07_5918_43a2_89a5_96388018a0f5.slice/crio-1047aeaca47b44e404c342d3b46df637e55a1cb5f307cc5ba406d6d89345550e WatchSource:0}: Error finding container 1047aeaca47b44e404c342d3b46df637e55a1cb5f307cc5ba406d6d89345550e: Status 404 returned error can't find the container with id 1047aeaca47b44e404c342d3b46df637e55a1cb5f307cc5ba406d6d89345550e
Oct 30 01:29:45.945594 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.945554    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h" event={"ID":"bc600f07-5918-43a2-89a5-96388018a0f5","Type":"ContainerStarted","Data":"8450466743a33e38f48fa07b9fa4209b24dcdde52566644cea459633c62b803d"}
Oct 30 01:29:45.945795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.945616    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h" event={"ID":"bc600f07-5918-43a2-89a5-96388018a0f5","Type":"ContainerStarted","Data":"1047aeaca47b44e404c342d3b46df637e55a1cb5f307cc5ba406d6d89345550e"}
Oct 30 01:29:45.945795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.945644    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"
Oct 30 01:29:45.986951 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:45.986883    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h" podStartSLOduration=0.986864939 podStartE2EDuration="986.864939ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:45.98239856 +0000 UTC m=+2894.732053871" watchObservedRunningTime="2026-10-30 01:29:45.986864939 +0000 UTC m=+2894.736520248"
Oct 30 01:29:48.083527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.083485    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"]
Oct 30 01:29:48.117065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.117031    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"]
Oct 30 01:29:48.122392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.122361    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:48.137163 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.137115    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9da187451cb6\""
Oct 30 01:29:48.147753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.147716    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"]
Oct 30 01:29:48.181320 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.181273    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5987\" (UniqueName: \"kubernetes.io/projected/48619b64-2123-4936-bda6-ae202994ebb5-kube-api-access-s5987\") pod \"payload-processing-external-model-5f6fbbf746-lrndc\" (UID: \"48619b64-2123-4936-bda6-ae202994ebb5\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:48.181572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.181362    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48619b64-2123-4936-bda6-ae202994ebb5-routing-overlay-volume\") pod \"payload-processing-external-model-5f6fbbf746-lrndc\" (UID: \"48619b64-2123-4936-bda6-ae202994ebb5\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:48.181572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.181523    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48619b64-2123-4936-bda6-ae202994ebb5-provider-credentials\") pod \"payload-processing-external-model-5f6fbbf746-lrndc\" (UID: \"48619b64-2123-4936-bda6-ae202994ebb5\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:48.181805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.181776    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48619b64-2123-4936-bda6-ae202994ebb5-plugins-config-volume\") pod \"payload-processing-external-model-5f6fbbf746-lrndc\" (UID: \"48619b64-2123-4936-bda6-ae202994ebb5\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:48.283434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.283390    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5987\" (UniqueName: \"kubernetes.io/projected/48619b64-2123-4936-bda6-ae202994ebb5-kube-api-access-s5987\") pod \"payload-processing-external-model-5f6fbbf746-lrndc\" (UID: \"48619b64-2123-4936-bda6-ae202994ebb5\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:48.283627 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.283453    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48619b64-2123-4936-bda6-ae202994ebb5-routing-overlay-volume\") pod \"payload-processing-external-model-5f6fbbf746-lrndc\" (UID: \"48619b64-2123-4936-bda6-ae202994ebb5\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:48.283627 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.283495    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48619b64-2123-4936-bda6-ae202994ebb5-provider-credentials\") pod \"payload-processing-external-model-5f6fbbf746-lrndc\" (UID: \"48619b64-2123-4936-bda6-ae202994ebb5\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:48.283701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.283643    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48619b64-2123-4936-bda6-ae202994ebb5-plugins-config-volume\") pod \"payload-processing-external-model-5f6fbbf746-lrndc\" (UID: \"48619b64-2123-4936-bda6-ae202994ebb5\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:48.284238 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.284210    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48619b64-2123-4936-bda6-ae202994ebb5-routing-overlay-volume\") pod \"payload-processing-external-model-5f6fbbf746-lrndc\" (UID: \"48619b64-2123-4936-bda6-ae202994ebb5\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:48.284370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.284332    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48619b64-2123-4936-bda6-ae202994ebb5-plugins-config-volume\") pod \"payload-processing-external-model-5f6fbbf746-lrndc\" (UID: \"48619b64-2123-4936-bda6-ae202994ebb5\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:48.286323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.286301    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48619b64-2123-4936-bda6-ae202994ebb5-provider-credentials\") pod \"payload-processing-external-model-5f6fbbf746-lrndc\" (UID: \"48619b64-2123-4936-bda6-ae202994ebb5\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:48.305118 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.305082    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5987\" (UniqueName: \"kubernetes.io/projected/48619b64-2123-4936-bda6-ae202994ebb5-kube-api-access-s5987\") pod \"payload-processing-external-model-5f6fbbf746-lrndc\" (UID: \"48619b64-2123-4936-bda6-ae202994ebb5\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:48.439519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.439426    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:48.755037 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.755000    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"]
Oct 30 01:29:48.755994 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:29:48.755966    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48619b64_2123_4936_bda6_ae202994ebb5.slice/crio-9c6348d594cc5e40cd006819f149613a0db2de8c13f43511e3be7997ac3241b8 WatchSource:0}: Error finding container 9c6348d594cc5e40cd006819f149613a0db2de8c13f43511e3be7997ac3241b8: Status 404 returned error can't find the container with id 9c6348d594cc5e40cd006819f149613a0db2de8c13f43511e3be7997ac3241b8
Oct 30 01:29:48.974538 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.974494    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc" event={"ID":"48619b64-2123-4936-bda6-ae202994ebb5","Type":"ContainerStarted","Data":"9a129f1af0ea4c8a4b77586b8f2a58a889bc9f005664889cdd70a629af4716f0"}
Oct 30 01:29:48.974740 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.974557    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc" event={"ID":"48619b64-2123-4936-bda6-ae202994ebb5","Type":"ContainerStarted","Data":"9c6348d594cc5e40cd006819f149613a0db2de8c13f43511e3be7997ac3241b8"}
Oct 30 01:29:48.974740 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:48.974586    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:49.012417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:49.012359    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc" podStartSLOduration=1.012340156 podStartE2EDuration="1.012340156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:49.010568062 +0000 UTC m=+2897.760223370" watchObservedRunningTime="2026-10-30 01:29:49.012340156 +0000 UTC m=+2897.761995465"
Oct 30 01:29:53.084554 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:53.084499    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h" podUID="bc600f07-5918-43a2-89a5-96388018a0f5" containerName="payload-processing" containerID="cri-o://8450466743a33e38f48fa07b9fa4209b24dcdde52566644cea459633c62b803d" gracePeriod=25
Oct 30 01:29:53.296347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:53.293710    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"]
Oct 30 01:29:53.297771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:53.297739    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:53.543236 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:53.540554    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"
Oct 30 01:29:53.673732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:53.673688    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc600f07-5918-43a2-89a5-96388018a0f5-kube-api-access-dq9jn\" (UniqueName: \"kubernetes.io/projected/bc600f07-5918-43a2-89a5-96388018a0f5-kube-api-access-dq9jn\") pod \"bc600f07-5918-43a2-89a5-96388018a0f5\" (UID: \"bc600f07-5918-43a2-89a5-96388018a0f5\") "
Oct 30 01:29:53.673964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:53.673818    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc600f07-5918-43a2-89a5-96388018a0f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc600f07-5918-43a2-89a5-96388018a0f5-plugins-config-volume\") pod \"bc600f07-5918-43a2-89a5-96388018a0f5\" (UID: \"bc600f07-5918-43a2-89a5-96388018a0f5\") "
Oct 30 01:29:53.673964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:53.673942    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc600f07-5918-43a2-89a5-96388018a0f5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc600f07-5918-43a2-89a5-96388018a0f5-routing-overlay-volume\") pod \"bc600f07-5918-43a2-89a5-96388018a0f5\" (UID: \"bc600f07-5918-43a2-89a5-96388018a0f5\") "
Oct 30 01:29:53.674223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:53.674101    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc600f07-5918-43a2-89a5-96388018a0f5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc600f07-5918-43a2-89a5-96388018a0f5-provider-credentials\") pod \"bc600f07-5918-43a2-89a5-96388018a0f5\" (UID: \"bc600f07-5918-43a2-89a5-96388018a0f5\") "
Oct 30 01:29:53.675334 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:53.675306    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc600f07-5918-43a2-89a5-96388018a0f5-routing-overlay-volume" pod "bc600f07-5918-43a2-89a5-96388018a0f5" (UID: "bc600f07-5918-43a2-89a5-96388018a0f5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:53.675619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:53.675593    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc600f07-5918-43a2-89a5-96388018a0f5-plugins-config-volume" pod "bc600f07-5918-43a2-89a5-96388018a0f5" (UID: "bc600f07-5918-43a2-89a5-96388018a0f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:53.677561 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:53.677533    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc600f07-5918-43a2-89a5-96388018a0f5-kube-api-access-dq9jn" pod "bc600f07-5918-43a2-89a5-96388018a0f5" (UID: "bc600f07-5918-43a2-89a5-96388018a0f5"). InnerVolumeSpecName "kube-api-access-dq9jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:53.677682 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:53.677638    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc600f07-5918-43a2-89a5-96388018a0f5-provider-credentials" pod "bc600f07-5918-43a2-89a5-96388018a0f5" (UID: "bc600f07-5918-43a2-89a5-96388018a0f5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:53.776338 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:53.776255    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc600f07-5918-43a2-89a5-96388018a0f5-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:53.776338 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:53.776292    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq9jn\" (UniqueName: \"kubernetes.io/projected/bc600f07-5918-43a2-89a5-96388018a0f5-kube-api-access-dq9jn\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:53.776338 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:53.776302    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc600f07-5918-43a2-89a5-96388018a0f5-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:53.776338 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:53.776313    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc600f07-5918-43a2-89a5-96388018a0f5-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:54.023019 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.022978    4425 generic.go:356] "Generic (PLEG): container finished" podID="bc600f07-5918-43a2-89a5-96388018a0f5" containerID="8450466743a33e38f48fa07b9fa4209b24dcdde52566644cea459633c62b803d" exitCode=0
Oct 30 01:29:54.023223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.023032    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"
Oct 30 01:29:54.023223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.023026    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h" event={"ID":"bc600f07-5918-43a2-89a5-96388018a0f5","Type":"ContainerDied","Data":"8450466743a33e38f48fa07b9fa4209b24dcdde52566644cea459633c62b803d"}
Oct 30 01:29:54.023223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.023176    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h" event={"ID":"bc600f07-5918-43a2-89a5-96388018a0f5","Type":"ContainerDied","Data":"1047aeaca47b44e404c342d3b46df637e55a1cb5f307cc5ba406d6d89345550e"}
Oct 30 01:29:54.023223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.023206    4425 scope.go:122] "RemoveContainer" containerID="8450466743a33e38f48fa07b9fa4209b24dcdde52566644cea459633c62b803d"
Oct 30 01:29:54.035296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.035271    4425 scope.go:122] "RemoveContainer" containerID="8450466743a33e38f48fa07b9fa4209b24dcdde52566644cea459633c62b803d"
Oct 30 01:29:54.035665 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:29:54.035641    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8450466743a33e38f48fa07b9fa4209b24dcdde52566644cea459633c62b803d\": container with ID starting with 8450466743a33e38f48fa07b9fa4209b24dcdde52566644cea459633c62b803d not found: ID does not exist" containerID="8450466743a33e38f48fa07b9fa4209b24dcdde52566644cea459633c62b803d"
Oct 30 01:29:54.035712 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.035677    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8450466743a33e38f48fa07b9fa4209b24dcdde52566644cea459633c62b803d"} err="failed to get container status \"8450466743a33e38f48fa07b9fa4209b24dcdde52566644cea459633c62b803d\": rpc error: code = NotFound desc = could not find container \"8450466743a33e38f48fa07b9fa4209b24dcdde52566644cea459633c62b803d\": container with ID starting with 8450466743a33e38f48fa07b9fa4209b24dcdde52566644cea459633c62b803d not found: ID does not exist"
Oct 30 01:29:54.076765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.076722    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"]
Oct 30 01:29:54.082767 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.082728    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-79c9fb4bbb-rdx2h"]
Oct 30 01:29:54.477216 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.477088    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"]
Oct 30 01:29:54.478335 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.478308    4425 cpu_manager.go:414] "containerMap: removing container" podUID="bc600f07-5918-43a2-89a5-96388018a0f5" containerName="payload-processing"
Oct 30 01:29:54.478456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.478339    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc600f07-5918-43a2-89a5-96388018a0f5" containerName="payload-processing"
Oct 30 01:29:54.478633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.478617    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc600f07-5918-43a2-89a5-96388018a0f5" containerName="payload-processing"
Oct 30 01:29:54.485268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.485235    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"
Oct 30 01:29:54.498016 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.497983    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:29:54.498225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.498128    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:29:54.502571 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.502543    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"]
Oct 30 01:29:54.589955 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.589910    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b914fb03-5880-4634-86c1-4d43a72ec34d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18b4734-wt9x8\" (UID: \"b914fb03-5880-4634-86c1-4d43a72ec34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"
Oct 30 01:29:54.590204 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.589983    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b914fb03-5880-4634-86c1-4d43a72ec34d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18b4734-wt9x8\" (UID: \"b914fb03-5880-4634-86c1-4d43a72ec34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"
Oct 30 01:29:54.590297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.590215    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwwd5\" (UniqueName: \"kubernetes.io/projected/b914fb03-5880-4634-86c1-4d43a72ec34d-kube-api-access-lwwd5\") pod \"maas-api-revoke-sub-f18b4734-wt9x8\" (UID: \"b914fb03-5880-4634-86c1-4d43a72ec34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"
Oct 30 01:29:54.691706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.691661    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b914fb03-5880-4634-86c1-4d43a72ec34d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18b4734-wt9x8\" (UID: \"b914fb03-5880-4634-86c1-4d43a72ec34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"
Oct 30 01:29:54.691941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.691738    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b914fb03-5880-4634-86c1-4d43a72ec34d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18b4734-wt9x8\" (UID: \"b914fb03-5880-4634-86c1-4d43a72ec34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"
Oct 30 01:29:54.691941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.691816    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwwd5\" (UniqueName: \"kubernetes.io/projected/b914fb03-5880-4634-86c1-4d43a72ec34d-kube-api-access-lwwd5\") pod \"maas-api-revoke-sub-f18b4734-wt9x8\" (UID: \"b914fb03-5880-4634-86c1-4d43a72ec34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"
Oct 30 01:29:54.692515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.692489    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b914fb03-5880-4634-86c1-4d43a72ec34d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18b4734-wt9x8\" (UID: \"b914fb03-5880-4634-86c1-4d43a72ec34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"
Oct 30 01:29:54.694427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.694396    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b914fb03-5880-4634-86c1-4d43a72ec34d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18b4734-wt9x8\" (UID: \"b914fb03-5880-4634-86c1-4d43a72ec34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"
Oct 30 01:29:54.713624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.713583    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwwd5\" (UniqueName: \"kubernetes.io/projected/b914fb03-5880-4634-86c1-4d43a72ec34d-kube-api-access-lwwd5\") pod \"maas-api-revoke-sub-f18b4734-wt9x8\" (UID: \"b914fb03-5880-4634-86c1-4d43a72ec34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"
Oct 30 01:29:54.806822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:54.806777    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"
Oct 30 01:29:55.123329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:55.123288    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"]
Oct 30 01:29:55.125759 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:29:55.125731    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb914fb03_5880_4634_86c1_4d43a72ec34d.slice/crio-8f77d82d0d397dbf49f4669d3fa19cbaffe2f19980a81e29b6edcf06ce0929bd WatchSource:0}: Error finding container 8f77d82d0d397dbf49f4669d3fa19cbaffe2f19980a81e29b6edcf06ce0929bd: Status 404 returned error can't find the container with id 8f77d82d0d397dbf49f4669d3fa19cbaffe2f19980a81e29b6edcf06ce0929bd
Oct 30 01:29:55.426184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:55.426076    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc600f07-5918-43a2-89a5-96388018a0f5" path="/var/lib/kubelet/pods/bc600f07-5918-43a2-89a5-96388018a0f5/volumes"
Oct 30 01:29:55.680334 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:55.680239    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"]
Oct 30 01:29:55.685123 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:55.685099    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:29:55.706098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:55.706057    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"]
Oct 30 01:29:55.810714 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:55.810678    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggxrt\" (UniqueName: \"kubernetes.io/projected/d435c726-6e32-4624-86ec-fa093d531c40-kube-api-access-ggxrt\") pod \"payload-processing-external-model-5f6fbbf746-8j44n\" (UID: \"d435c726-6e32-4624-86ec-fa093d531c40\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:29:55.810909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:55.810871    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d435c726-6e32-4624-86ec-fa093d531c40-provider-credentials\") pod \"payload-processing-external-model-5f6fbbf746-8j44n\" (UID: \"d435c726-6e32-4624-86ec-fa093d531c40\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:29:55.810957 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:55.810910    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d435c726-6e32-4624-86ec-fa093d531c40-routing-overlay-volume\") pod \"payload-processing-external-model-5f6fbbf746-8j44n\" (UID: \"d435c726-6e32-4624-86ec-fa093d531c40\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:29:55.811087 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:55.811067    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d435c726-6e32-4624-86ec-fa093d531c40-plugins-config-volume\") pod \"payload-processing-external-model-5f6fbbf746-8j44n\" (UID: \"d435c726-6e32-4624-86ec-fa093d531c40\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:29:55.912686 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:55.912649    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d435c726-6e32-4624-86ec-fa093d531c40-routing-overlay-volume\") pod \"payload-processing-external-model-5f6fbbf746-8j44n\" (UID: \"d435c726-6e32-4624-86ec-fa093d531c40\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:29:55.912892 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:55.912777    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d435c726-6e32-4624-86ec-fa093d531c40-plugins-config-volume\") pod \"payload-processing-external-model-5f6fbbf746-8j44n\" (UID: \"d435c726-6e32-4624-86ec-fa093d531c40\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:29:55.912892 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:55.912850    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggxrt\" (UniqueName: \"kubernetes.io/projected/d435c726-6e32-4624-86ec-fa093d531c40-kube-api-access-ggxrt\") pod \"payload-processing-external-model-5f6fbbf746-8j44n\" (UID: \"d435c726-6e32-4624-86ec-fa093d531c40\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:29:55.913006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:55.912997    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d435c726-6e32-4624-86ec-fa093d531c40-provider-credentials\") pod \"payload-processing-external-model-5f6fbbf746-8j44n\" (UID: \"d435c726-6e32-4624-86ec-fa093d531c40\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:29:55.913548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:55.913521    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d435c726-6e32-4624-86ec-fa093d531c40-routing-overlay-volume\") pod \"payload-processing-external-model-5f6fbbf746-8j44n\" (UID: \"d435c726-6e32-4624-86ec-fa093d531c40\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:29:55.913667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:55.913612    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d435c726-6e32-4624-86ec-fa093d531c40-plugins-config-volume\") pod \"payload-processing-external-model-5f6fbbf746-8j44n\" (UID: \"d435c726-6e32-4624-86ec-fa093d531c40\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:29:55.915699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:55.915678    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d435c726-6e32-4624-86ec-fa093d531c40-provider-credentials\") pod \"payload-processing-external-model-5f6fbbf746-8j44n\" (UID: \"d435c726-6e32-4624-86ec-fa093d531c40\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:29:55.934174 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:55.934085    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggxrt\" (UniqueName: \"kubernetes.io/projected/d435c726-6e32-4624-86ec-fa093d531c40-kube-api-access-ggxrt\") pod \"payload-processing-external-model-5f6fbbf746-8j44n\" (UID: \"d435c726-6e32-4624-86ec-fa093d531c40\") " pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:29:56.020343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:56.020301    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:29:56.057001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:56.056955    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8" event={"ID":"b914fb03-5880-4634-86c1-4d43a72ec34d","Type":"ContainerStarted","Data":"31598938d29994dfdcc48011a9d878ed46a1a8a53bd028150ff272fd6f27f712"}
Oct 30 01:29:56.057267 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:56.057029    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8" event={"ID":"b914fb03-5880-4634-86c1-4d43a72ec34d","Type":"ContainerStarted","Data":"8f77d82d0d397dbf49f4669d3fa19cbaffe2f19980a81e29b6edcf06ce0929bd"}
Oct 30 01:29:56.095517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:56.093840    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8" podStartSLOduration=2.093826303 podStartE2EDuration="2.093826303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:56.093047666 +0000 UTC m=+2904.842703010" watchObservedRunningTime="2026-10-30 01:29:56.093826303 +0000 UTC m=+2904.843481612"
Oct 30 01:29:56.338426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:56.338390    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"]
Oct 30 01:29:56.340117 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:29:56.340077    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd435c726_6e32_4624_86ec_fa093d531c40.slice/crio-315d2009a95a3feb24710e4f1e0d5519112f3f1e1f3cf83e5c30a197560a476d WatchSource:0}: Error finding container 315d2009a95a3feb24710e4f1e0d5519112f3f1e1f3cf83e5c30a197560a476d: Status 404 returned error can't find the container with id 315d2009a95a3feb24710e4f1e0d5519112f3f1e1f3cf83e5c30a197560a476d
Oct 30 01:29:57.067485 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:57.067448    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n" event={"ID":"d435c726-6e32-4624-86ec-fa093d531c40","Type":"ContainerStarted","Data":"ac2b3961a85e773253681e5a26a40824176021adb175792c5461516fc1ea6085"}
Oct 30 01:29:57.067934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:57.067507    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n" event={"ID":"d435c726-6e32-4624-86ec-fa093d531c40","Type":"ContainerStarted","Data":"315d2009a95a3feb24710e4f1e0d5519112f3f1e1f3cf83e5c30a197560a476d"}
Oct 30 01:29:57.067934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:57.067561    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:29:57.069289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:57.069266    4425 generic.go:356] "Generic (PLEG): container finished" podID="b914fb03-5880-4634-86c1-4d43a72ec34d" containerID="31598938d29994dfdcc48011a9d878ed46a1a8a53bd028150ff272fd6f27f712" exitCode=0
Oct 30 01:29:57.069373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:57.069346    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8" event={"ID":"b914fb03-5880-4634-86c1-4d43a72ec34d","Type":"ContainerDied","Data":"31598938d29994dfdcc48011a9d878ed46a1a8a53bd028150ff272fd6f27f712"}
Oct 30 01:29:57.110561 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:57.110504    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n" podStartSLOduration=2.110488123 podStartE2EDuration="2.110488123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:57.108076156 +0000 UTC m=+2905.857731467" watchObservedRunningTime="2026-10-30 01:29:57.110488123 +0000 UTC m=+2905.860143432"
Oct 30 01:29:58.281423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.281390    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"
Oct 30 01:29:58.294169 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.294111    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc" podUID="48619b64-2123-4936-bda6-ae202994ebb5" containerName="payload-processing" containerID="cri-o://9a129f1af0ea4c8a4b77586b8f2a58a889bc9f005664889cdd70a629af4716f0" gracePeriod=25
Oct 30 01:29:58.448580 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.448540    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b914fb03-5880-4634-86c1-4d43a72ec34d-kube-api-access-lwwd5\" (UniqueName: \"kubernetes.io/projected/b914fb03-5880-4634-86c1-4d43a72ec34d-kube-api-access-lwwd5\") pod \"b914fb03-5880-4634-86c1-4d43a72ec34d\" (UID: \"b914fb03-5880-4634-86c1-4d43a72ec34d\") "
Oct 30 01:29:58.448813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.448661    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b914fb03-5880-4634-86c1-4d43a72ec34d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b914fb03-5880-4634-86c1-4d43a72ec34d-maas-api-openshift-ca\") pod \"b914fb03-5880-4634-86c1-4d43a72ec34d\" (UID: \"b914fb03-5880-4634-86c1-4d43a72ec34d\") "
Oct 30 01:29:58.448881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.448816    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b914fb03-5880-4634-86c1-4d43a72ec34d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b914fb03-5880-4634-86c1-4d43a72ec34d-maas-api-xks-ca\") pod \"b914fb03-5880-4634-86c1-4d43a72ec34d\" (UID: \"b914fb03-5880-4634-86c1-4d43a72ec34d\") "
Oct 30 01:29:58.449049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.449023    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b914fb03-5880-4634-86c1-4d43a72ec34d-maas-api-openshift-ca" pod "b914fb03-5880-4634-86c1-4d43a72ec34d" (UID: "b914fb03-5880-4634-86c1-4d43a72ec34d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:58.449381 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.449362    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b914fb03-5880-4634-86c1-4d43a72ec34d-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:58.450943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.450910    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b914fb03-5880-4634-86c1-4d43a72ec34d-maas-api-xks-ca" pod "b914fb03-5880-4634-86c1-4d43a72ec34d" (UID: "b914fb03-5880-4634-86c1-4d43a72ec34d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:29:58.451099 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.451074    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b914fb03-5880-4634-86c1-4d43a72ec34d-kube-api-access-lwwd5" pod "b914fb03-5880-4634-86c1-4d43a72ec34d" (UID: "b914fb03-5880-4634-86c1-4d43a72ec34d"). InnerVolumeSpecName "kube-api-access-lwwd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:58.550596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.550553    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b914fb03-5880-4634-86c1-4d43a72ec34d-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:58.550596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.550587    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwwd5\" (UniqueName: \"kubernetes.io/projected/b914fb03-5880-4634-86c1-4d43a72ec34d-kube-api-access-lwwd5\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:58.556763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.556734    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:58.651818 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.651715    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48619b64-2123-4936-bda6-ae202994ebb5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/48619b64-2123-4936-bda6-ae202994ebb5-provider-credentials\") pod \"48619b64-2123-4936-bda6-ae202994ebb5\" (UID: \"48619b64-2123-4936-bda6-ae202994ebb5\") "
Oct 30 01:29:58.652020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.651855    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48619b64-2123-4936-bda6-ae202994ebb5-kube-api-access-s5987\" (UniqueName: \"kubernetes.io/projected/48619b64-2123-4936-bda6-ae202994ebb5-kube-api-access-s5987\") pod \"48619b64-2123-4936-bda6-ae202994ebb5\" (UID: \"48619b64-2123-4936-bda6-ae202994ebb5\") "
Oct 30 01:29:58.652020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.651903    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48619b64-2123-4936-bda6-ae202994ebb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48619b64-2123-4936-bda6-ae202994ebb5-plugins-config-volume\") pod \"48619b64-2123-4936-bda6-ae202994ebb5\" (UID: \"48619b64-2123-4936-bda6-ae202994ebb5\") "
Oct 30 01:29:58.652020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.651965    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48619b64-2123-4936-bda6-ae202994ebb5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48619b64-2123-4936-bda6-ae202994ebb5-routing-overlay-volume\") pod \"48619b64-2123-4936-bda6-ae202994ebb5\" (UID: \"48619b64-2123-4936-bda6-ae202994ebb5\") "
Oct 30 01:29:58.652421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.652382    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48619b64-2123-4936-bda6-ae202994ebb5-routing-overlay-volume" pod "48619b64-2123-4936-bda6-ae202994ebb5" (UID: "48619b64-2123-4936-bda6-ae202994ebb5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:58.652421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.652402    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48619b64-2123-4936-bda6-ae202994ebb5-plugins-config-volume" pod "48619b64-2123-4936-bda6-ae202994ebb5" (UID: "48619b64-2123-4936-bda6-ae202994ebb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:29:58.652820 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.652791    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48619b64-2123-4936-bda6-ae202994ebb5-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:58.652916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.652825    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/48619b64-2123-4936-bda6-ae202994ebb5-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:58.654262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.654232    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48619b64-2123-4936-bda6-ae202994ebb5-provider-credentials" pod "48619b64-2123-4936-bda6-ae202994ebb5" (UID: "48619b64-2123-4936-bda6-ae202994ebb5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:58.654384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.654365    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48619b64-2123-4936-bda6-ae202994ebb5-kube-api-access-s5987" pod "48619b64-2123-4936-bda6-ae202994ebb5" (UID: "48619b64-2123-4936-bda6-ae202994ebb5"). InnerVolumeSpecName "kube-api-access-s5987". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:29:58.754014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.753966    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/48619b64-2123-4936-bda6-ae202994ebb5-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:58.754014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:58.754005    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5987\" (UniqueName: \"kubernetes.io/projected/48619b64-2123-4936-bda6-ae202994ebb5-kube-api-access-s5987\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:29:59.087247 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:59.087215    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"
Oct 30 01:29:59.087459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:59.087251    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8" event={"ID":"b914fb03-5880-4634-86c1-4d43a72ec34d","Type":"ContainerDied","Data":"8f77d82d0d397dbf49f4669d3fa19cbaffe2f19980a81e29b6edcf06ce0929bd"}
Oct 30 01:29:59.087459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:59.087315    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f77d82d0d397dbf49f4669d3fa19cbaffe2f19980a81e29b6edcf06ce0929bd"
Oct 30 01:29:59.088855 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:59.088832    4425 generic.go:356] "Generic (PLEG): container finished" podID="48619b64-2123-4936-bda6-ae202994ebb5" containerID="9a129f1af0ea4c8a4b77586b8f2a58a889bc9f005664889cdd70a629af4716f0" exitCode=0
Oct 30 01:29:59.089001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:59.088879    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"
Oct 30 01:29:59.089001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:59.088908    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc" event={"ID":"48619b64-2123-4936-bda6-ae202994ebb5","Type":"ContainerDied","Data":"9a129f1af0ea4c8a4b77586b8f2a58a889bc9f005664889cdd70a629af4716f0"}
Oct 30 01:29:59.089001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:59.088954    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc" event={"ID":"48619b64-2123-4936-bda6-ae202994ebb5","Type":"ContainerDied","Data":"9c6348d594cc5e40cd006819f149613a0db2de8c13f43511e3be7997ac3241b8"}
Oct 30 01:29:59.089001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:59.088977    4425 scope.go:122] "RemoveContainer" containerID="9a129f1af0ea4c8a4b77586b8f2a58a889bc9f005664889cdd70a629af4716f0"
Oct 30 01:29:59.102312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:59.102291    4425 scope.go:122] "RemoveContainer" containerID="9a129f1af0ea4c8a4b77586b8f2a58a889bc9f005664889cdd70a629af4716f0"
Oct 30 01:29:59.102654 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:29:59.102635    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a129f1af0ea4c8a4b77586b8f2a58a889bc9f005664889cdd70a629af4716f0\": container with ID starting with 9a129f1af0ea4c8a4b77586b8f2a58a889bc9f005664889cdd70a629af4716f0 not found: ID does not exist" containerID="9a129f1af0ea4c8a4b77586b8f2a58a889bc9f005664889cdd70a629af4716f0"
Oct 30 01:29:59.102701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:59.102665    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a129f1af0ea4c8a4b77586b8f2a58a889bc9f005664889cdd70a629af4716f0"} err="failed to get container status \"9a129f1af0ea4c8a4b77586b8f2a58a889bc9f005664889cdd70a629af4716f0\": rpc error: code = NotFound desc = could not find container \"9a129f1af0ea4c8a4b77586b8f2a58a889bc9f005664889cdd70a629af4716f0\": container with ID starting with 9a129f1af0ea4c8a4b77586b8f2a58a889bc9f005664889cdd70a629af4716f0 not found: ID does not exist"
Oct 30 01:29:59.139599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:59.139556    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"]
Oct 30 01:29:59.149814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:59.149625    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f6fbbf746-lrndc"]
Oct 30 01:29:59.433014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:29:59.432913    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48619b64-2123-4936-bda6-ae202994ebb5" path="/var/lib/kubelet/pods/48619b64-2123-4936-bda6-ae202994ebb5/volumes"
Oct 30 01:30:00.141736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.141696    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm"]
Oct 30 01:30:00.142717 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.142692    4425 cpu_manager.go:414] "containerMap: removing container" podUID="48619b64-2123-4936-bda6-ae202994ebb5" containerName="payload-processing"
Oct 30 01:30:00.142717 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.142717    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48619b64-2123-4936-bda6-ae202994ebb5" containerName="payload-processing"
Oct 30 01:30:00.142928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.142775    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b914fb03-5880-4634-86c1-4d43a72ec34d" containerName="revoke-keys"
Oct 30 01:30:00.142928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.142781    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b914fb03-5880-4634-86c1-4d43a72ec34d" containerName="revoke-keys"
Oct 30 01:30:00.142928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.142896    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b914fb03-5880-4634-86c1-4d43a72ec34d" containerName="revoke-keys"
Oct 30 01:30:00.142928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.142907    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="48619b64-2123-4936-bda6-ae202994ebb5" containerName="payload-processing"
Oct 30 01:30:00.148224 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.148202    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct"]
Oct 30 01:30:00.148361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.148337    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm"
Oct 30 01:30:00.153807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.153783    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7"]
Oct 30 01:30:00.153936 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.153909    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct"
Oct 30 01:30:00.159974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.159948    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9"]
Oct 30 01:30:00.160092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.160066    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7"
Oct 30 01:30:00.164510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.164485    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:30:00.166794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.166769    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9"]
Oct 30 01:30:00.166933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.166905    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9"
Oct 30 01:30:00.173442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.173358    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx"]
Oct 30 01:30:00.174034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.174014    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9"
Oct 30 01:30:00.178834 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.178809    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8"]
Oct 30 01:30:00.178977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.178948    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx"
Oct 30 01:30:00.196173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.192354    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n"]
Oct 30 01:30:00.200849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.200817    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8"
Oct 30 01:30:00.214206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.214034    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk"]
Oct 30 01:30:00.220944 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.215997    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n"
Oct 30 01:30:00.223345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.223317    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct"]
Oct 30 01:30:00.223479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.223365    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm"]
Oct 30 01:30:00.223479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.223390    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9"]
Oct 30 01:30:00.223625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.223480    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk"
Oct 30 01:30:00.224382 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.224347    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7"]
Oct 30 01:30:00.231253 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.230500    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9"]
Oct 30 01:30:00.237341 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.236071    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862"]
Oct 30 01:30:00.241764 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.241740    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb"]
Oct 30 01:30:00.257385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.257349    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862"
Oct 30 01:30:00.269133 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.269104    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n"]
Oct 30 01:30:00.269438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.269398    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx"]
Oct 30 01:30:00.269606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.269592    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8"]
Oct 30 01:30:00.269793 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.269777    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862"]
Oct 30 01:30:00.270006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.269536    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb"
Oct 30 01:30:00.272712 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.272684    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk"]
Oct 30 01:30:00.282578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.276723    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c"]
Oct 30 01:30:00.284166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.283371    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n24ff\" (UniqueName: \"kubernetes.io/projected/53190760-7148-47da-93f0-1123e5df62e2-kube-api-access-n24ff\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9\" (UID: \"53190760-7148-47da-93f0-1123e5df62e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9"
Oct 30 01:30:00.284166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.283426    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3cc419-c375-4b6b-a359-677d816e907b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7\" (UID: \"be3cc419-c375-4b6b-a359-677d816e907b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7"
Oct 30 01:30:00.284166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.283460    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wtxd\" (UniqueName: \"kubernetes.io/projected/03eaf7ac-cf67-4da9-b253-3c1994ad57fa-kube-api-access-5wtxd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm\" (UID: \"03eaf7ac-cf67-4da9-b253-3c1994ad57fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm"
Oct 30 01:30:00.284166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.283516    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58t9\" (UniqueName: \"kubernetes.io/projected/47e2b0e7-6204-402f-a79c-fa8ff22b3a70-kube-api-access-d58t9\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct\" (UID: \"47e2b0e7-6204-402f-a79c-fa8ff22b3a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct"
Oct 30 01:30:00.284166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.283558    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03eaf7ac-cf67-4da9-b253-3c1994ad57fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm\" (UID: \"03eaf7ac-cf67-4da9-b253-3c1994ad57fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm"
Oct 30 01:30:00.284166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.283594    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e2b0e7-6204-402f-a79c-fa8ff22b3a70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct\" (UID: \"47e2b0e7-6204-402f-a79c-fa8ff22b3a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct"
Oct 30 01:30:00.284166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.283654    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c284f618-116f-426d-b25d-fde4ca3c686e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9\" (UID: \"c284f618-116f-426d-b25d-fde4ca3c686e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9"
Oct 30 01:30:00.284166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.283694    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c894e6-ae30-4978-850a-dfbde9fd69bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8\" (UID: \"61c894e6-ae30-4978-850a-dfbde9fd69bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8"
Oct 30 01:30:00.284166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.283719    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53190760-7148-47da-93f0-1123e5df62e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9\" (UID: \"53190760-7148-47da-93f0-1123e5df62e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9"
Oct 30 01:30:00.284166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.283794    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8gv\" (UniqueName: \"kubernetes.io/projected/be3cc419-c375-4b6b-a359-677d816e907b-kube-api-access-ws8gv\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7\" (UID: \"be3cc419-c375-4b6b-a359-677d816e907b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7"
Oct 30 01:30:00.284166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.283851    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lfgs\" (UniqueName: \"kubernetes.io/projected/fd93d36a-1693-4cbb-bf97-204d5c295c0d-kube-api-access-8lfgs\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx\" (UID: \"fd93d36a-1693-4cbb-bf97-204d5c295c0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx"
Oct 30 01:30:00.284166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.283927    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx89m\" (UniqueName: \"kubernetes.io/projected/61c894e6-ae30-4978-850a-dfbde9fd69bd-kube-api-access-tx89m\") pod \"maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8\" (UID: \"61c894e6-ae30-4978-850a-dfbde9fd69bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8"
Oct 30 01:30:00.284166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.283953    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd93d36a-1693-4cbb-bf97-204d5c295c0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx\" (UID: \"fd93d36a-1693-4cbb-bf97-204d5c295c0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx"
Oct 30 01:30:00.284166 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.283978    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl7wb\" (UniqueName: \"kubernetes.io/projected/c284f618-116f-426d-b25d-fde4ca3c686e-kube-api-access-sl7wb\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9\" (UID: \"c284f618-116f-426d-b25d-fde4ca3c686e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9"
Oct 30 01:30:00.297179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.294383    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c"
Oct 30 01:30:00.297179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.294938    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt"]
Oct 30 01:30:00.312631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.311962    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb"]
Oct 30 01:30:00.312631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.312083    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt"
Oct 30 01:30:00.314629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.314602    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c"]
Oct 30 01:30:00.329725 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.326635    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd"]
Oct 30 01:30:00.341556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.341528    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd"]
Oct 30 01:30:00.341698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.341689    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd"
Oct 30 01:30:00.353436 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.353399    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt"]
Oct 30 01:30:00.385538 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.385506    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lfgs\" (UniqueName: \"kubernetes.io/projected/fd93d36a-1693-4cbb-bf97-204d5c295c0d-kube-api-access-8lfgs\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx\" (UID: \"fd93d36a-1693-4cbb-bf97-204d5c295c0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx"
Oct 30 01:30:00.385676 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.385572    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bxz7\" (UniqueName: \"kubernetes.io/projected/b3d49b88-2ba6-4ded-a160-806e4e8fed2b-kube-api-access-2bxz7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n\" (UID: \"b3d49b88-2ba6-4ded-a160-806e4e8fed2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n"
Oct 30 01:30:00.385676 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.385658    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx89m\" (UniqueName: \"kubernetes.io/projected/61c894e6-ae30-4978-850a-dfbde9fd69bd-kube-api-access-tx89m\") pod \"maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8\" (UID: \"61c894e6-ae30-4978-850a-dfbde9fd69bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8"
Oct 30 01:30:00.385755 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.385688    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqcgt\" (UniqueName: \"kubernetes.io/projected/dd86e619-32bb-4d11-afa2-d4a9d723d886-kube-api-access-vqcgt\") pod \"maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk\" (UID: \"dd86e619-32bb-4d11-afa2-d4a9d723d886\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk"
Oct 30 01:30:00.385755 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.385723    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd93d36a-1693-4cbb-bf97-204d5c295c0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx\" (UID: \"fd93d36a-1693-4cbb-bf97-204d5c295c0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx"
Oct 30 01:30:00.385834 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.385755    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcp9m\" (UniqueName: \"kubernetes.io/projected/4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1-kube-api-access-xcp9m\") pod \"maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb\" (UID: \"4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb"
Oct 30 01:30:00.385935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.385917    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl7wb\" (UniqueName: \"kubernetes.io/projected/c284f618-116f-426d-b25d-fde4ca3c686e-kube-api-access-sl7wb\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9\" (UID: \"c284f618-116f-426d-b25d-fde4ca3c686e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9"
Oct 30 01:30:00.385989 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.385956    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n24ff\" (UniqueName: \"kubernetes.io/projected/53190760-7148-47da-93f0-1123e5df62e2-kube-api-access-n24ff\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9\" (UID: \"53190760-7148-47da-93f0-1123e5df62e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9"
Oct 30 01:30:00.386028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.385997    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3cc419-c375-4b6b-a359-677d816e907b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7\" (UID: \"be3cc419-c375-4b6b-a359-677d816e907b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7"
Oct 30 01:30:00.386062 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.386024    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230424d1-69ed-40c3-8afd-660d905f9d8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862\" (UID: \"230424d1-69ed-40c3-8afd-660d905f9d8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862"
Oct 30 01:30:00.386107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.386062    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wtxd\" (UniqueName: \"kubernetes.io/projected/03eaf7ac-cf67-4da9-b253-3c1994ad57fa-kube-api-access-5wtxd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm\" (UID: \"03eaf7ac-cf67-4da9-b253-3c1994ad57fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm"
Oct 30 01:30:00.386147 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.386114    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d49b88-2ba6-4ded-a160-806e4e8fed2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n\" (UID: \"b3d49b88-2ba6-4ded-a160-806e4e8fed2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n"
Oct 30 01:30:00.386220 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.386168    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58t9\" (UniqueName: \"kubernetes.io/projected/47e2b0e7-6204-402f-a79c-fa8ff22b3a70-kube-api-access-d58t9\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct\" (UID: \"47e2b0e7-6204-402f-a79c-fa8ff22b3a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct"
Oct 30 01:30:00.386516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.386490    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03eaf7ac-cf67-4da9-b253-3c1994ad57fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm\" (UID: \"03eaf7ac-cf67-4da9-b253-3c1994ad57fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm"
Oct 30 01:30:00.386585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.386547    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qddg4\" (UniqueName: \"kubernetes.io/projected/230424d1-69ed-40c3-8afd-660d905f9d8d-kube-api-access-qddg4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862\" (UID: \"230424d1-69ed-40c3-8afd-660d905f9d8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862"
Oct 30 01:30:00.386621 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.386572    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd93d36a-1693-4cbb-bf97-204d5c295c0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx\" (UID: \"fd93d36a-1693-4cbb-bf97-204d5c295c0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx"
Oct 30 01:30:00.386621 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.386581    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e2b0e7-6204-402f-a79c-fa8ff22b3a70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct\" (UID: \"47e2b0e7-6204-402f-a79c-fa8ff22b3a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct"
Oct 30 01:30:00.386693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.386658    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pbcv\" (UniqueName: \"kubernetes.io/projected/fba425f6-60cc-44b1-89da-533a37750a41-kube-api-access-4pbcv\") pod \"maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c\" (UID: \"fba425f6-60cc-44b1-89da-533a37750a41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c"
Oct 30 01:30:00.386737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.386703    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd86e619-32bb-4d11-afa2-d4a9d723d886-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk\" (UID: \"dd86e619-32bb-4d11-afa2-d4a9d723d886\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk"
Oct 30 01:30:00.386782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.386748    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c284f618-116f-426d-b25d-fde4ca3c686e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9\" (UID: \"c284f618-116f-426d-b25d-fde4ca3c686e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9"
Oct 30 01:30:00.386824 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.386792    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c894e6-ae30-4978-850a-dfbde9fd69bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8\" (UID: \"61c894e6-ae30-4978-850a-dfbde9fd69bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8"
Oct 30 01:30:00.386861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.386820    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53190760-7148-47da-93f0-1123e5df62e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9\" (UID: \"53190760-7148-47da-93f0-1123e5df62e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9"
Oct 30 01:30:00.386917 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.386901    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb\" (UID: \"4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb"
Oct 30 01:30:00.386965 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.386943    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8gv\" (UniqueName: \"kubernetes.io/projected/be3cc419-c375-4b6b-a359-677d816e907b-kube-api-access-ws8gv\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7\" (UID: \"be3cc419-c375-4b6b-a359-677d816e907b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7"
Oct 30 01:30:00.387003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.386990    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fba425f6-60cc-44b1-89da-533a37750a41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c\" (UID: \"fba425f6-60cc-44b1-89da-533a37750a41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c"
Oct 30 01:30:00.387187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.387144    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03eaf7ac-cf67-4da9-b253-3c1994ad57fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm\" (UID: \"03eaf7ac-cf67-4da9-b253-3c1994ad57fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm"
Oct 30 01:30:00.387348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.387263    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e2b0e7-6204-402f-a79c-fa8ff22b3a70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct\" (UID: \"47e2b0e7-6204-402f-a79c-fa8ff22b3a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct"
Oct 30 01:30:00.387429 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.386904    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3cc419-c375-4b6b-a359-677d816e907b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7\" (UID: \"be3cc419-c375-4b6b-a359-677d816e907b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7"
Oct 30 01:30:00.387606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.387588    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c284f618-116f-426d-b25d-fde4ca3c686e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9\" (UID: \"c284f618-116f-426d-b25d-fde4ca3c686e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9"
Oct 30 01:30:00.387866 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.387845    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c894e6-ae30-4978-850a-dfbde9fd69bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8\" (UID: \"61c894e6-ae30-4978-850a-dfbde9fd69bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8"
Oct 30 01:30:00.387961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.387941    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53190760-7148-47da-93f0-1123e5df62e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9\" (UID: \"53190760-7148-47da-93f0-1123e5df62e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9"
Oct 30 01:30:00.409392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.409344    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lfgs\" (UniqueName: \"kubernetes.io/projected/fd93d36a-1693-4cbb-bf97-204d5c295c0d-kube-api-access-8lfgs\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx\" (UID: \"fd93d36a-1693-4cbb-bf97-204d5c295c0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx"
Oct 30 01:30:00.410090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.410064    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wtxd\" (UniqueName: \"kubernetes.io/projected/03eaf7ac-cf67-4da9-b253-3c1994ad57fa-kube-api-access-5wtxd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm\" (UID: \"03eaf7ac-cf67-4da9-b253-3c1994ad57fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm"
Oct 30 01:30:00.410957 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.410934    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n24ff\" (UniqueName: \"kubernetes.io/projected/53190760-7148-47da-93f0-1123e5df62e2-kube-api-access-n24ff\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9\" (UID: \"53190760-7148-47da-93f0-1123e5df62e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9"
Oct 30 01:30:00.411047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.410979    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx89m\" (UniqueName: \"kubernetes.io/projected/61c894e6-ae30-4978-850a-dfbde9fd69bd-kube-api-access-tx89m\") pod \"maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8\" (UID: \"61c894e6-ae30-4978-850a-dfbde9fd69bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8"
Oct 30 01:30:00.411292 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.411270    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8gv\" (UniqueName: \"kubernetes.io/projected/be3cc419-c375-4b6b-a359-677d816e907b-kube-api-access-ws8gv\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7\" (UID: \"be3cc419-c375-4b6b-a359-677d816e907b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7"
Oct 30 01:30:00.411514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.411493    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58t9\" (UniqueName: \"kubernetes.io/projected/47e2b0e7-6204-402f-a79c-fa8ff22b3a70-kube-api-access-d58t9\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct\" (UID: \"47e2b0e7-6204-402f-a79c-fa8ff22b3a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct"
Oct 30 01:30:00.412078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.412062    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl7wb\" (UniqueName: \"kubernetes.io/projected/c284f618-116f-426d-b25d-fde4ca3c686e-kube-api-access-sl7wb\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9\" (UID: \"c284f618-116f-426d-b25d-fde4ca3c686e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9"
Oct 30 01:30:00.478365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.478329    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm"
Oct 30 01:30:00.489853 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.489821    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbcv\" (UniqueName: \"kubernetes.io/projected/fba425f6-60cc-44b1-89da-533a37750a41-kube-api-access-4pbcv\") pod \"maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c\" (UID: \"fba425f6-60cc-44b1-89da-533a37750a41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c"
Oct 30 01:30:00.490024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.489866    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd86e619-32bb-4d11-afa2-d4a9d723d886-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk\" (UID: \"dd86e619-32bb-4d11-afa2-d4a9d723d886\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk"
Oct 30 01:30:00.490024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.489948    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb\" (UID: \"4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb"
Oct 30 01:30:00.490024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.489995    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgrl8\" (UniqueName: \"kubernetes.io/projected/3699d466-411f-4f43-b62f-e6955340d1da-kube-api-access-mgrl8\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd\" (UID: \"3699d466-411f-4f43-b62f-e6955340d1da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd"
Oct 30 01:30:00.490348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.490323    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct"
Oct 30 01:30:00.490671 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.490648    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb\" (UID: \"4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb"
Oct 30 01:30:00.490743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.490700    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fba425f6-60cc-44b1-89da-533a37750a41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c\" (UID: \"fba425f6-60cc-44b1-89da-533a37750a41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c"
Oct 30 01:30:00.490743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.490721    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd86e619-32bb-4d11-afa2-d4a9d723d886-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk\" (UID: \"dd86e619-32bb-4d11-afa2-d4a9d723d886\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk"
Oct 30 01:30:00.490825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.490760    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xpth\" (UniqueName: \"kubernetes.io/projected/107404d2-96de-452e-8e7d-24acb8246db3-kube-api-access-2xpth\") pod \"maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt\" (UID: \"107404d2-96de-452e-8e7d-24acb8246db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt"
Oct 30 01:30:00.490825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.490794    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxz7\" (UniqueName: \"kubernetes.io/projected/b3d49b88-2ba6-4ded-a160-806e4e8fed2b-kube-api-access-2bxz7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n\" (UID: \"b3d49b88-2ba6-4ded-a160-806e4e8fed2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n"
Oct 30 01:30:00.490907 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.490857    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fba425f6-60cc-44b1-89da-533a37750a41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c\" (UID: \"fba425f6-60cc-44b1-89da-533a37750a41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c"
Oct 30 01:30:00.490907 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.490888    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/107404d2-96de-452e-8e7d-24acb8246db3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt\" (UID: \"107404d2-96de-452e-8e7d-24acb8246db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt"
Oct 30 01:30:00.491002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.490925    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqcgt\" (UniqueName: \"kubernetes.io/projected/dd86e619-32bb-4d11-afa2-d4a9d723d886-kube-api-access-vqcgt\") pod \"maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk\" (UID: \"dd86e619-32bb-4d11-afa2-d4a9d723d886\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk"
Oct 30 01:30:00.491002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.490950    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3699d466-411f-4f43-b62f-e6955340d1da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd\" (UID: \"3699d466-411f-4f43-b62f-e6955340d1da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd"
Oct 30 01:30:00.491079 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.491030    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcp9m\" (UniqueName: \"kubernetes.io/projected/4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1-kube-api-access-xcp9m\") pod \"maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb\" (UID: \"4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb"
Oct 30 01:30:00.491137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.491075    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230424d1-69ed-40c3-8afd-660d905f9d8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862\" (UID: \"230424d1-69ed-40c3-8afd-660d905f9d8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862"
Oct 30 01:30:00.491234 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.491146    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d49b88-2ba6-4ded-a160-806e4e8fed2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n\" (UID: \"b3d49b88-2ba6-4ded-a160-806e4e8fed2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n"
Oct 30 01:30:00.491302 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.491243    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qddg4\" (UniqueName: \"kubernetes.io/projected/230424d1-69ed-40c3-8afd-660d905f9d8d-kube-api-access-qddg4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862\" (UID: \"230424d1-69ed-40c3-8afd-660d905f9d8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862"
Oct 30 01:30:00.491661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.491641    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230424d1-69ed-40c3-8afd-660d905f9d8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862\" (UID: \"230424d1-69ed-40c3-8afd-660d905f9d8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862"
Oct 30 01:30:00.491746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.491720    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d49b88-2ba6-4ded-a160-806e4e8fed2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n\" (UID: \"b3d49b88-2ba6-4ded-a160-806e4e8fed2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n"
Oct 30 01:30:00.504265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.504237    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7"
Oct 30 01:30:00.514022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.513990    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pbcv\" (UniqueName: \"kubernetes.io/projected/fba425f6-60cc-44b1-89da-533a37750a41-kube-api-access-4pbcv\") pod \"maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c\" (UID: \"fba425f6-60cc-44b1-89da-533a37750a41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c"
Oct 30 01:30:00.515066 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.514961    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qddg4\" (UniqueName: \"kubernetes.io/projected/230424d1-69ed-40c3-8afd-660d905f9d8d-kube-api-access-qddg4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862\" (UID: \"230424d1-69ed-40c3-8afd-660d905f9d8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862"
Oct 30 01:30:00.516252 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.515254    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqcgt\" (UniqueName: \"kubernetes.io/projected/dd86e619-32bb-4d11-afa2-d4a9d723d886-kube-api-access-vqcgt\") pod \"maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk\" (UID: \"dd86e619-32bb-4d11-afa2-d4a9d723d886\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk"
Oct 30 01:30:00.516252 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.515633    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bxz7\" (UniqueName: \"kubernetes.io/projected/b3d49b88-2ba6-4ded-a160-806e4e8fed2b-kube-api-access-2bxz7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n\" (UID: \"b3d49b88-2ba6-4ded-a160-806e4e8fed2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n"
Oct 30 01:30:00.516694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.516596    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcp9m\" (UniqueName: \"kubernetes.io/projected/4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1-kube-api-access-xcp9m\") pod \"maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb\" (UID: \"4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb"
Oct 30 01:30:00.529646 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.529451    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9"
Oct 30 01:30:00.578047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.575419    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9"
Oct 30 01:30:00.600488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.591440    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx"
Oct 30 01:30:00.600488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.593638    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgrl8\" (UniqueName: \"kubernetes.io/projected/3699d466-411f-4f43-b62f-e6955340d1da-kube-api-access-mgrl8\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd\" (UID: \"3699d466-411f-4f43-b62f-e6955340d1da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd"
Oct 30 01:30:00.600488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.593723    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xpth\" (UniqueName: \"kubernetes.io/projected/107404d2-96de-452e-8e7d-24acb8246db3-kube-api-access-2xpth\") pod \"maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt\" (UID: \"107404d2-96de-452e-8e7d-24acb8246db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt"
Oct 30 01:30:00.600488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.593806    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/107404d2-96de-452e-8e7d-24acb8246db3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt\" (UID: \"107404d2-96de-452e-8e7d-24acb8246db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt"
Oct 30 01:30:00.600488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.593851    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3699d466-411f-4f43-b62f-e6955340d1da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd\" (UID: \"3699d466-411f-4f43-b62f-e6955340d1da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd"
Oct 30 01:30:00.600488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.596182    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/107404d2-96de-452e-8e7d-24acb8246db3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt\" (UID: \"107404d2-96de-452e-8e7d-24acb8246db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt"
Oct 30 01:30:00.600488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.597085    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3699d466-411f-4f43-b62f-e6955340d1da-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd\" (UID: \"3699d466-411f-4f43-b62f-e6955340d1da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd"
Oct 30 01:30:00.636302 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.634733    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8"
Oct 30 01:30:00.646388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.645332    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgrl8\" (UniqueName: \"kubernetes.io/projected/3699d466-411f-4f43-b62f-e6955340d1da-kube-api-access-mgrl8\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd\" (UID: \"3699d466-411f-4f43-b62f-e6955340d1da\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd"
Oct 30 01:30:00.648021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.647992    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xpth\" (UniqueName: \"kubernetes.io/projected/107404d2-96de-452e-8e7d-24acb8246db3-kube-api-access-2xpth\") pod \"maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt\" (UID: \"107404d2-96de-452e-8e7d-24acb8246db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt"
Oct 30 01:30:00.651576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.651551    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n"
Oct 30 01:30:00.670084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.670056    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk"
Oct 30 01:30:00.682005 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.681971    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862"
Oct 30 01:30:00.693783 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.693750    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb"
Oct 30 01:30:00.702553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.702522    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c"
Oct 30 01:30:00.710704 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.710601    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt"
Oct 30 01:30:00.749904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:00.749619    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd"
Oct 30 01:30:01.024039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.024011    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm"]
Oct 30 01:30:01.036349 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.036321    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7"]
Oct 30 01:30:01.042169 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:01.039619    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3cc419_c375_4b6b_a359_677d816e907b.slice/crio-124ad287086f77da2a9d9cfc7548806d8b1bd09a6ee6cefbd0177bbfb55f39d2 WatchSource:0}: Error finding container 124ad287086f77da2a9d9cfc7548806d8b1bd09a6ee6cefbd0177bbfb55f39d2: Status 404 returned error can't find the container with id 124ad287086f77da2a9d9cfc7548806d8b1bd09a6ee6cefbd0177bbfb55f39d2
Oct 30 01:30:01.047958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.047928    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct"]
Oct 30 01:30:01.053051 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:01.053011    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e2b0e7_6204_402f_a79c_fa8ff22b3a70.slice/crio-7c352f094ccc676f315786b3f5fb51d73eefaf02d0e75846f742304c62f51e8e WatchSource:0}: Error finding container 7c352f094ccc676f315786b3f5fb51d73eefaf02d0e75846f742304c62f51e8e: Status 404 returned error can't find the container with id 7c352f094ccc676f315786b3f5fb51d73eefaf02d0e75846f742304c62f51e8e
Oct 30 01:30:01.087508 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.084935    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9"]
Oct 30 01:30:01.089767 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:01.089724    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53190760_7148_47da_93f0_1123e5df62e2.slice/crio-bda6af1d42d0155e1a224b240e554f074853d5e7021503b52e7bf6b1622a170e WatchSource:0}: Error finding container bda6af1d42d0155e1a224b240e554f074853d5e7021503b52e7bf6b1622a170e: Status 404 returned error can't find the container with id bda6af1d42d0155e1a224b240e554f074853d5e7021503b52e7bf6b1622a170e
Oct 30 01:30:01.114331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.114284    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9" event={"ID":"53190760-7148-47da-93f0-1123e5df62e2","Type":"ContainerStarted","Data":"bda6af1d42d0155e1a224b240e554f074853d5e7021503b52e7bf6b1622a170e"}
Oct 30 01:30:01.121144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.121108    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm" event={"ID":"03eaf7ac-cf67-4da9-b253-3c1994ad57fa","Type":"ContainerStarted","Data":"fd8b9adc3c38e185be110395601a273cfe6acb8ee2bec9d1aac9278e63687da8"}
Oct 30 01:30:01.127928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.127839    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7" event={"ID":"be3cc419-c375-4b6b-a359-677d816e907b","Type":"ContainerStarted","Data":"124ad287086f77da2a9d9cfc7548806d8b1bd09a6ee6cefbd0177bbfb55f39d2"}
Oct 30 01:30:01.134845 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.134819    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct" event={"ID":"47e2b0e7-6204-402f-a79c-fa8ff22b3a70","Type":"ContainerStarted","Data":"7c352f094ccc676f315786b3f5fb51d73eefaf02d0e75846f742304c62f51e8e"}
Oct 30 01:30:01.151935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.151870    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9"]
Oct 30 01:30:01.152558 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:01.152532    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc284f618_116f_426d_b25d_fde4ca3c686e.slice/crio-8b066266decc0607bbe3947eac0e0ca28358d4185b657d15fcd9289b61eca834 WatchSource:0}: Error finding container 8b066266decc0607bbe3947eac0e0ca28358d4185b657d15fcd9289b61eca834: Status 404 returned error can't find the container with id 8b066266decc0607bbe3947eac0e0ca28358d4185b657d15fcd9289b61eca834
Oct 30 01:30:01.163701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.163675    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx"]
Oct 30 01:30:01.201031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.201003    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8"]
Oct 30 01:30:01.233385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.233356    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n"]
Oct 30 01:30:01.237662 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.237634    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862"]
Oct 30 01:30:01.256792 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:01.256756    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd93d36a_1693_4cbb_bf97_204d5c295c0d.slice/crio-a111760e50e13129162662d840307d4de3d1e2faf96a862e8ab1ca0ccfb9ec41 WatchSource:0}: Error finding container a111760e50e13129162662d840307d4de3d1e2faf96a862e8ab1ca0ccfb9ec41: Status 404 returned error can't find the container with id a111760e50e13129162662d840307d4de3d1e2faf96a862e8ab1ca0ccfb9ec41
Oct 30 01:30:01.258816 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:01.258795    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c894e6_ae30_4978_850a_dfbde9fd69bd.slice/crio-045d1d8268ec42d9d36ad9e6cb92fe2f1722aad6e3877b28447a64c7ca72a2b7 WatchSource:0}: Error finding container 045d1d8268ec42d9d36ad9e6cb92fe2f1722aad6e3877b28447a64c7ca72a2b7: Status 404 returned error can't find the container with id 045d1d8268ec42d9d36ad9e6cb92fe2f1722aad6e3877b28447a64c7ca72a2b7
Oct 30 01:30:01.265919 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:01.265886    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d49b88_2ba6_4ded_a160_806e4e8fed2b.slice/crio-9e5a8683b1b3b4865614144bcee0dd0d91efa531a2741ede2cdfff2911b4504e WatchSource:0}: Error finding container 9e5a8683b1b3b4865614144bcee0dd0d91efa531a2741ede2cdfff2911b4504e: Status 404 returned error can't find the container with id 9e5a8683b1b3b4865614144bcee0dd0d91efa531a2741ede2cdfff2911b4504e
Oct 30 01:30:01.266292 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:01.266274    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230424d1_69ed_40c3_8afd_660d905f9d8d.slice/crio-02bbc638119701c6f0a6ca9cfe391bdc26f37bce6dbaef579a28ea311a1e15ce WatchSource:0}: Error finding container 02bbc638119701c6f0a6ca9cfe391bdc26f37bce6dbaef579a28ea311a1e15ce: Status 404 returned error can't find the container with id 02bbc638119701c6f0a6ca9cfe391bdc26f37bce6dbaef579a28ea311a1e15ce
Oct 30 01:30:01.315976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.315947    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk"]
Oct 30 01:30:01.323343 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:01.323308    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd86e619_32bb_4d11_afa2_d4a9d723d886.slice/crio-5df696c4968768ecd40df569455d149988d558e7134a12650c46752ebca1ed8d WatchSource:0}: Error finding container 5df696c4968768ecd40df569455d149988d558e7134a12650c46752ebca1ed8d: Status 404 returned error can't find the container with id 5df696c4968768ecd40df569455d149988d558e7134a12650c46752ebca1ed8d
Oct 30 01:30:01.336411 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.336386    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb"]
Oct 30 01:30:01.339272 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:01.339248    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ded5d10_ee2b_4ef7_a4a7_9aceb229a7c1.slice/crio-a4600020613b0fb72d208a4eb43d9899da3dcd0e9ee3c23254553c80a0acccf0 WatchSource:0}: Error finding container a4600020613b0fb72d208a4eb43d9899da3dcd0e9ee3c23254553c80a0acccf0: Status 404 returned error can't find the container with id a4600020613b0fb72d208a4eb43d9899da3dcd0e9ee3c23254553c80a0acccf0
Oct 30 01:30:01.355575 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.354778    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt"]
Oct 30 01:30:01.370728 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:01.370619    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3699d466_411f_4f43_b62f_e6955340d1da.slice/crio-9bd9c6311ae46a68c5d267c009aa7439830d13e81d441bd13453332753ece554 WatchSource:0}: Error finding container 9bd9c6311ae46a68c5d267c009aa7439830d13e81d441bd13453332753ece554: Status 404 returned error can't find the container with id 9bd9c6311ae46a68c5d267c009aa7439830d13e81d441bd13453332753ece554
Oct 30 01:30:01.372658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.372639    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd"]
Oct 30 01:30:01.377840 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:01.377813    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c"]
Oct 30 01:30:02.148042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.148001    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862" event={"ID":"230424d1-69ed-40c3-8afd-660d905f9d8d","Type":"ContainerStarted","Data":"02bbc638119701c6f0a6ca9cfe391bdc26f37bce6dbaef579a28ea311a1e15ce"}
Oct 30 01:30:02.159494 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.159451    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9" event={"ID":"c284f618-116f-426d-b25d-fde4ca3c686e","Type":"ContainerStarted","Data":"0e8a44a10a039670a8453dfaf18596e2178d363bb65d5c4ec9b4eec4c527f08b"}
Oct 30 01:30:02.159662 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.159523    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9" event={"ID":"c284f618-116f-426d-b25d-fde4ca3c686e","Type":"ContainerStarted","Data":"8b066266decc0607bbe3947eac0e0ca28358d4185b657d15fcd9289b61eca834"}
Oct 30 01:30:02.163698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.163663    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9" event={"ID":"53190760-7148-47da-93f0-1123e5df62e2","Type":"ContainerStarted","Data":"54c058322e1573dfaebd559f42c641084f10d53fd95061bf796e5bef1da16ea4"}
Oct 30 01:30:02.165460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.165429    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd" event={"ID":"3699d466-411f-4f43-b62f-e6955340d1da","Type":"ContainerStarted","Data":"9bd9c6311ae46a68c5d267c009aa7439830d13e81d441bd13453332753ece554"}
Oct 30 01:30:02.167387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.167363    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt" event={"ID":"107404d2-96de-452e-8e7d-24acb8246db3","Type":"ContainerStarted","Data":"f35a73d9bd22aab6a79e4acda2b125ba2cef6d485e834877febee5246fd87bc6"}
Oct 30 01:30:02.167516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.167405    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt" event={"ID":"107404d2-96de-452e-8e7d-24acb8246db3","Type":"ContainerStarted","Data":"0e049e1334171af215eecf4c6a5656a627a30a6b2b90ccb450767605faee7b75"}
Oct 30 01:30:02.169323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.169300    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx" event={"ID":"fd93d36a-1693-4cbb-bf97-204d5c295c0d","Type":"ContainerStarted","Data":"4c304c43ff39eea3b41a2b59c225f216514c313c6ab1f6c805606cd012c54f22"}
Oct 30 01:30:02.169453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.169338    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx" event={"ID":"fd93d36a-1693-4cbb-bf97-204d5c295c0d","Type":"ContainerStarted","Data":"a111760e50e13129162662d840307d4de3d1e2faf96a862e8ab1ca0ccfb9ec41"}
Oct 30 01:30:02.170883 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.170863    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n" event={"ID":"b3d49b88-2ba6-4ded-a160-806e4e8fed2b","Type":"ContainerStarted","Data":"5ee9c071cc404b721d9b675a49b54739fd1a012abaea52478efe1445123d6e6d"}
Oct 30 01:30:02.170987 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.170895    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n" event={"ID":"b3d49b88-2ba6-4ded-a160-806e4e8fed2b","Type":"ContainerStarted","Data":"9e5a8683b1b3b4865614144bcee0dd0d91efa531a2741ede2cdfff2911b4504e"}
Oct 30 01:30:02.172516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.172490    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm" event={"ID":"03eaf7ac-cf67-4da9-b253-3c1994ad57fa","Type":"ContainerStarted","Data":"1a63857779a7abfd00eb02701a89390e4b61586c46da927045959662877babd1"}
Oct 30 01:30:02.174143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.174119    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb" event={"ID":"4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1","Type":"ContainerStarted","Data":"3ee74e65f0f108a988d74978d278039811d2a3db79bc54a36e5269d569e9351a"}
Oct 30 01:30:02.174284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.174169    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb" event={"ID":"4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1","Type":"ContainerStarted","Data":"a4600020613b0fb72d208a4eb43d9899da3dcd0e9ee3c23254553c80a0acccf0"}
Oct 30 01:30:02.175788 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.175759    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7" event={"ID":"be3cc419-c375-4b6b-a359-677d816e907b","Type":"ContainerStarted","Data":"38dcd62daf20f8e2271e0bade04bfc38712068d12054e302f5e2de91d0be41c1"}
Oct 30 01:30:02.177701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.177682    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct" event={"ID":"47e2b0e7-6204-402f-a79c-fa8ff22b3a70","Type":"ContainerStarted","Data":"f99311c09328992ab751b044dab0bafb8c612a244cad193d9b9864f4dfe287ae"}
Oct 30 01:30:02.179117 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.179097    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c" event={"ID":"fba425f6-60cc-44b1-89da-533a37750a41","Type":"ContainerStarted","Data":"e6ad8c6a7e2ceeea889f7b52750515da740a5db520720a14bcea37b2192f69c1"}
Oct 30 01:30:02.181210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.181181    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8" event={"ID":"61c894e6-ae30-4978-850a-dfbde9fd69bd","Type":"ContainerStarted","Data":"7e191a74e56529c9ad260fee468d242f108c58f435b58071484f3d6519d133f5"}
Oct 30 01:30:02.181350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.181225    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8" event={"ID":"61c894e6-ae30-4978-850a-dfbde9fd69bd","Type":"ContainerStarted","Data":"045d1d8268ec42d9d36ad9e6cb92fe2f1722aad6e3877b28447a64c7ca72a2b7"}
Oct 30 01:30:02.182703 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.182684    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk" event={"ID":"dd86e619-32bb-4d11-afa2-d4a9d723d886","Type":"ContainerStarted","Data":"f4ed0d34a731bee7c7b2ff048ea9407a8312f4cd27fe0965754ba0496c9f1386"}
Oct 30 01:30:02.182780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.182716    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk" event={"ID":"dd86e619-32bb-4d11-afa2-d4a9d723d886","Type":"ContainerStarted","Data":"5df696c4968768ecd40df569455d149988d558e7134a12650c46752ebca1ed8d"}
Oct 30 01:30:02.198595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.198539    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9" podStartSLOduration=2.198524258 podStartE2EDuration="2.198524258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.19509453 +0000 UTC m=+2910.944749841" watchObservedRunningTime="2026-10-30 01:30:02.198524258 +0000 UTC m=+2910.948179568"
Oct 30 01:30:02.232466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.232310    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8" podStartSLOduration=2.232270405 podStartE2EDuration="2.232270405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.229422974 +0000 UTC m=+2910.979078282" watchObservedRunningTime="2026-10-30 01:30:02.232270405 +0000 UTC m=+2910.981925714"
Oct 30 01:30:02.268179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.268118    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb" podStartSLOduration=2.268104741 podStartE2EDuration="2.268104741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.267483123 +0000 UTC m=+2911.017138436" watchObservedRunningTime="2026-10-30 01:30:02.268104741 +0000 UTC m=+2911.017760050"
Oct 30 01:30:02.301930 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.301882    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt" podStartSLOduration=2.301868688 podStartE2EDuration="2.301868688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.301719327 +0000 UTC m=+2911.051374636" watchObservedRunningTime="2026-10-30 01:30:02.301868688 +0000 UTC m=+2911.051523996"
Oct 30 01:30:02.337574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.337523    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk" podStartSLOduration=2.337509453 podStartE2EDuration="2.337509453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.336607491 +0000 UTC m=+2911.086262814" watchObservedRunningTime="2026-10-30 01:30:02.337509453 +0000 UTC m=+2911.087164763"
Oct 30 01:30:02.419013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.418958    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9" podStartSLOduration=2.418943623 podStartE2EDuration="2.418943623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.418229905 +0000 UTC m=+2911.167885225" watchObservedRunningTime="2026-10-30 01:30:02.418943623 +0000 UTC m=+2911.168598931"
Oct 30 01:30:02.421564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.421518    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm" podStartSLOduration=2.421506342 podStartE2EDuration="2.421506342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.380836521 +0000 UTC m=+2911.130491832" watchObservedRunningTime="2026-10-30 01:30:02.421506342 +0000 UTC m=+2911.171161651"
Oct 30 01:30:02.455706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.455634    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx" podStartSLOduration=2.455616285 podStartE2EDuration="2.455616285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.454683445 +0000 UTC m=+2911.204338755" watchObservedRunningTime="2026-10-30 01:30:02.455616285 +0000 UTC m=+2911.205271596"
Oct 30 01:30:02.494061 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.494012    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n" podStartSLOduration=2.493997612 podStartE2EDuration="2.493997612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.489746687 +0000 UTC m=+2911.239402010" watchObservedRunningTime="2026-10-30 01:30:02.493997612 +0000 UTC m=+2911.243652920"
Oct 30 01:30:02.527476 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.527406    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7" podStartSLOduration=2.527389451 podStartE2EDuration="2.527389451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.524507008 +0000 UTC m=+2911.274162317" watchObservedRunningTime="2026-10-30 01:30:02.527389451 +0000 UTC m=+2911.277044761"
Oct 30 01:30:02.564662 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:02.564603    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct" podStartSLOduration=2.564584674 podStartE2EDuration="2.564584674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:02.55974632 +0000 UTC m=+2911.309401621" watchObservedRunningTime="2026-10-30 01:30:02.564584674 +0000 UTC m=+2911.314239984"
Oct 30 01:30:03.202580 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.202538    4425 generic.go:356] "Generic (PLEG): container finished" podID="47e2b0e7-6204-402f-a79c-fa8ff22b3a70" containerID="f99311c09328992ab751b044dab0bafb8c612a244cad193d9b9864f4dfe287ae" exitCode=60
Oct 30 01:30:03.203137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.202625    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct" event={"ID":"47e2b0e7-6204-402f-a79c-fa8ff22b3a70","Type":"ContainerDied","Data":"f99311c09328992ab751b044dab0bafb8c612a244cad193d9b9864f4dfe287ae"}
Oct 30 01:30:03.203137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.202913    4425 scope.go:122] "RemoveContainer" containerID="f99311c09328992ab751b044dab0bafb8c612a244cad193d9b9864f4dfe287ae"
Oct 30 01:30:03.204658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.204638    4425 generic.go:356] "Generic (PLEG): container finished" podID="fba425f6-60cc-44b1-89da-533a37750a41" containerID="259af0dd2819df3f82ed5f89003b1aec3322e3043c39482f78b0eb43bc62ebfd" exitCode=60
Oct 30 01:30:03.204913 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.204664    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c" event={"ID":"fba425f6-60cc-44b1-89da-533a37750a41","Type":"ContainerDied","Data":"259af0dd2819df3f82ed5f89003b1aec3322e3043c39482f78b0eb43bc62ebfd"}
Oct 30 01:30:03.205004 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.204985    4425 scope.go:122] "RemoveContainer" containerID="259af0dd2819df3f82ed5f89003b1aec3322e3043c39482f78b0eb43bc62ebfd"
Oct 30 01:30:03.206722 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.206699    4425 generic.go:356] "Generic (PLEG): container finished" podID="61c894e6-ae30-4978-850a-dfbde9fd69bd" containerID="7e191a74e56529c9ad260fee468d242f108c58f435b58071484f3d6519d133f5" exitCode=60
Oct 30 01:30:03.206814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.206781    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8" event={"ID":"61c894e6-ae30-4978-850a-dfbde9fd69bd","Type":"ContainerDied","Data":"7e191a74e56529c9ad260fee468d242f108c58f435b58071484f3d6519d133f5"}
Oct 30 01:30:03.207001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.206976    4425 scope.go:122] "RemoveContainer" containerID="7e191a74e56529c9ad260fee468d242f108c58f435b58071484f3d6519d133f5"
Oct 30 01:30:03.208458 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.208440    4425 generic.go:356] "Generic (PLEG): container finished" podID="dd86e619-32bb-4d11-afa2-d4a9d723d886" containerID="f4ed0d34a731bee7c7b2ff048ea9407a8312f4cd27fe0965754ba0496c9f1386" exitCode=60
Oct 30 01:30:03.208573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.208542    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk" event={"ID":"dd86e619-32bb-4d11-afa2-d4a9d723d886","Type":"ContainerDied","Data":"f4ed0d34a731bee7c7b2ff048ea9407a8312f4cd27fe0965754ba0496c9f1386"}
Oct 30 01:30:03.208679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.208666    4425 scope.go:122] "RemoveContainer" containerID="f4ed0d34a731bee7c7b2ff048ea9407a8312f4cd27fe0965754ba0496c9f1386"
Oct 30 01:30:03.210210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.210186    4425 generic.go:356] "Generic (PLEG): container finished" podID="230424d1-69ed-40c3-8afd-660d905f9d8d" containerID="1ab1c3fc83303288c048046d3262801c0a335ed26972fc62d16f0a0a97db2431" exitCode=60
Oct 30 01:30:03.210288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.210266    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862" event={"ID":"230424d1-69ed-40c3-8afd-660d905f9d8d","Type":"ContainerDied","Data":"1ab1c3fc83303288c048046d3262801c0a335ed26972fc62d16f0a0a97db2431"}
Oct 30 01:30:03.210444 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.210431    4425 scope.go:122] "RemoveContainer" containerID="1ab1c3fc83303288c048046d3262801c0a335ed26972fc62d16f0a0a97db2431"
Oct 30 01:30:03.211738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.211718    4425 generic.go:356] "Generic (PLEG): container finished" podID="c284f618-116f-426d-b25d-fde4ca3c686e" containerID="0e8a44a10a039670a8453dfaf18596e2178d363bb65d5c4ec9b4eec4c527f08b" exitCode=60
Oct 30 01:30:03.211832 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.211804    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9" event={"ID":"c284f618-116f-426d-b25d-fde4ca3c686e","Type":"ContainerDied","Data":"0e8a44a10a039670a8453dfaf18596e2178d363bb65d5c4ec9b4eec4c527f08b"}
Oct 30 01:30:03.212004 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.211985    4425 scope.go:122] "RemoveContainer" containerID="0e8a44a10a039670a8453dfaf18596e2178d363bb65d5c4ec9b4eec4c527f08b"
Oct 30 01:30:03.213591 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.213569    4425 generic.go:356] "Generic (PLEG): container finished" podID="53190760-7148-47da-93f0-1123e5df62e2" containerID="54c058322e1573dfaebd559f42c641084f10d53fd95061bf796e5bef1da16ea4" exitCode=60
Oct 30 01:30:03.213709 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.213654    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9" event={"ID":"53190760-7148-47da-93f0-1123e5df62e2","Type":"ContainerDied","Data":"54c058322e1573dfaebd559f42c641084f10d53fd95061bf796e5bef1da16ea4"}
Oct 30 01:30:03.213817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.213801    4425 scope.go:122] "RemoveContainer" containerID="54c058322e1573dfaebd559f42c641084f10d53fd95061bf796e5bef1da16ea4"
Oct 30 01:30:03.215275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.215254    4425 generic.go:356] "Generic (PLEG): container finished" podID="3699d466-411f-4f43-b62f-e6955340d1da" containerID="e82111470584a8ed21b21e30799bd368fa974f8c883ae94762ea6f657997384b" exitCode=60
Oct 30 01:30:03.215368 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.215332    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd" event={"ID":"3699d466-411f-4f43-b62f-e6955340d1da","Type":"ContainerDied","Data":"e82111470584a8ed21b21e30799bd368fa974f8c883ae94762ea6f657997384b"}
Oct 30 01:30:03.215506 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.215488    4425 scope.go:122] "RemoveContainer" containerID="e82111470584a8ed21b21e30799bd368fa974f8c883ae94762ea6f657997384b"
Oct 30 01:30:03.218555 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.218122    4425 generic.go:356] "Generic (PLEG): container finished" podID="107404d2-96de-452e-8e7d-24acb8246db3" containerID="f35a73d9bd22aab6a79e4acda2b125ba2cef6d485e834877febee5246fd87bc6" exitCode=60
Oct 30 01:30:03.218555 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.218194    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt" event={"ID":"107404d2-96de-452e-8e7d-24acb8246db3","Type":"ContainerDied","Data":"f35a73d9bd22aab6a79e4acda2b125ba2cef6d485e834877febee5246fd87bc6"}
Oct 30 01:30:03.218555 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.218410    4425 scope.go:122] "RemoveContainer" containerID="f35a73d9bd22aab6a79e4acda2b125ba2cef6d485e834877febee5246fd87bc6"
Oct 30 01:30:03.219913 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.219890    4425 generic.go:356] "Generic (PLEG): container finished" podID="fd93d36a-1693-4cbb-bf97-204d5c295c0d" containerID="4c304c43ff39eea3b41a2b59c225f216514c313c6ab1f6c805606cd012c54f22" exitCode=60
Oct 30 01:30:03.220021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.219973    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx" event={"ID":"fd93d36a-1693-4cbb-bf97-204d5c295c0d","Type":"ContainerDied","Data":"4c304c43ff39eea3b41a2b59c225f216514c313c6ab1f6c805606cd012c54f22"}
Oct 30 01:30:03.220223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.220204    4425 scope.go:122] "RemoveContainer" containerID="4c304c43ff39eea3b41a2b59c225f216514c313c6ab1f6c805606cd012c54f22"
Oct 30 01:30:03.221866 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.221845    4425 generic.go:356] "Generic (PLEG): container finished" podID="b3d49b88-2ba6-4ded-a160-806e4e8fed2b" containerID="5ee9c071cc404b721d9b675a49b54739fd1a012abaea52478efe1445123d6e6d" exitCode=60
Oct 30 01:30:03.221999 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.221917    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n" event={"ID":"b3d49b88-2ba6-4ded-a160-806e4e8fed2b","Type":"ContainerDied","Data":"5ee9c071cc404b721d9b675a49b54739fd1a012abaea52478efe1445123d6e6d"}
Oct 30 01:30:03.222132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.222112    4425 scope.go:122] "RemoveContainer" containerID="5ee9c071cc404b721d9b675a49b54739fd1a012abaea52478efe1445123d6e6d"
Oct 30 01:30:03.223641 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.223623    4425 generic.go:356] "Generic (PLEG): container finished" podID="03eaf7ac-cf67-4da9-b253-3c1994ad57fa" containerID="1a63857779a7abfd00eb02701a89390e4b61586c46da927045959662877babd1" exitCode=60
Oct 30 01:30:03.223741 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.223700    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm" event={"ID":"03eaf7ac-cf67-4da9-b253-3c1994ad57fa","Type":"ContainerDied","Data":"1a63857779a7abfd00eb02701a89390e4b61586c46da927045959662877babd1"}
Oct 30 01:30:03.223899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.223883    4425 scope.go:122] "RemoveContainer" containerID="1a63857779a7abfd00eb02701a89390e4b61586c46da927045959662877babd1"
Oct 30 01:30:03.225418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.225399    4425 generic.go:356] "Generic (PLEG): container finished" podID="4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1" containerID="3ee74e65f0f108a988d74978d278039811d2a3db79bc54a36e5269d569e9351a" exitCode=60
Oct 30 01:30:03.225505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.225483    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb" event={"ID":"4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1","Type":"ContainerDied","Data":"3ee74e65f0f108a988d74978d278039811d2a3db79bc54a36e5269d569e9351a"}
Oct 30 01:30:03.225662 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.225647    4425 scope.go:122] "RemoveContainer" containerID="3ee74e65f0f108a988d74978d278039811d2a3db79bc54a36e5269d569e9351a"
Oct 30 01:30:03.235210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.235148    4425 generic.go:356] "Generic (PLEG): container finished" podID="be3cc419-c375-4b6b-a359-677d816e907b" containerID="38dcd62daf20f8e2271e0bade04bfc38712068d12054e302f5e2de91d0be41c1" exitCode=60
Oct 30 01:30:03.235301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.235219    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7" event={"ID":"be3cc419-c375-4b6b-a359-677d816e907b","Type":"ContainerDied","Data":"38dcd62daf20f8e2271e0bade04bfc38712068d12054e302f5e2de91d0be41c1"}
Oct 30 01:30:03.235508 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:03.235490    4425 scope.go:122] "RemoveContainer" containerID="38dcd62daf20f8e2271e0bade04bfc38712068d12054e302f5e2de91d0be41c1"
Oct 30 01:30:04.248347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.248309    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct" event={"ID":"47e2b0e7-6204-402f-a79c-fa8ff22b3a70","Type":"ContainerStarted","Data":"e182a4d3c1ed0114812a757789bf511716fe95038c3eca11ddd9a07c03509ab7"}
Oct 30 01:30:04.256128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.256093    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c" event={"ID":"fba425f6-60cc-44b1-89da-533a37750a41","Type":"ContainerStarted","Data":"889da4fb98cc1af7fab06ccf8f18bb3ecbe4c372d0f9d020799e7917212d2b14"}
Oct 30 01:30:04.258947 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.258911    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8" event={"ID":"61c894e6-ae30-4978-850a-dfbde9fd69bd","Type":"ContainerStarted","Data":"2d2a45e2c9ccbda0e78795cddf8b4146601c4e76f3ac7fe9a8f21ca955ebf1c4"}
Oct 30 01:30:04.264069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.264041    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk" event={"ID":"dd86e619-32bb-4d11-afa2-d4a9d723d886","Type":"ContainerStarted","Data":"0d593bddbc8ac25ad06dc5a7ec54f8ce6f0a7fb5364759284c045766ccffc782"}
Oct 30 01:30:04.267500 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.267470    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862" event={"ID":"230424d1-69ed-40c3-8afd-660d905f9d8d","Type":"ContainerStarted","Data":"91f3efcc9801b8778d1f3914497097e45bbe5181097701b9de0c7e4330656613"}
Oct 30 01:30:04.269955 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.269931    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9" event={"ID":"c284f618-116f-426d-b25d-fde4ca3c686e","Type":"ContainerStarted","Data":"932207fe531f80a91a1eb16d5ddb9ad053e1ed8b1c909a69aaf747b9d7c29c92"}
Oct 30 01:30:04.272799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.272775    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9" event={"ID":"53190760-7148-47da-93f0-1123e5df62e2","Type":"ContainerStarted","Data":"9c876ac4b5d789d9c20f9556061198428a82cf841c112d4cb3fe5f15895d9f28"}
Oct 30 01:30:04.276240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.276214    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd" event={"ID":"3699d466-411f-4f43-b62f-e6955340d1da","Type":"ContainerStarted","Data":"00a54dc39418f2348243269bad8b3de89caea2154e18d1f83d351138176c7b11"}
Oct 30 01:30:04.278534 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.278506    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt" event={"ID":"107404d2-96de-452e-8e7d-24acb8246db3","Type":"ContainerStarted","Data":"9bcf92783744881dd1b077efe90ec0987b3b0045367db5ecdb0e6b45202808c3"}
Oct 30 01:30:04.281446 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.281426    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx" event={"ID":"fd93d36a-1693-4cbb-bf97-204d5c295c0d","Type":"ContainerStarted","Data":"adc634652b6cd08572e5f16c16dc0b399a89ebab5379b6d2a748181a937d908e"}
Oct 30 01:30:04.284071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.284051    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm" event={"ID":"03eaf7ac-cf67-4da9-b253-3c1994ad57fa","Type":"ContainerStarted","Data":"dd48f8d1d1a5954756fdefa022668918f477763902de74f0a526cc6d2fa53ee1"}
Oct 30 01:30:04.513596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.513526    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c" podStartSLOduration=4.513507231 podStartE2EDuration="4.513507231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:04.50807684 +0000 UTC m=+2913.257732148" watchObservedRunningTime="2026-10-30 01:30:04.513507231 +0000 UTC m=+2913.263162541"
Oct 30 01:30:04.543825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.543761    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd" podStartSLOduration=4.543746716 podStartE2EDuration="4.543746716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:04.542562258 +0000 UTC m=+2913.292217577" watchObservedRunningTime="2026-10-30 01:30:04.543746716 +0000 UTC m=+2913.293402024"
Oct 30 01:30:04.581103 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.581044    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862" podStartSLOduration=4.5810274589999995 podStartE2EDuration="4.581027459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:04.578748339 +0000 UTC m=+2913.328403649" watchObservedRunningTime="2026-10-30 01:30:04.581027459 +0000 UTC m=+2913.330682795"
Oct 30 01:30:04.797742 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.797705    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"]
Oct 30 01:30:04.803981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.803954    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"
Oct 30 01:30:04.822990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.822953    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"]
Oct 30 01:30:04.867485 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.867456    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:30:04.970938 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.970849    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90aa3e58-7zgbr\" (UID: \"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"
Oct 30 01:30:04.970938 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.970903    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhk6n\" (UniqueName: \"kubernetes.io/projected/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-kube-api-access-jhk6n\") pod \"maas-api-revoke-sub-90aa3e58-7zgbr\" (UID: \"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"
Oct 30 01:30:04.971180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:04.971147    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90aa3e58-7zgbr\" (UID: \"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"
Oct 30 01:30:05.072237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.072202    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90aa3e58-7zgbr\" (UID: \"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"
Oct 30 01:30:05.072421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.072335    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90aa3e58-7zgbr\" (UID: \"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"
Oct 30 01:30:05.072421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.072365    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhk6n\" (UniqueName: \"kubernetes.io/projected/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-kube-api-access-jhk6n\") pod \"maas-api-revoke-sub-90aa3e58-7zgbr\" (UID: \"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"
Oct 30 01:30:05.073134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.073114    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90aa3e58-7zgbr\" (UID: \"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"
Oct 30 01:30:05.074751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.074720    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90aa3e58-7zgbr\" (UID: \"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"
Oct 30 01:30:05.093259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.093227    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhk6n\" (UniqueName: \"kubernetes.io/projected/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-kube-api-access-jhk6n\") pod \"maas-api-revoke-sub-90aa3e58-7zgbr\" (UID: \"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"
Oct 30 01:30:05.296746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.296715    4425 generic.go:356] "Generic (PLEG): container finished" podID="be3cc419-c375-4b6b-a359-677d816e907b" containerID="740018d6eafa31dd0d56755582c3803ecce1280a766837af781db9d49f2240b9" exitCode=60
Oct 30 01:30:05.297184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.296793    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7" event={"ID":"be3cc419-c375-4b6b-a359-677d816e907b","Type":"ContainerDied","Data":"740018d6eafa31dd0d56755582c3803ecce1280a766837af781db9d49f2240b9"}
Oct 30 01:30:05.297184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.296854    4425 scope.go:122] "RemoveContainer" containerID="38dcd62daf20f8e2271e0bade04bfc38712068d12054e302f5e2de91d0be41c1"
Oct 30 01:30:05.297184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.297113    4425 scope.go:122] "RemoveContainer" containerID="740018d6eafa31dd0d56755582c3803ecce1280a766837af781db9d49f2240b9"
Oct 30 01:30:05.297520 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:05.297489    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7_odh-ai-gateway-infra(be3cc419-c375-4b6b-a359-677d816e907b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7" podUID="be3cc419-c375-4b6b-a359-677d816e907b"
Oct 30 01:30:05.299177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.299143    4425 generic.go:356] "Generic (PLEG): container finished" podID="47e2b0e7-6204-402f-a79c-fa8ff22b3a70" containerID="e182a4d3c1ed0114812a757789bf511716fe95038c3eca11ddd9a07c03509ab7" exitCode=60
Oct 30 01:30:05.299473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.299198    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct" event={"ID":"47e2b0e7-6204-402f-a79c-fa8ff22b3a70","Type":"ContainerDied","Data":"e182a4d3c1ed0114812a757789bf511716fe95038c3eca11ddd9a07c03509ab7"}
Oct 30 01:30:05.299553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.299489    4425 scope.go:122] "RemoveContainer" containerID="e182a4d3c1ed0114812a757789bf511716fe95038c3eca11ddd9a07c03509ab7"
Oct 30 01:30:05.299844 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:05.299809    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct_odh-ai-gateway-infra(47e2b0e7-6204-402f-a79c-fa8ff22b3a70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct" podUID="47e2b0e7-6204-402f-a79c-fa8ff22b3a70"
Oct 30 01:30:05.304212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.304193    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"
Oct 30 01:30:05.304569 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.304542    4425 generic.go:356] "Generic (PLEG): container finished" podID="fba425f6-60cc-44b1-89da-533a37750a41" containerID="889da4fb98cc1af7fab06ccf8f18bb3ecbe4c372d0f9d020799e7917212d2b14" exitCode=60
Oct 30 01:30:05.304664 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.304620    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c" event={"ID":"fba425f6-60cc-44b1-89da-533a37750a41","Type":"ContainerDied","Data":"889da4fb98cc1af7fab06ccf8f18bb3ecbe4c372d0f9d020799e7917212d2b14"}
Oct 30 01:30:05.304870 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.304850    4425 scope.go:122] "RemoveContainer" containerID="889da4fb98cc1af7fab06ccf8f18bb3ecbe4c372d0f9d020799e7917212d2b14"
Oct 30 01:30:05.305148 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:05.305129    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c_odh-ai-gateway-infra(fba425f6-60cc-44b1-89da-533a37750a41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c" podUID="fba425f6-60cc-44b1-89da-533a37750a41"
Oct 30 01:30:05.306943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.306924    4425 generic.go:356] "Generic (PLEG): container finished" podID="61c894e6-ae30-4978-850a-dfbde9fd69bd" containerID="2d2a45e2c9ccbda0e78795cddf8b4146601c4e76f3ac7fe9a8f21ca955ebf1c4" exitCode=60
Oct 30 01:30:05.307021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.306998    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8" event={"ID":"61c894e6-ae30-4978-850a-dfbde9fd69bd","Type":"ContainerDied","Data":"2d2a45e2c9ccbda0e78795cddf8b4146601c4e76f3ac7fe9a8f21ca955ebf1c4"}
Oct 30 01:30:05.307278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.307262    4425 scope.go:122] "RemoveContainer" containerID="2d2a45e2c9ccbda0e78795cddf8b4146601c4e76f3ac7fe9a8f21ca955ebf1c4"
Oct 30 01:30:05.307531 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:05.307510    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8_odh-ai-gateway-infra(61c894e6-ae30-4978-850a-dfbde9fd69bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8" podUID="61c894e6-ae30-4978-850a-dfbde9fd69bd"
Oct 30 01:30:05.308937 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.308918    4425 generic.go:356] "Generic (PLEG): container finished" podID="dd86e619-32bb-4d11-afa2-d4a9d723d886" containerID="0d593bddbc8ac25ad06dc5a7ec54f8ce6f0a7fb5364759284c045766ccffc782" exitCode=60
Oct 30 01:30:05.309027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.309004    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk" event={"ID":"dd86e619-32bb-4d11-afa2-d4a9d723d886","Type":"ContainerDied","Data":"0d593bddbc8ac25ad06dc5a7ec54f8ce6f0a7fb5364759284c045766ccffc782"}
Oct 30 01:30:05.309265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.309214    4425 scope.go:122] "RemoveContainer" containerID="0d593bddbc8ac25ad06dc5a7ec54f8ce6f0a7fb5364759284c045766ccffc782"
Oct 30 01:30:05.309454 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:05.309432    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk_odh-ai-gateway-infra(dd86e619-32bb-4d11-afa2-d4a9d723d886)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk" podUID="dd86e619-32bb-4d11-afa2-d4a9d723d886"
Oct 30 01:30:05.310999 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.310975    4425 generic.go:356] "Generic (PLEG): container finished" podID="230424d1-69ed-40c3-8afd-660d905f9d8d" containerID="91f3efcc9801b8778d1f3914497097e45bbe5181097701b9de0c7e4330656613" exitCode=60
Oct 30 01:30:05.311094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.311044    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862" event={"ID":"230424d1-69ed-40c3-8afd-660d905f9d8d","Type":"ContainerDied","Data":"91f3efcc9801b8778d1f3914497097e45bbe5181097701b9de0c7e4330656613"}
Oct 30 01:30:05.311260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.311241    4425 scope.go:122] "RemoveContainer" containerID="91f3efcc9801b8778d1f3914497097e45bbe5181097701b9de0c7e4330656613"
Oct 30 01:30:05.311486 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:05.311465    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862_odh-ai-gateway-infra(230424d1-69ed-40c3-8afd-660d905f9d8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862" podUID="230424d1-69ed-40c3-8afd-660d905f9d8d"
Oct 30 01:30:05.313229 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.313092    4425 generic.go:356] "Generic (PLEG): container finished" podID="c284f618-116f-426d-b25d-fde4ca3c686e" containerID="932207fe531f80a91a1eb16d5ddb9ad053e1ed8b1c909a69aaf747b9d7c29c92" exitCode=60
Oct 30 01:30:05.313293 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.313224    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9" event={"ID":"c284f618-116f-426d-b25d-fde4ca3c686e","Type":"ContainerDied","Data":"932207fe531f80a91a1eb16d5ddb9ad053e1ed8b1c909a69aaf747b9d7c29c92"}
Oct 30 01:30:05.313361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.313341    4425 scope.go:122] "RemoveContainer" containerID="f99311c09328992ab751b044dab0bafb8c612a244cad193d9b9864f4dfe287ae"
Oct 30 01:30:05.313423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.313381    4425 scope.go:122] "RemoveContainer" containerID="932207fe531f80a91a1eb16d5ddb9ad053e1ed8b1c909a69aaf747b9d7c29c92"
Oct 30 01:30:05.313649 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:05.313627    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9_odh-ai-gateway-infra(c284f618-116f-426d-b25d-fde4ca3c686e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9" podUID="c284f618-116f-426d-b25d-fde4ca3c686e"
Oct 30 01:30:05.315550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.315530    4425 generic.go:356] "Generic (PLEG): container finished" podID="53190760-7148-47da-93f0-1123e5df62e2" containerID="9c876ac4b5d789d9c20f9556061198428a82cf841c112d4cb3fe5f15895d9f28" exitCode=60
Oct 30 01:30:05.315651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.315596    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9" event={"ID":"53190760-7148-47da-93f0-1123e5df62e2","Type":"ContainerDied","Data":"9c876ac4b5d789d9c20f9556061198428a82cf841c112d4cb3fe5f15895d9f28"}
Oct 30 01:30:05.315842 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.315823    4425 scope.go:122] "RemoveContainer" containerID="9c876ac4b5d789d9c20f9556061198428a82cf841c112d4cb3fe5f15895d9f28"
Oct 30 01:30:05.316068 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:05.316042    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9_odh-ai-gateway-infra(53190760-7148-47da-93f0-1123e5df62e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9" podUID="53190760-7148-47da-93f0-1123e5df62e2"
Oct 30 01:30:05.317418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.317398    4425 generic.go:356] "Generic (PLEG): container finished" podID="3699d466-411f-4f43-b62f-e6955340d1da" containerID="00a54dc39418f2348243269bad8b3de89caea2154e18d1f83d351138176c7b11" exitCode=60
Oct 30 01:30:05.317517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.317482    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd" event={"ID":"3699d466-411f-4f43-b62f-e6955340d1da","Type":"ContainerDied","Data":"00a54dc39418f2348243269bad8b3de89caea2154e18d1f83d351138176c7b11"}
Oct 30 01:30:05.317652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.317638    4425 scope.go:122] "RemoveContainer" containerID="00a54dc39418f2348243269bad8b3de89caea2154e18d1f83d351138176c7b11"
Oct 30 01:30:05.317871 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:05.317846    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd_odh-ai-gateway-infra(3699d466-411f-4f43-b62f-e6955340d1da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd" podUID="3699d466-411f-4f43-b62f-e6955340d1da"
Oct 30 01:30:05.319387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.319369    4425 generic.go:356] "Generic (PLEG): container finished" podID="107404d2-96de-452e-8e7d-24acb8246db3" containerID="9bcf92783744881dd1b077efe90ec0987b3b0045367db5ecdb0e6b45202808c3" exitCode=60
Oct 30 01:30:05.319471 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.319426    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt" event={"ID":"107404d2-96de-452e-8e7d-24acb8246db3","Type":"ContainerDied","Data":"9bcf92783744881dd1b077efe90ec0987b3b0045367db5ecdb0e6b45202808c3"}
Oct 30 01:30:05.319661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.319645    4425 scope.go:122] "RemoveContainer" containerID="9bcf92783744881dd1b077efe90ec0987b3b0045367db5ecdb0e6b45202808c3"
Oct 30 01:30:05.319911 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:05.319885    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt_odh-ai-gateway-infra(107404d2-96de-452e-8e7d-24acb8246db3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt" podUID="107404d2-96de-452e-8e7d-24acb8246db3"
Oct 30 01:30:05.321463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.321441    4425 generic.go:356] "Generic (PLEG): container finished" podID="fd93d36a-1693-4cbb-bf97-204d5c295c0d" containerID="adc634652b6cd08572e5f16c16dc0b399a89ebab5379b6d2a748181a937d908e" exitCode=60
Oct 30 01:30:05.321559 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.321521    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx" event={"ID":"fd93d36a-1693-4cbb-bf97-204d5c295c0d","Type":"ContainerDied","Data":"adc634652b6cd08572e5f16c16dc0b399a89ebab5379b6d2a748181a937d908e"}
Oct 30 01:30:05.321727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.321709    4425 scope.go:122] "RemoveContainer" containerID="adc634652b6cd08572e5f16c16dc0b399a89ebab5379b6d2a748181a937d908e"
Oct 30 01:30:05.321997 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:05.321974    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx_odh-ai-gateway-infra(fd93d36a-1693-4cbb-bf97-204d5c295c0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx" podUID="fd93d36a-1693-4cbb-bf97-204d5c295c0d"
Oct 30 01:30:05.323686 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.323664    4425 generic.go:356] "Generic (PLEG): container finished" podID="b3d49b88-2ba6-4ded-a160-806e4e8fed2b" containerID="29cb85b75507f7ba894dd3b2328871ccb27a00f28b1cb6e195ae2552ad4a31ec" exitCode=60
Oct 30 01:30:05.323771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.323728    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n" event={"ID":"b3d49b88-2ba6-4ded-a160-806e4e8fed2b","Type":"ContainerDied","Data":"29cb85b75507f7ba894dd3b2328871ccb27a00f28b1cb6e195ae2552ad4a31ec"}
Oct 30 01:30:05.323988 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.323970    4425 scope.go:122] "RemoveContainer" containerID="29cb85b75507f7ba894dd3b2328871ccb27a00f28b1cb6e195ae2552ad4a31ec"
Oct 30 01:30:05.324239 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:05.324213    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n_odh-ai-gateway-infra(b3d49b88-2ba6-4ded-a160-806e4e8fed2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n" podUID="b3d49b88-2ba6-4ded-a160-806e4e8fed2b"
Oct 30 01:30:05.325970 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.325948    4425 generic.go:356] "Generic (PLEG): container finished" podID="03eaf7ac-cf67-4da9-b253-3c1994ad57fa" containerID="dd48f8d1d1a5954756fdefa022668918f477763902de74f0a526cc6d2fa53ee1" exitCode=60
Oct 30 01:30:05.326068 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.325987    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm" event={"ID":"03eaf7ac-cf67-4da9-b253-3c1994ad57fa","Type":"ContainerDied","Data":"dd48f8d1d1a5954756fdefa022668918f477763902de74f0a526cc6d2fa53ee1"}
Oct 30 01:30:05.326304 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.326214    4425 scope.go:122] "RemoveContainer" containerID="dd48f8d1d1a5954756fdefa022668918f477763902de74f0a526cc6d2fa53ee1"
Oct 30 01:30:05.326403 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:05.326385    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm_odh-ai-gateway-infra(03eaf7ac-cf67-4da9-b253-3c1994ad57fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm" podUID="03eaf7ac-cf67-4da9-b253-3c1994ad57fa"
Oct 30 01:30:05.328338 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.328318    4425 generic.go:356] "Generic (PLEG): container finished" podID="4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1" containerID="373c11c77b2d93326c8f31f5042f56f9f0cf463efa70e8c0e0cf5464e4f63acc" exitCode=60
Oct 30 01:30:05.328453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.328373    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb" event={"ID":"4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1","Type":"ContainerDied","Data":"373c11c77b2d93326c8f31f5042f56f9f0cf463efa70e8c0e0cf5464e4f63acc"}
Oct 30 01:30:05.328617 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.328602    4425 scope.go:122] "RemoveContainer" containerID="373c11c77b2d93326c8f31f5042f56f9f0cf463efa70e8c0e0cf5464e4f63acc"
Oct 30 01:30:05.328828 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:05.328808    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb_odh-ai-gateway-infra(4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb" podUID="4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1"
Oct 30 01:30:05.364584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.364559    4425 scope.go:122] "RemoveContainer" containerID="259af0dd2819df3f82ed5f89003b1aec3322e3043c39482f78b0eb43bc62ebfd"
Oct 30 01:30:05.384194 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.384136    4425 scope.go:122] "RemoveContainer" containerID="7e191a74e56529c9ad260fee468d242f108c58f435b58071484f3d6519d133f5"
Oct 30 01:30:05.418911 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.418642    4425 scope.go:122] "RemoveContainer" containerID="f4ed0d34a731bee7c7b2ff048ea9407a8312f4cd27fe0965754ba0496c9f1386"
Oct 30 01:30:05.435330 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.435310    4425 scope.go:122] "RemoveContainer" containerID="1ab1c3fc83303288c048046d3262801c0a335ed26972fc62d16f0a0a97db2431"
Oct 30 01:30:05.458684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.458660    4425 scope.go:122] "RemoveContainer" containerID="0e8a44a10a039670a8453dfaf18596e2178d363bb65d5c4ec9b4eec4c527f08b"
Oct 30 01:30:05.476249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.476223    4425 scope.go:122] "RemoveContainer" containerID="54c058322e1573dfaebd559f42c641084f10d53fd95061bf796e5bef1da16ea4"
Oct 30 01:30:05.491874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.491854    4425 scope.go:122] "RemoveContainer" containerID="e82111470584a8ed21b21e30799bd368fa974f8c883ae94762ea6f657997384b"
Oct 30 01:30:05.508975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.508922    4425 scope.go:122] "RemoveContainer" containerID="f35a73d9bd22aab6a79e4acda2b125ba2cef6d485e834877febee5246fd87bc6"
Oct 30 01:30:05.527303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.527277    4425 scope.go:122] "RemoveContainer" containerID="4c304c43ff39eea3b41a2b59c225f216514c313c6ab1f6c805606cd012c54f22"
Oct 30 01:30:05.543087 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.543057    4425 scope.go:122] "RemoveContainer" containerID="5ee9c071cc404b721d9b675a49b54739fd1a012abaea52478efe1445123d6e6d"
Oct 30 01:30:05.557070 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.557051    4425 scope.go:122] "RemoveContainer" containerID="1a63857779a7abfd00eb02701a89390e4b61586c46da927045959662877babd1"
Oct 30 01:30:05.571398 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.571379    4425 scope.go:122] "RemoveContainer" containerID="3ee74e65f0f108a988d74978d278039811d2a3db79bc54a36e5269d569e9351a"
Oct 30 01:30:05.637533 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:05.637503    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac63f3a_d1cc_4383_abe3_cbbff6e968d5.slice/crio-98ab58a7c97474e1e49b332386c931cae369f771cbbf34af4150b1dca16964cc WatchSource:0}: Error finding container 98ab58a7c97474e1e49b332386c931cae369f771cbbf34af4150b1dca16964cc: Status 404 returned error can't find the container with id 98ab58a7c97474e1e49b332386c931cae369f771cbbf34af4150b1dca16964cc
Oct 30 01:30:05.638766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:05.638745    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"]
Oct 30 01:30:06.339638 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:06.339602    4425 scope.go:122] "RemoveContainer" containerID="0d593bddbc8ac25ad06dc5a7ec54f8ce6f0a7fb5364759284c045766ccffc782"
Oct 30 01:30:06.340122 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:06.339951    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk_odh-ai-gateway-infra(dd86e619-32bb-4d11-afa2-d4a9d723d886)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk" podUID="dd86e619-32bb-4d11-afa2-d4a9d723d886"
Oct 30 01:30:06.341077 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:06.341043    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr" event={"ID":"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5","Type":"ContainerStarted","Data":"88f0fc6c65a5a2c2b4de4d8a979f7f47e0a85d0e42a8fd3d30ad1ee4177304cf"}
Oct 30 01:30:06.341245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:06.341099    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr" event={"ID":"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5","Type":"ContainerStarted","Data":"98ab58a7c97474e1e49b332386c931cae369f771cbbf34af4150b1dca16964cc"}
Oct 30 01:30:06.343273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:06.343252    4425 scope.go:122] "RemoveContainer" containerID="91f3efcc9801b8778d1f3914497097e45bbe5181097701b9de0c7e4330656613"
Oct 30 01:30:06.346582 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:06.343919    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862_odh-ai-gateway-infra(230424d1-69ed-40c3-8afd-660d905f9d8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862" podUID="230424d1-69ed-40c3-8afd-660d905f9d8d"
Oct 30 01:30:06.352654 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:06.352183    4425 scope.go:122] "RemoveContainer" containerID="932207fe531f80a91a1eb16d5ddb9ad053e1ed8b1c909a69aaf747b9d7c29c92"
Oct 30 01:30:06.357170 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:06.353240    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9_odh-ai-gateway-infra(c284f618-116f-426d-b25d-fde4ca3c686e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9" podUID="c284f618-116f-426d-b25d-fde4ca3c686e"
Oct 30 01:30:06.359545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:06.359524    4425 scope.go:122] "RemoveContainer" containerID="9c876ac4b5d789d9c20f9556061198428a82cf841c112d4cb3fe5f15895d9f28"
Oct 30 01:30:06.360134 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:06.360108    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9_odh-ai-gateway-infra(53190760-7148-47da-93f0-1123e5df62e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9" podUID="53190760-7148-47da-93f0-1123e5df62e2"
Oct 30 01:30:06.361812 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:06.361792    4425 scope.go:122] "RemoveContainer" containerID="00a54dc39418f2348243269bad8b3de89caea2154e18d1f83d351138176c7b11"
Oct 30 01:30:06.362274 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:06.362245    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd_odh-ai-gateway-infra(3699d466-411f-4f43-b62f-e6955340d1da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd" podUID="3699d466-411f-4f43-b62f-e6955340d1da"
Oct 30 01:30:06.363757 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:06.363736    4425 scope.go:122] "RemoveContainer" containerID="9bcf92783744881dd1b077efe90ec0987b3b0045367db5ecdb0e6b45202808c3"
Oct 30 01:30:06.364015 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:06.363991    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt_odh-ai-gateway-infra(107404d2-96de-452e-8e7d-24acb8246db3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt" podUID="107404d2-96de-452e-8e7d-24acb8246db3"
Oct 30 01:30:06.365319 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:06.365303    4425 scope.go:122] "RemoveContainer" containerID="adc634652b6cd08572e5f16c16dc0b399a89ebab5379b6d2a748181a937d908e"
Oct 30 01:30:06.365556 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:06.365532    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx_odh-ai-gateway-infra(fd93d36a-1693-4cbb-bf97-204d5c295c0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx" podUID="fd93d36a-1693-4cbb-bf97-204d5c295c0d"
Oct 30 01:30:06.366886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:06.366870    4425 scope.go:122] "RemoveContainer" containerID="29cb85b75507f7ba894dd3b2328871ccb27a00f28b1cb6e195ae2552ad4a31ec"
Oct 30 01:30:06.367087 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:06.367067    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n_odh-ai-gateway-infra(b3d49b88-2ba6-4ded-a160-806e4e8fed2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n" podUID="b3d49b88-2ba6-4ded-a160-806e4e8fed2b"
Oct 30 01:30:06.368472 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:06.368452    4425 scope.go:122] "RemoveContainer" containerID="dd48f8d1d1a5954756fdefa022668918f477763902de74f0a526cc6d2fa53ee1"
Oct 30 01:30:06.368698 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:06.368679    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm_odh-ai-gateway-infra(03eaf7ac-cf67-4da9-b253-3c1994ad57fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm" podUID="03eaf7ac-cf67-4da9-b253-3c1994ad57fa"
Oct 30 01:30:06.369946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:06.369930    4425 scope.go:122] "RemoveContainer" containerID="373c11c77b2d93326c8f31f5042f56f9f0cf463efa70e8c0e0cf5464e4f63acc"
Oct 30 01:30:06.370134 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:06.370113    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb_odh-ai-gateway-infra(4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb" podUID="4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1"
Oct 30 01:30:06.371452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:06.371435    4425 scope.go:122] "RemoveContainer" containerID="740018d6eafa31dd0d56755582c3803ecce1280a766837af781db9d49f2240b9"
Oct 30 01:30:06.371689 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:06.371638    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7_odh-ai-gateway-infra(be3cc419-c375-4b6b-a359-677d816e907b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7" podUID="be3cc419-c375-4b6b-a359-677d816e907b"
Oct 30 01:30:06.373082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:06.373066    4425 scope.go:122] "RemoveContainer" containerID="e182a4d3c1ed0114812a757789bf511716fe95038c3eca11ddd9a07c03509ab7"
Oct 30 01:30:06.373308 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:06.373290    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct_odh-ai-gateway-infra(47e2b0e7-6204-402f-a79c-fa8ff22b3a70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct" podUID="47e2b0e7-6204-402f-a79c-fa8ff22b3a70"
Oct 30 01:30:06.374559 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:06.374540    4425 scope.go:122] "RemoveContainer" containerID="889da4fb98cc1af7fab06ccf8f18bb3ecbe4c372d0f9d020799e7917212d2b14"
Oct 30 01:30:06.374765 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:06.374748    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c_odh-ai-gateway-infra(fba425f6-60cc-44b1-89da-533a37750a41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c" podUID="fba425f6-60cc-44b1-89da-533a37750a41"
Oct 30 01:30:06.375900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:06.375884    4425 scope.go:122] "RemoveContainer" containerID="2d2a45e2c9ccbda0e78795cddf8b4146601c4e76f3ac7fe9a8f21ca955ebf1c4"
Oct 30 01:30:06.376063 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:06.376047    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8_odh-ai-gateway-infra(61c894e6-ae30-4978-850a-dfbde9fd69bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8" podUID="61c894e6-ae30-4978-850a-dfbde9fd69bd"
Oct 30 01:30:06.419148 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:06.419045    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr" podStartSLOduration=2.419028531 podStartE2EDuration="2.419028531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:06.416552013 +0000 UTC m=+2915.166207322" watchObservedRunningTime="2026-10-30 01:30:06.419028531 +0000 UTC m=+2915.168683841"
Oct 30 01:30:07.385106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:07.385070    4425 generic.go:356] "Generic (PLEG): container finished" podID="2ac63f3a-d1cc-4383-abe3-cbbff6e968d5" containerID="88f0fc6c65a5a2c2b4de4d8a979f7f47e0a85d0e42a8fd3d30ad1ee4177304cf" exitCode=0
Oct 30 01:30:07.385592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:07.385148    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr" event={"ID":"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5","Type":"ContainerDied","Data":"88f0fc6c65a5a2c2b4de4d8a979f7f47e0a85d0e42a8fd3d30ad1ee4177304cf"}
Oct 30 01:30:07.560266 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:07.560220    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:30:08.080677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:08.080640    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:30:08.158394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:08.158359    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"]
Oct 30 01:30:08.557996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:08.557969    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"
Oct 30 01:30:08.720301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:08.720215    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-kube-api-access-jhk6n\" (UniqueName: \"kubernetes.io/projected/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-kube-api-access-jhk6n\") pod \"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5\" (UID: \"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5\") "
Oct 30 01:30:08.720545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:08.720346    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-maas-api-xks-ca\") pod \"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5\" (UID: \"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5\") "
Oct 30 01:30:08.720545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:08.720428    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-maas-api-openshift-ca\") pod \"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5\" (UID: \"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5\") "
Oct 30 01:30:08.720784 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:08.720756    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-maas-api-openshift-ca" pod "2ac63f3a-d1cc-4383-abe3-cbbff6e968d5" (UID: "2ac63f3a-d1cc-4383-abe3-cbbff6e968d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:08.720915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:08.720901    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:08.722331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:08.722302    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-kube-api-access-jhk6n" pod "2ac63f3a-d1cc-4383-abe3-cbbff6e968d5" (UID: "2ac63f3a-d1cc-4383-abe3-cbbff6e968d5"). InnerVolumeSpecName "kube-api-access-jhk6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:08.722513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:08.722494    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-maas-api-xks-ca" pod "2ac63f3a-d1cc-4383-abe3-cbbff6e968d5" (UID: "2ac63f3a-d1cc-4383-abe3-cbbff6e968d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:08.822378 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:08.822342    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhk6n\" (UniqueName: \"kubernetes.io/projected/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-kube-api-access-jhk6n\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:08.822378 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:08.822371    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:09.407581 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:09.407549    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"
Oct 30 01:30:09.407782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:09.407586    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr" event={"ID":"2ac63f3a-d1cc-4383-abe3-cbbff6e968d5","Type":"ContainerDied","Data":"98ab58a7c97474e1e49b332386c931cae369f771cbbf34af4150b1dca16964cc"}
Oct 30 01:30:09.407782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:09.407649    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98ab58a7c97474e1e49b332386c931cae369f771cbbf34af4150b1dca16964cc"
Oct 30 01:30:12.716763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.715999    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"]
Oct 30 01:30:12.717251 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.717081    4425 cpu_manager.go:414] "containerMap: removing container" podUID="2ac63f3a-d1cc-4383-abe3-cbbff6e968d5" containerName="revoke-keys"
Oct 30 01:30:12.717251 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.717099    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ac63f3a-d1cc-4383-abe3-cbbff6e968d5" containerName="revoke-keys"
Oct 30 01:30:12.717419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.717264    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ac63f3a-d1cc-4383-abe3-cbbff6e968d5" containerName="revoke-keys"
Oct 30 01:30:12.722021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.721996    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"
Oct 30 01:30:12.740001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.739975    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"]
Oct 30 01:30:12.786248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.783460    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5fqd\" (UniqueName: \"kubernetes.io/projected/600ad629-faf2-49eb-9d14-05f9cf294799-kube-api-access-t5fqd\") pod \"payload-processing-external-model-d7c4fc775-sgwth\" (UID: \"600ad629-faf2-49eb-9d14-05f9cf294799\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"
Oct 30 01:30:12.786248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.783557    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/600ad629-faf2-49eb-9d14-05f9cf294799-provider-credentials\") pod \"payload-processing-external-model-d7c4fc775-sgwth\" (UID: \"600ad629-faf2-49eb-9d14-05f9cf294799\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"
Oct 30 01:30:12.786248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.783602    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600ad629-faf2-49eb-9d14-05f9cf294799-plugins-config-volume\") pod \"payload-processing-external-model-d7c4fc775-sgwth\" (UID: \"600ad629-faf2-49eb-9d14-05f9cf294799\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"
Oct 30 01:30:12.786248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.783650    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/600ad629-faf2-49eb-9d14-05f9cf294799-routing-overlay-volume\") pod \"payload-processing-external-model-d7c4fc775-sgwth\" (UID: \"600ad629-faf2-49eb-9d14-05f9cf294799\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"
Oct 30 01:30:12.885305 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.885269    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/600ad629-faf2-49eb-9d14-05f9cf294799-routing-overlay-volume\") pod \"payload-processing-external-model-d7c4fc775-sgwth\" (UID: \"600ad629-faf2-49eb-9d14-05f9cf294799\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"
Oct 30 01:30:12.885520 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.885399    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5fqd\" (UniqueName: \"kubernetes.io/projected/600ad629-faf2-49eb-9d14-05f9cf294799-kube-api-access-t5fqd\") pod \"payload-processing-external-model-d7c4fc775-sgwth\" (UID: \"600ad629-faf2-49eb-9d14-05f9cf294799\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"
Oct 30 01:30:12.885520 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.885494    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/600ad629-faf2-49eb-9d14-05f9cf294799-provider-credentials\") pod \"payload-processing-external-model-d7c4fc775-sgwth\" (UID: \"600ad629-faf2-49eb-9d14-05f9cf294799\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"
Oct 30 01:30:12.885650 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.885541    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600ad629-faf2-49eb-9d14-05f9cf294799-plugins-config-volume\") pod \"payload-processing-external-model-d7c4fc775-sgwth\" (UID: \"600ad629-faf2-49eb-9d14-05f9cf294799\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"
Oct 30 01:30:12.886271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.886240    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/600ad629-faf2-49eb-9d14-05f9cf294799-routing-overlay-volume\") pod \"payload-processing-external-model-d7c4fc775-sgwth\" (UID: \"600ad629-faf2-49eb-9d14-05f9cf294799\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"
Oct 30 01:30:12.886409 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.886324    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600ad629-faf2-49eb-9d14-05f9cf294799-plugins-config-volume\") pod \"payload-processing-external-model-d7c4fc775-sgwth\" (UID: \"600ad629-faf2-49eb-9d14-05f9cf294799\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"
Oct 30 01:30:12.888236 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.888210    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/600ad629-faf2-49eb-9d14-05f9cf294799-provider-credentials\") pod \"payload-processing-external-model-d7c4fc775-sgwth\" (UID: \"600ad629-faf2-49eb-9d14-05f9cf294799\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"
Oct 30 01:30:12.907098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:12.907069    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5fqd\" (UniqueName: \"kubernetes.io/projected/600ad629-faf2-49eb-9d14-05f9cf294799-kube-api-access-t5fqd\") pod \"payload-processing-external-model-d7c4fc775-sgwth\" (UID: \"600ad629-faf2-49eb-9d14-05f9cf294799\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"
Oct 30 01:30:13.039368 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.039327    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"
Oct 30 01:30:13.161560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.161324    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz" podUID="88e7d729-60a5-46d5-b177-fe3217157f61" containerName="payload-processing" containerID="cri-o://c0d0230ff0b0ea27331da6dff70708ea324a34b3f9c7f4e737a86e9b3f7cd553" gracePeriod=25
Oct 30 01:30:13.363930 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:13.363713    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600ad629_faf2_49eb_9d14_05f9cf294799.slice/crio-a2bcae866ddb6c9c43762b571478dd5c933787c35b69e745da37dbdd34603d06 WatchSource:0}: Error finding container a2bcae866ddb6c9c43762b571478dd5c933787c35b69e745da37dbdd34603d06: Status 404 returned error can't find the container with id a2bcae866ddb6c9c43762b571478dd5c933787c35b69e745da37dbdd34603d06
Oct 30 01:30:13.368497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.366788    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"]
Oct 30 01:30:13.448231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.448195    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth" event={"ID":"600ad629-faf2-49eb-9d14-05f9cf294799","Type":"ContainerStarted","Data":"a2bcae866ddb6c9c43762b571478dd5c933787c35b69e745da37dbdd34603d06"}
Oct 30 01:30:13.449830 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.449798    4425 generic.go:356] "Generic (PLEG): container finished" podID="88e7d729-60a5-46d5-b177-fe3217157f61" containerID="c0d0230ff0b0ea27331da6dff70708ea324a34b3f9c7f4e737a86e9b3f7cd553" exitCode=0
Oct 30 01:30:13.449957 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.449846    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz" event={"ID":"88e7d729-60a5-46d5-b177-fe3217157f61","Type":"ContainerDied","Data":"c0d0230ff0b0ea27331da6dff70708ea324a34b3f9c7f4e737a86e9b3f7cd553"}
Oct 30 01:30:13.449957 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.449889    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz" event={"ID":"88e7d729-60a5-46d5-b177-fe3217157f61","Type":"ContainerDied","Data":"2c2896c2e486f535d213c2130736fc2f8a769fba7e3227fab7e0f70c47c7aec8"}
Oct 30 01:30:13.449957 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.449918    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c2896c2e486f535d213c2130736fc2f8a769fba7e3227fab7e0f70c47c7aec8"
Oct 30 01:30:13.452858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.452835    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:30:13.474481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.474453    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"]
Oct 30 01:30:13.494163 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.494083    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e7d729-60a5-46d5-b177-fe3217157f61-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88e7d729-60a5-46d5-b177-fe3217157f61-routing-overlay-volume\") pod \"88e7d729-60a5-46d5-b177-fe3217157f61\" (UID: \"88e7d729-60a5-46d5-b177-fe3217157f61\") "
Oct 30 01:30:13.494323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.494184    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e7d729-60a5-46d5-b177-fe3217157f61-provider-credentials\" (UniqueName: \"kubernetes.io/projected/88e7d729-60a5-46d5-b177-fe3217157f61-provider-credentials\") pod \"88e7d729-60a5-46d5-b177-fe3217157f61\" (UID: \"88e7d729-60a5-46d5-b177-fe3217157f61\") "
Oct 30 01:30:13.494323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.494298    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e7d729-60a5-46d5-b177-fe3217157f61-kube-api-access-vl7g9\" (UniqueName: \"kubernetes.io/projected/88e7d729-60a5-46d5-b177-fe3217157f61-kube-api-access-vl7g9\") pod \"88e7d729-60a5-46d5-b177-fe3217157f61\" (UID: \"88e7d729-60a5-46d5-b177-fe3217157f61\") "
Oct 30 01:30:13.494504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.494485    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e7d729-60a5-46d5-b177-fe3217157f61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e7d729-60a5-46d5-b177-fe3217157f61-plugins-config-volume\") pod \"88e7d729-60a5-46d5-b177-fe3217157f61\" (UID: \"88e7d729-60a5-46d5-b177-fe3217157f61\") "
Oct 30 01:30:13.494624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.494554    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e7d729-60a5-46d5-b177-fe3217157f61-routing-overlay-volume" pod "88e7d729-60a5-46d5-b177-fe3217157f61" (UID: "88e7d729-60a5-46d5-b177-fe3217157f61"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:13.495013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.494971    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e7d729-60a5-46d5-b177-fe3217157f61-plugins-config-volume" pod "88e7d729-60a5-46d5-b177-fe3217157f61" (UID: "88e7d729-60a5-46d5-b177-fe3217157f61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:13.495130 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.495110    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88e7d729-60a5-46d5-b177-fe3217157f61-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:13.495225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.495130    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88e7d729-60a5-46d5-b177-fe3217157f61-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:13.496475 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.496454    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e7d729-60a5-46d5-b177-fe3217157f61-provider-credentials" pod "88e7d729-60a5-46d5-b177-fe3217157f61" (UID: "88e7d729-60a5-46d5-b177-fe3217157f61"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:13.497207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.497180    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e7d729-60a5-46d5-b177-fe3217157f61-kube-api-access-vl7g9" pod "88e7d729-60a5-46d5-b177-fe3217157f61" (UID: "88e7d729-60a5-46d5-b177-fe3217157f61"). InnerVolumeSpecName "kube-api-access-vl7g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:13.509808 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.509777    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"]
Oct 30 01:30:13.510946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.510921    4425 cpu_manager.go:414] "containerMap: removing container" podUID="88e7d729-60a5-46d5-b177-fe3217157f61" containerName="payload-processing"
Oct 30 01:30:13.511055 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.510947    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e7d729-60a5-46d5-b177-fe3217157f61" containerName="payload-processing"
Oct 30 01:30:13.511301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.511283    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e7d729-60a5-46d5-b177-fe3217157f61" containerName="payload-processing"
Oct 30 01:30:13.517284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.516098    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:13.532306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.532282    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-80acc4c49ef4\""
Oct 30 01:30:13.546843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.546815    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"]
Oct 30 01:30:13.596829 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.596743    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nth42\" (UniqueName: \"kubernetes.io/projected/f4defc5b-55df-4d40-ada5-a9463b99e7f3-kube-api-access-nth42\") pod \"payload-processing-external-model-6cdcb67694-5tpqt\" (UID: \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\") " pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:13.596829 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.596810    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4defc5b-55df-4d40-ada5-a9463b99e7f3-provider-credentials\") pod \"payload-processing-external-model-6cdcb67694-5tpqt\" (UID: \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\") " pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:13.597034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.596980    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4defc5b-55df-4d40-ada5-a9463b99e7f3-plugins-config-volume\") pod \"payload-processing-external-model-6cdcb67694-5tpqt\" (UID: \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\") " pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:13.597108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.597093    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4defc5b-55df-4d40-ada5-a9463b99e7f3-routing-overlay-volume\") pod \"payload-processing-external-model-6cdcb67694-5tpqt\" (UID: \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\") " pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:13.597300 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.597284    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88e7d729-60a5-46d5-b177-fe3217157f61-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:13.597361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.597302    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl7g9\" (UniqueName: \"kubernetes.io/projected/88e7d729-60a5-46d5-b177-fe3217157f61-kube-api-access-vl7g9\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:13.698739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.698702    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4defc5b-55df-4d40-ada5-a9463b99e7f3-provider-credentials\") pod \"payload-processing-external-model-6cdcb67694-5tpqt\" (UID: \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\") " pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:13.698916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.698805    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4defc5b-55df-4d40-ada5-a9463b99e7f3-plugins-config-volume\") pod \"payload-processing-external-model-6cdcb67694-5tpqt\" (UID: \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\") " pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:13.698916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.698871    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4defc5b-55df-4d40-ada5-a9463b99e7f3-routing-overlay-volume\") pod \"payload-processing-external-model-6cdcb67694-5tpqt\" (UID: \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\") " pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:13.699023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.698984    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nth42\" (UniqueName: \"kubernetes.io/projected/f4defc5b-55df-4d40-ada5-a9463b99e7f3-kube-api-access-nth42\") pod \"payload-processing-external-model-6cdcb67694-5tpqt\" (UID: \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\") " pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:13.699589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.699561    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4defc5b-55df-4d40-ada5-a9463b99e7f3-routing-overlay-volume\") pod \"payload-processing-external-model-6cdcb67694-5tpqt\" (UID: \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\") " pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:13.699723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.699614    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4defc5b-55df-4d40-ada5-a9463b99e7f3-plugins-config-volume\") pod \"payload-processing-external-model-6cdcb67694-5tpqt\" (UID: \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\") " pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:13.701313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.701297    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4defc5b-55df-4d40-ada5-a9463b99e7f3-provider-credentials\") pod \"payload-processing-external-model-6cdcb67694-5tpqt\" (UID: \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\") " pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:13.720214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.720145    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nth42\" (UniqueName: \"kubernetes.io/projected/f4defc5b-55df-4d40-ada5-a9463b99e7f3-kube-api-access-nth42\") pod \"payload-processing-external-model-6cdcb67694-5tpqt\" (UID: \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\") " pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:13.837393 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:13.837359    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:14.150883 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:14.150855    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4defc5b_55df_4d40_ada5_a9463b99e7f3.slice/crio-effd20b55e253caa7c1dfb7f90089f614f54e3dca004221289401e3c77548b6e WatchSource:0}: Error finding container effd20b55e253caa7c1dfb7f90089f614f54e3dca004221289401e3c77548b6e: Status 404 returned error can't find the container with id effd20b55e253caa7c1dfb7f90089f614f54e3dca004221289401e3c77548b6e
Oct 30 01:30:14.151943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:14.151915    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"]
Oct 30 01:30:14.460843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:14.460724    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt" event={"ID":"f4defc5b-55df-4d40-ada5-a9463b99e7f3","Type":"ContainerStarted","Data":"95175b8179b90efe27ee987e4c011c661e16f9cb1c0f791cc159387551c747bd"}
Oct 30 01:30:14.460843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:14.460783    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt" event={"ID":"f4defc5b-55df-4d40-ada5-a9463b99e7f3","Type":"ContainerStarted","Data":"effd20b55e253caa7c1dfb7f90089f614f54e3dca004221289401e3c77548b6e"}
Oct 30 01:30:14.461087 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:14.460971    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:14.462527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:14.462500    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth" event={"ID":"600ad629-faf2-49eb-9d14-05f9cf294799","Type":"ContainerStarted","Data":"bacb03000ea169fafb82cad098399326bba0f6f91e2780d60f05d387ca678b9f"}
Oct 30 01:30:14.462658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:14.462514    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"
Oct 30 01:30:14.462658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:14.462637    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"
Oct 30 01:30:14.544589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:14.544538    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt" podStartSLOduration=1.544523623 podStartE2EDuration="1.544523623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:14.540350022 +0000 UTC m=+2923.290005331" watchObservedRunningTime="2026-10-30 01:30:14.544523623 +0000 UTC m=+2923.294178931"
Oct 30 01:30:14.584684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:14.584654    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"]
Oct 30 01:30:14.588971 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:14.588944    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-574d87dd89-s7xlz"]
Oct 30 01:30:14.624236 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:14.624187    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth" podStartSLOduration=2.624172894 podStartE2EDuration="2.624172894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:14.620564656 +0000 UTC m=+2923.370219965" watchObservedRunningTime="2026-10-30 01:30:14.624172894 +0000 UTC m=+2923.373828197"
Oct 30 01:30:15.415788 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:15.415756    4425 scope.go:122] "RemoveContainer" containerID="0d593bddbc8ac25ad06dc5a7ec54f8ce6f0a7fb5364759284c045766ccffc782"
Oct 30 01:30:15.416279 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:15.415957    4425 scope.go:122] "RemoveContainer" containerID="2d2a45e2c9ccbda0e78795cddf8b4146601c4e76f3ac7fe9a8f21ca955ebf1c4"
Oct 30 01:30:15.416357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:15.416307    4425 scope.go:122] "RemoveContainer" containerID="889da4fb98cc1af7fab06ccf8f18bb3ecbe4c372d0f9d020799e7917212d2b14"
Oct 30 01:30:15.416613 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:15.416514    4425 scope.go:122] "RemoveContainer" containerID="9c876ac4b5d789d9c20f9556061198428a82cf841c112d4cb3fe5f15895d9f28"
Oct 30 01:30:15.427989 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:15.427968    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e7d729-60a5-46d5-b177-fe3217157f61" path="/var/lib/kubelet/pods/88e7d729-60a5-46d5-b177-fe3217157f61/volumes"
Oct 30 01:30:16.414934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:16.414909    4425 scope.go:122] "RemoveContainer" containerID="9bcf92783744881dd1b077efe90ec0987b3b0045367db5ecdb0e6b45202808c3"
Oct 30 01:30:16.415066 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:16.415040    4425 scope.go:122] "RemoveContainer" containerID="29cb85b75507f7ba894dd3b2328871ccb27a00f28b1cb6e195ae2552ad4a31ec"
Oct 30 01:30:16.415193 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:16.415170    4425 scope.go:122] "RemoveContainer" containerID="373c11c77b2d93326c8f31f5042f56f9f0cf463efa70e8c0e0cf5464e4f63acc"
Oct 30 01:30:16.490345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:16.490318    4425 generic.go:356] "Generic (PLEG): container finished" podID="fba425f6-60cc-44b1-89da-533a37750a41" containerID="5212d56f243696175a90852b082f2a7cf2e2adaf0e66ce8d8149b1439ec042a6" exitCode=60
Oct 30 01:30:16.490746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:16.490391    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c" event={"ID":"fba425f6-60cc-44b1-89da-533a37750a41","Type":"ContainerDied","Data":"5212d56f243696175a90852b082f2a7cf2e2adaf0e66ce8d8149b1439ec042a6"}
Oct 30 01:30:16.490746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:16.490469    4425 scope.go:122] "RemoveContainer" containerID="889da4fb98cc1af7fab06ccf8f18bb3ecbe4c372d0f9d020799e7917212d2b14"
Oct 30 01:30:16.490915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:16.490897    4425 scope.go:122] "RemoveContainer" containerID="5212d56f243696175a90852b082f2a7cf2e2adaf0e66ce8d8149b1439ec042a6"
Oct 30 01:30:16.491327 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:16.491298    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c_odh-ai-gateway-infra(fba425f6-60cc-44b1-89da-533a37750a41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c" podUID="fba425f6-60cc-44b1-89da-533a37750a41"
Oct 30 01:30:16.492533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:16.492513    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8" event={"ID":"61c894e6-ae30-4978-850a-dfbde9fd69bd","Type":"ContainerStarted","Data":"7878acfe1ffcf4d2bf61f5cdea21df4d1a7a3f179127278fafedaabcf121a8f6"}
Oct 30 01:30:16.494422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:16.494398    4425 generic.go:356] "Generic (PLEG): container finished" podID="dd86e619-32bb-4d11-afa2-d4a9d723d886" containerID="3314b0b657075c00b30d3143d16801eeb8955bbb592eb836dd3cfb943f50fe38" exitCode=60
Oct 30 01:30:16.494521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:16.494463    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk" event={"ID":"dd86e619-32bb-4d11-afa2-d4a9d723d886","Type":"ContainerDied","Data":"3314b0b657075c00b30d3143d16801eeb8955bbb592eb836dd3cfb943f50fe38"}
Oct 30 01:30:16.494859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:16.494841    4425 scope.go:122] "RemoveContainer" containerID="3314b0b657075c00b30d3143d16801eeb8955bbb592eb836dd3cfb943f50fe38"
Oct 30 01:30:16.495190 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:16.495142    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk_odh-ai-gateway-infra(dd86e619-32bb-4d11-afa2-d4a9d723d886)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk" podUID="dd86e619-32bb-4d11-afa2-d4a9d723d886"
Oct 30 01:30:16.496293 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:16.496269    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9" event={"ID":"53190760-7148-47da-93f0-1123e5df62e2","Type":"ContainerStarted","Data":"28391365b9408a82ed946832350646ba6854a0af55b4a803ede933bf14e9d64b"}
Oct 30 01:30:16.535665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:16.535638    4425 scope.go:122] "RemoveContainer" containerID="0d593bddbc8ac25ad06dc5a7ec54f8ce6f0a7fb5364759284c045766ccffc782"
Oct 30 01:30:17.311366 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.311284    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"]
Oct 30 01:30:17.316502 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.316481    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"
Oct 30 01:30:17.337261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.337233    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"]
Oct 30 01:30:17.356196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.356131    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c880cf0-b76c-4db8-a26c-d7af8345a80b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk\" (UID: \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"
Oct 30 01:30:17.356374 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.356266    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c880cf0-b76c-4db8-a26c-d7af8345a80b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk\" (UID: \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"
Oct 30 01:30:17.356468 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.356453    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c880cf0-b76c-4db8-a26c-d7af8345a80b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk\" (UID: \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"
Oct 30 01:30:17.356527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.356475    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzfw7\" (UniqueName: \"kubernetes.io/projected/1c880cf0-b76c-4db8-a26c-d7af8345a80b-kube-api-access-dzfw7\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk\" (UID: \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"
Oct 30 01:30:17.415821 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.415479    4425 scope.go:122] "RemoveContainer" containerID="e182a4d3c1ed0114812a757789bf511716fe95038c3eca11ddd9a07c03509ab7"
Oct 30 01:30:17.415821 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.415719    4425 scope.go:122] "RemoveContainer" containerID="dd48f8d1d1a5954756fdefa022668918f477763902de74f0a526cc6d2fa53ee1"
Oct 30 01:30:17.460235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.459680    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c880cf0-b76c-4db8-a26c-d7af8345a80b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk\" (UID: \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"
Oct 30 01:30:17.460235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.459727    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzfw7\" (UniqueName: \"kubernetes.io/projected/1c880cf0-b76c-4db8-a26c-d7af8345a80b-kube-api-access-dzfw7\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk\" (UID: \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"
Oct 30 01:30:17.460235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.459866    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c880cf0-b76c-4db8-a26c-d7af8345a80b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk\" (UID: \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"
Oct 30 01:30:17.460235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.459994    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c880cf0-b76c-4db8-a26c-d7af8345a80b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk\" (UID: \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"
Oct 30 01:30:17.461052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.460819    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c880cf0-b76c-4db8-a26c-d7af8345a80b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk\" (UID: \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"
Oct 30 01:30:17.462118 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.462066    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c880cf0-b76c-4db8-a26c-d7af8345a80b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk\" (UID: \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"
Oct 30 01:30:17.465724 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.465668    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c880cf0-b76c-4db8-a26c-d7af8345a80b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk\" (UID: \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"
Oct 30 01:30:17.482899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.482863    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzfw7\" (UniqueName: \"kubernetes.io/projected/1c880cf0-b76c-4db8-a26c-d7af8345a80b-kube-api-access-dzfw7\") pod \"payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk\" (UID: \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"
Oct 30 01:30:17.512280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.512237    4425 generic.go:356] "Generic (PLEG): container finished" podID="53190760-7148-47da-93f0-1123e5df62e2" containerID="28391365b9408a82ed946832350646ba6854a0af55b4a803ede933bf14e9d64b" exitCode=60
Oct 30 01:30:17.512747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.512323    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9" event={"ID":"53190760-7148-47da-93f0-1123e5df62e2","Type":"ContainerDied","Data":"28391365b9408a82ed946832350646ba6854a0af55b4a803ede933bf14e9d64b"}
Oct 30 01:30:17.512747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.512394    4425 scope.go:122] "RemoveContainer" containerID="9c876ac4b5d789d9c20f9556061198428a82cf841c112d4cb3fe5f15895d9f28"
Oct 30 01:30:17.512889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.512795    4425 scope.go:122] "RemoveContainer" containerID="28391365b9408a82ed946832350646ba6854a0af55b4a803ede933bf14e9d64b"
Oct 30 01:30:17.513275 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:17.513238    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9_odh-ai-gateway-infra(53190760-7148-47da-93f0-1123e5df62e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9" podUID="53190760-7148-47da-93f0-1123e5df62e2"
Oct 30 01:30:17.515783 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.515758    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt" event={"ID":"107404d2-96de-452e-8e7d-24acb8246db3","Type":"ContainerStarted","Data":"63d00090671b6000073e92975ec31f81f64678392a3dfa2cc28410c59194407c"}
Oct 30 01:30:17.517897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.517872    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n" event={"ID":"b3d49b88-2ba6-4ded-a160-806e4e8fed2b","Type":"ContainerStarted","Data":"0edcf5f31617eb25343a3eafd6805369a2780b6e8d2650bed857e4e2a4d9f737"}
Oct 30 01:30:17.520590 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.520566    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb" event={"ID":"4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1","Type":"ContainerStarted","Data":"24824f4e3f5aef315a82ebb26fecb92a6850ef4c3f8ba16f24a7e85f6c63ef79"}
Oct 30 01:30:17.526229 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.526206    4425 generic.go:356] "Generic (PLEG): container finished" podID="61c894e6-ae30-4978-850a-dfbde9fd69bd" containerID="7878acfe1ffcf4d2bf61f5cdea21df4d1a7a3f179127278fafedaabcf121a8f6" exitCode=60
Oct 30 01:30:17.526326 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.526277    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8" event={"ID":"61c894e6-ae30-4978-850a-dfbde9fd69bd","Type":"ContainerDied","Data":"7878acfe1ffcf4d2bf61f5cdea21df4d1a7a3f179127278fafedaabcf121a8f6"}
Oct 30 01:30:17.526665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.526647    4425 scope.go:122] "RemoveContainer" containerID="7878acfe1ffcf4d2bf61f5cdea21df4d1a7a3f179127278fafedaabcf121a8f6"
Oct 30 01:30:17.526955 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:17.526926    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8_odh-ai-gateway-infra(61c894e6-ae30-4978-850a-dfbde9fd69bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8" podUID="61c894e6-ae30-4978-850a-dfbde9fd69bd"
Oct 30 01:30:17.531013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.530996    4425 scope.go:122] "RemoveContainer" containerID="2d2a45e2c9ccbda0e78795cddf8b4146601c4e76f3ac7fe9a8f21ca955ebf1c4"
Oct 30 01:30:17.540022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.540000    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c"]
Oct 30 01:30:17.577987 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.577944    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8"]
Oct 30 01:30:17.608460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.608425    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk"]
Oct 30 01:30:17.631713 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.631677    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"
Oct 30 01:30:17.648418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.648391    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9"]
Oct 30 01:30:17.921753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.921721    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c"
Oct 30 01:30:17.925230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.925204    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk"
Oct 30 01:30:17.999270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.999064    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba425f6-60cc-44b1-89da-533a37750a41-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fba425f6-60cc-44b1-89da-533a37750a41-maas-api-service-ca\") pod \"fba425f6-60cc-44b1-89da-533a37750a41\" (UID: \"fba425f6-60cc-44b1-89da-533a37750a41\") "
Oct 30 01:30:17.999270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:17.999190    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd86e619-32bb-4d11-afa2-d4a9d723d886-kube-api-access-vqcgt\" (UniqueName: \"kubernetes.io/projected/dd86e619-32bb-4d11-afa2-d4a9d723d886-kube-api-access-vqcgt\") pod \"dd86e619-32bb-4d11-afa2-d4a9d723d886\" (UID: \"dd86e619-32bb-4d11-afa2-d4a9d723d886\") "
Oct 30 01:30:18.001294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.001229    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd86e619-32bb-4d11-afa2-d4a9d723d886-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd86e619-32bb-4d11-afa2-d4a9d723d886-maas-api-service-ca\") pod \"dd86e619-32bb-4d11-afa2-d4a9d723d886\" (UID: \"dd86e619-32bb-4d11-afa2-d4a9d723d886\") "
Oct 30 01:30:18.001934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.001754    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba425f6-60cc-44b1-89da-533a37750a41-maas-api-service-ca" pod "fba425f6-60cc-44b1-89da-533a37750a41" (UID: "fba425f6-60cc-44b1-89da-533a37750a41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:18.004707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.002188    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba425f6-60cc-44b1-89da-533a37750a41-kube-api-access-4pbcv\" (UniqueName: \"kubernetes.io/projected/fba425f6-60cc-44b1-89da-533a37750a41-kube-api-access-4pbcv\") pod \"fba425f6-60cc-44b1-89da-533a37750a41\" (UID: \"fba425f6-60cc-44b1-89da-533a37750a41\") "
Oct 30 01:30:18.005704 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.005406    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fba425f6-60cc-44b1-89da-533a37750a41-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:18.005901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.005865    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd86e619-32bb-4d11-afa2-d4a9d723d886-maas-api-service-ca" pod "dd86e619-32bb-4d11-afa2-d4a9d723d886" (UID: "dd86e619-32bb-4d11-afa2-d4a9d723d886"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:18.008990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.008957    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba425f6-60cc-44b1-89da-533a37750a41-kube-api-access-4pbcv" pod "fba425f6-60cc-44b1-89da-533a37750a41" (UID: "fba425f6-60cc-44b1-89da-533a37750a41"). InnerVolumeSpecName "kube-api-access-4pbcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:18.009276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.009254    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd86e619-32bb-4d11-afa2-d4a9d723d886-kube-api-access-vqcgt" pod "dd86e619-32bb-4d11-afa2-d4a9d723d886" (UID: "dd86e619-32bb-4d11-afa2-d4a9d723d886"). InnerVolumeSpecName "kube-api-access-vqcgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:18.066201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.064789    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"]
Oct 30 01:30:18.068382 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:18.068353    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c880cf0_b76c_4db8_a26c_d7af8345a80b.slice/crio-0995f74bc6a6013b8aaa01da76437258f60ba1454d05c9f5783efb5786713476 WatchSource:0}: Error finding container 0995f74bc6a6013b8aaa01da76437258f60ba1454d05c9f5783efb5786713476: Status 404 returned error can't find the container with id 0995f74bc6a6013b8aaa01da76437258f60ba1454d05c9f5783efb5786713476
Oct 30 01:30:18.094242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.094170    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"]
Oct 30 01:30:18.101765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.101739    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"]
Oct 30 01:30:18.108417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.108388    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqcgt\" (UniqueName: \"kubernetes.io/projected/dd86e619-32bb-4d11-afa2-d4a9d723d886-kube-api-access-vqcgt\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:18.108547 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.108419    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd86e619-32bb-4d11-afa2-d4a9d723d886-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:18.108547 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.108437    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pbcv\" (UniqueName: \"kubernetes.io/projected/fba425f6-60cc-44b1-89da-533a37750a41-kube-api-access-4pbcv\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:18.415859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.415834    4425 scope.go:122] "RemoveContainer" containerID="932207fe531f80a91a1eb16d5ddb9ad053e1ed8b1c909a69aaf747b9d7c29c92"
Oct 30 01:30:18.416032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.415970    4425 scope.go:122] "RemoveContainer" containerID="adc634652b6cd08572e5f16c16dc0b399a89ebab5379b6d2a748181a937d908e"
Oct 30 01:30:18.416090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.416046    4425 scope.go:122] "RemoveContainer" containerID="91f3efcc9801b8778d1f3914497097e45bbe5181097701b9de0c7e4330656613"
Oct 30 01:30:18.545630 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.545601    4425 generic.go:356] "Generic (PLEG): container finished" podID="4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1" containerID="24824f4e3f5aef315a82ebb26fecb92a6850ef4c3f8ba16f24a7e85f6c63ef79" exitCode=60
Oct 30 01:30:18.546054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.545650    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb" event={"ID":"4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1","Type":"ContainerDied","Data":"24824f4e3f5aef315a82ebb26fecb92a6850ef4c3f8ba16f24a7e85f6c63ef79"}
Oct 30 01:30:18.546054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.545728    4425 scope.go:122] "RemoveContainer" containerID="373c11c77b2d93326c8f31f5042f56f9f0cf463efa70e8c0e0cf5464e4f63acc"
Oct 30 01:30:18.546225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.546134    4425 scope.go:122] "RemoveContainer" containerID="24824f4e3f5aef315a82ebb26fecb92a6850ef4c3f8ba16f24a7e85f6c63ef79"
Oct 30 01:30:18.546519 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:18.546498    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb_odh-ai-gateway-infra(4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb" podUID="4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1"
Oct 30 01:30:18.548037 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.548017    4425 generic.go:356] "Generic (PLEG): container finished" podID="47e2b0e7-6204-402f-a79c-fa8ff22b3a70" containerID="2f998c6b72312c77ea8468569be36386c97909988e851fb45773e235e8e9499a" exitCode=60
Oct 30 01:30:18.548116 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.548086    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct" event={"ID":"47e2b0e7-6204-402f-a79c-fa8ff22b3a70","Type":"ContainerDied","Data":"2f998c6b72312c77ea8468569be36386c97909988e851fb45773e235e8e9499a"}
Oct 30 01:30:18.548518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.548500    4425 scope.go:122] "RemoveContainer" containerID="2f998c6b72312c77ea8468569be36386c97909988e851fb45773e235e8e9499a"
Oct 30 01:30:18.548792 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:18.548772    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct_odh-ai-gateway-infra(47e2b0e7-6204-402f-a79c-fa8ff22b3a70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct" podUID="47e2b0e7-6204-402f-a79c-fa8ff22b3a70"
Oct 30 01:30:18.549676 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.549655    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c" event={"ID":"fba425f6-60cc-44b1-89da-533a37750a41","Type":"ContainerDied","Data":"e6ad8c6a7e2ceeea889f7b52750515da740a5db520720a14bcea37b2192f69c1"}
Oct 30 01:30:18.549797 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.549682    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c"
Oct 30 01:30:18.556648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.556629    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk"
Oct 30 01:30:18.556648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.556640    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk" event={"ID":"dd86e619-32bb-4d11-afa2-d4a9d723d886","Type":"ContainerDied","Data":"5df696c4968768ecd40df569455d149988d558e7134a12650c46752ebca1ed8d"}
Oct 30 01:30:18.558748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.558720    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk" event={"ID":"1c880cf0-b76c-4db8-a26c-d7af8345a80b","Type":"ContainerStarted","Data":"d853f8ce036fe9f6f27a9b6e214f6dbe8cd4281027434b2ad41667388d56f055"}
Oct 30 01:30:18.558868 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.558765    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk" event={"ID":"1c880cf0-b76c-4db8-a26c-d7af8345a80b","Type":"ContainerStarted","Data":"0995f74bc6a6013b8aaa01da76437258f60ba1454d05c9f5783efb5786713476"}
Oct 30 01:30:18.558868 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.558795    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"
Oct 30 01:30:18.564895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.564879    4425 scope.go:122] "RemoveContainer" containerID="e182a4d3c1ed0114812a757789bf511716fe95038c3eca11ddd9a07c03509ab7"
Oct 30 01:30:18.565381 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.565362    4425 generic.go:356] "Generic (PLEG): container finished" podID="107404d2-96de-452e-8e7d-24acb8246db3" containerID="63d00090671b6000073e92975ec31f81f64678392a3dfa2cc28410c59194407c" exitCode=60
Oct 30 01:30:18.565535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.565426    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt" event={"ID":"107404d2-96de-452e-8e7d-24acb8246db3","Type":"ContainerDied","Data":"63d00090671b6000073e92975ec31f81f64678392a3dfa2cc28410c59194407c"}
Oct 30 01:30:18.565822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.565803    4425 scope.go:122] "RemoveContainer" containerID="63d00090671b6000073e92975ec31f81f64678392a3dfa2cc28410c59194407c"
Oct 30 01:30:18.566046 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:18.566027    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt_odh-ai-gateway-infra(107404d2-96de-452e-8e7d-24acb8246db3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt" podUID="107404d2-96de-452e-8e7d-24acb8246db3"
Oct 30 01:30:18.567867 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.567701    4425 generic.go:356] "Generic (PLEG): container finished" podID="b3d49b88-2ba6-4ded-a160-806e4e8fed2b" containerID="0edcf5f31617eb25343a3eafd6805369a2780b6e8d2650bed857e4e2a4d9f737" exitCode=60
Oct 30 01:30:18.567867 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.567766    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n" event={"ID":"b3d49b88-2ba6-4ded-a160-806e4e8fed2b","Type":"ContainerDied","Data":"0edcf5f31617eb25343a3eafd6805369a2780b6e8d2650bed857e4e2a4d9f737"}
Oct 30 01:30:18.568149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.568118    4425 scope.go:122] "RemoveContainer" containerID="0edcf5f31617eb25343a3eafd6805369a2780b6e8d2650bed857e4e2a4d9f737"
Oct 30 01:30:18.568664 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:18.568409    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n_odh-ai-gateway-infra(b3d49b88-2ba6-4ded-a160-806e4e8fed2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n" podUID="b3d49b88-2ba6-4ded-a160-806e4e8fed2b"
Oct 30 01:30:18.569901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.569848    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm" event={"ID":"03eaf7ac-cf67-4da9-b253-3c1994ad57fa","Type":"ContainerStarted","Data":"85dbbd0ea2f2da3c48712f1dbbd291131cd0db36d6bb16443a827efca0ba8a12"}
Oct 30 01:30:18.581739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.581712    4425 scope.go:122] "RemoveContainer" containerID="5212d56f243696175a90852b082f2a7cf2e2adaf0e66ce8d8149b1439ec042a6"
Oct 30 01:30:18.591990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.591972    4425 scope.go:122] "RemoveContainer" containerID="3314b0b657075c00b30d3143d16801eeb8955bbb592eb836dd3cfb943f50fe38"
Oct 30 01:30:18.604035 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.603995    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n"]
Oct 30 01:30:18.608858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.608839    4425 scope.go:122] "RemoveContainer" containerID="9bcf92783744881dd1b077efe90ec0987b3b0045367db5ecdb0e6b45202808c3"
Oct 30 01:30:18.624472 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.624455    4425 scope.go:122] "RemoveContainer" containerID="29cb85b75507f7ba894dd3b2328871ccb27a00f28b1cb6e195ae2552ad4a31ec"
Oct 30 01:30:18.651336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.651314    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk"]
Oct 30 01:30:18.656872 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.656822    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-94ea1e-29888730-wtbdk"]
Oct 30 01:30:18.675197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.672196    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb"]
Oct 30 01:30:18.692680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.692629    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk" podStartSLOduration=1.692613547 podStartE2EDuration="1.692613547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:18.687618163 +0000 UTC m=+2927.437273472" watchObservedRunningTime="2026-10-30 01:30:18.692613547 +0000 UTC m=+2927.442268856"
Oct 30 01:30:18.718987 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.717268    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt"]
Oct 30 01:30:18.836244 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.836216    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c"]
Oct 30 01:30:18.840813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.840783    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c48628-29888730-l7g8c"]
Oct 30 01:30:18.892268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.891221    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9"
Oct 30 01:30:18.895860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:18.895837    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8"
Oct 30 01:30:19.028989 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.028949    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53190760-7148-47da-93f0-1123e5df62e2-kube-api-access-n24ff\" (UniqueName: \"kubernetes.io/projected/53190760-7148-47da-93f0-1123e5df62e2-kube-api-access-n24ff\") pod \"53190760-7148-47da-93f0-1123e5df62e2\" (UID: \"53190760-7148-47da-93f0-1123e5df62e2\") "
Oct 30 01:30:19.029228 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.029210    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53190760-7148-47da-93f0-1123e5df62e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53190760-7148-47da-93f0-1123e5df62e2-maas-api-service-ca\") pod \"53190760-7148-47da-93f0-1123e5df62e2\" (UID: \"53190760-7148-47da-93f0-1123e5df62e2\") "
Oct 30 01:30:19.029315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.029268    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c894e6-ae30-4978-850a-dfbde9fd69bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c894e6-ae30-4978-850a-dfbde9fd69bd-maas-api-service-ca\") pod \"61c894e6-ae30-4978-850a-dfbde9fd69bd\" (UID: \"61c894e6-ae30-4978-850a-dfbde9fd69bd\") "
Oct 30 01:30:19.029315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.029305    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c894e6-ae30-4978-850a-dfbde9fd69bd-kube-api-access-tx89m\" (UniqueName: \"kubernetes.io/projected/61c894e6-ae30-4978-850a-dfbde9fd69bd-kube-api-access-tx89m\") pod \"61c894e6-ae30-4978-850a-dfbde9fd69bd\" (UID: \"61c894e6-ae30-4978-850a-dfbde9fd69bd\") "
Oct 30 01:30:19.030121 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.029938    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c894e6-ae30-4978-850a-dfbde9fd69bd-maas-api-service-ca" pod "61c894e6-ae30-4978-850a-dfbde9fd69bd" (UID: "61c894e6-ae30-4978-850a-dfbde9fd69bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:19.030121 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.030073    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53190760-7148-47da-93f0-1123e5df62e2-maas-api-service-ca" pod "53190760-7148-47da-93f0-1123e5df62e2" (UID: "53190760-7148-47da-93f0-1123e5df62e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:19.030772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.030750    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53190760-7148-47da-93f0-1123e5df62e2-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:19.030870 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.030778    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c894e6-ae30-4978-850a-dfbde9fd69bd-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:19.037354 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.037323    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c894e6-ae30-4978-850a-dfbde9fd69bd-kube-api-access-tx89m" pod "61c894e6-ae30-4978-850a-dfbde9fd69bd" (UID: "61c894e6-ae30-4978-850a-dfbde9fd69bd"). InnerVolumeSpecName "kube-api-access-tx89m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:19.059311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.059276    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53190760-7148-47da-93f0-1123e5df62e2-kube-api-access-n24ff" pod "53190760-7148-47da-93f0-1123e5df62e2" (UID: "53190760-7148-47da-93f0-1123e5df62e2"). InnerVolumeSpecName "kube-api-access-n24ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:19.131798 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.131713    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx89m\" (UniqueName: \"kubernetes.io/projected/61c894e6-ae30-4978-850a-dfbde9fd69bd-kube-api-access-tx89m\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:19.131798 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.131749    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n24ff\" (UniqueName: \"kubernetes.io/projected/53190760-7148-47da-93f0-1123e5df62e2-kube-api-access-n24ff\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:19.415902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.415819    4425 scope.go:122] "RemoveContainer" containerID="740018d6eafa31dd0d56755582c3803ecce1280a766837af781db9d49f2240b9"
Oct 30 01:30:19.416077 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.415941    4425 scope.go:122] "RemoveContainer" containerID="00a54dc39418f2348243269bad8b3de89caea2154e18d1f83d351138176c7b11"
Oct 30 01:30:19.427391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.427366    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd86e619-32bb-4d11-afa2-d4a9d723d886" path="/var/lib/kubelet/pods/dd86e619-32bb-4d11-afa2-d4a9d723d886/volumes"
Oct 30 01:30:19.428063 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.428040    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba425f6-60cc-44b1-89da-533a37750a41" path="/var/lib/kubelet/pods/fba425f6-60cc-44b1-89da-533a37750a41/volumes"
Oct 30 01:30:19.482619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.482593    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"]
Oct 30 01:30:19.485712 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.485695    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:19.526440 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.526411    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"]
Oct 30 01:30:19.527480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527457    4425 cpu_manager.go:414] "containerMap: removing container" podUID="fba425f6-60cc-44b1-89da-533a37750a41" containerName="cleanup"
Oct 30 01:30:19.527480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527478    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba425f6-60cc-44b1-89da-533a37750a41" containerName="cleanup"
Oct 30 01:30:19.527642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527489    4425 cpu_manager.go:414] "containerMap: removing container" podUID="53190760-7148-47da-93f0-1123e5df62e2" containerName="cleanup"
Oct 30 01:30:19.527642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527495    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53190760-7148-47da-93f0-1123e5df62e2" containerName="cleanup"
Oct 30 01:30:19.527642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527502    4425 cpu_manager.go:414] "containerMap: removing container" podUID="53190760-7148-47da-93f0-1123e5df62e2" containerName="cleanup"
Oct 30 01:30:19.527642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527508    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53190760-7148-47da-93f0-1123e5df62e2" containerName="cleanup"
Oct 30 01:30:19.527642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527528    4425 cpu_manager.go:414] "containerMap: removing container" podUID="61c894e6-ae30-4978-850a-dfbde9fd69bd" containerName="cleanup"
Oct 30 01:30:19.527642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527533    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c894e6-ae30-4978-850a-dfbde9fd69bd" containerName="cleanup"
Oct 30 01:30:19.527642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527552    4425 cpu_manager.go:414] "containerMap: removing container" podUID="dd86e619-32bb-4d11-afa2-d4a9d723d886" containerName="cleanup"
Oct 30 01:30:19.527642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527556    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd86e619-32bb-4d11-afa2-d4a9d723d886" containerName="cleanup"
Oct 30 01:30:19.527642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527569    4425 cpu_manager.go:414] "containerMap: removing container" podUID="fba425f6-60cc-44b1-89da-533a37750a41" containerName="cleanup"
Oct 30 01:30:19.527642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527574    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba425f6-60cc-44b1-89da-533a37750a41" containerName="cleanup"
Oct 30 01:30:19.527642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527583    4425 cpu_manager.go:414] "containerMap: removing container" podUID="dd86e619-32bb-4d11-afa2-d4a9d723d886" containerName="cleanup"
Oct 30 01:30:19.527642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527588    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd86e619-32bb-4d11-afa2-d4a9d723d886" containerName="cleanup"
Oct 30 01:30:19.527642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527600    4425 cpu_manager.go:414] "containerMap: removing container" podUID="dd86e619-32bb-4d11-afa2-d4a9d723d886" containerName="cleanup"
Oct 30 01:30:19.527642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527605    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd86e619-32bb-4d11-afa2-d4a9d723d886" containerName="cleanup"
Oct 30 01:30:19.527642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527631    4425 cpu_manager.go:414] "containerMap: removing container" podUID="61c894e6-ae30-4978-850a-dfbde9fd69bd" containerName="cleanup"
Oct 30 01:30:19.527642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527636    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c894e6-ae30-4978-850a-dfbde9fd69bd" containerName="cleanup"
Oct 30 01:30:19.528136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527759    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd86e619-32bb-4d11-afa2-d4a9d723d886" containerName="cleanup"
Oct 30 01:30:19.528136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527771    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="53190760-7148-47da-93f0-1123e5df62e2" containerName="cleanup"
Oct 30 01:30:19.528136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527777    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c894e6-ae30-4978-850a-dfbde9fd69bd" containerName="cleanup"
Oct 30 01:30:19.528136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527785    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c894e6-ae30-4978-850a-dfbde9fd69bd" containerName="cleanup"
Oct 30 01:30:19.528136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527793    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba425f6-60cc-44b1-89da-533a37750a41" containerName="cleanup"
Oct 30 01:30:19.528136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527798    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="53190760-7148-47da-93f0-1123e5df62e2" containerName="cleanup"
Oct 30 01:30:19.528136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527803    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="53190760-7148-47da-93f0-1123e5df62e2" containerName="cleanup"
Oct 30 01:30:19.528136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527810    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c894e6-ae30-4978-850a-dfbde9fd69bd" containerName="cleanup"
Oct 30 01:30:19.528136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527819    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba425f6-60cc-44b1-89da-533a37750a41" containerName="cleanup"
Oct 30 01:30:19.528136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527824    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba425f6-60cc-44b1-89da-533a37750a41" containerName="cleanup"
Oct 30 01:30:19.528136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.527829    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd86e619-32bb-4d11-afa2-d4a9d723d886" containerName="cleanup"
Oct 30 01:30:19.533692 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.533671    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:30:19.559610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.558371    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"]
Oct 30 01:30:19.586799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.586764    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862" event={"ID":"230424d1-69ed-40c3-8afd-660d905f9d8d","Type":"ContainerStarted","Data":"31e333f4a9155c56e1846a57664340adbe13d0f34b9dcd36b1f1a68b7ddffe5d"}
Oct 30 01:30:19.589071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.589044    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9" event={"ID":"c284f618-116f-426d-b25d-fde4ca3c686e","Type":"ContainerStarted","Data":"f4bf8322d1b022d4f932e06092a90ac8930e360729ffd81397bcd9fc7f410143"}
Oct 30 01:30:19.590657 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.590624    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9"
Oct 30 01:30:19.590791 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.590633    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9" event={"ID":"53190760-7148-47da-93f0-1123e5df62e2","Type":"ContainerDied","Data":"bda6af1d42d0155e1a224b240e554f074853d5e7021503b52e7bf6b1622a170e"}
Oct 30 01:30:19.590791 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.590758    4425 scope.go:122] "RemoveContainer" containerID="28391365b9408a82ed946832350646ba6854a0af55b4a803ede933bf14e9d64b"
Oct 30 01:30:19.601660 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.596406    4425 generic.go:356] "Generic (PLEG): container finished" podID="fd93d36a-1693-4cbb-bf97-204d5c295c0d" containerID="ed513ebffab0ed8ac33af071c0f0ef25c52147390009e97be1af928626e47ad6" exitCode=60
Oct 30 01:30:19.601660 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.596506    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx" event={"ID":"fd93d36a-1693-4cbb-bf97-204d5c295c0d","Type":"ContainerDied","Data":"ed513ebffab0ed8ac33af071c0f0ef25c52147390009e97be1af928626e47ad6"}
Oct 30 01:30:19.601660 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.596891    4425 scope.go:122] "RemoveContainer" containerID="ed513ebffab0ed8ac33af071c0f0ef25c52147390009e97be1af928626e47ad6"
Oct 30 01:30:19.601660 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:19.597235    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx_odh-ai-gateway-infra(fd93d36a-1693-4cbb-bf97-204d5c295c0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx" podUID="fd93d36a-1693-4cbb-bf97-204d5c295c0d"
Oct 30 01:30:19.601896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.601723    4425 generic.go:356] "Generic (PLEG): container finished" podID="03eaf7ac-cf67-4da9-b253-3c1994ad57fa" containerID="85dbbd0ea2f2da3c48712f1dbbd291131cd0db36d6bb16443a827efca0ba8a12" exitCode=60
Oct 30 01:30:19.601896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.601792    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm" event={"ID":"03eaf7ac-cf67-4da9-b253-3c1994ad57fa","Type":"ContainerDied","Data":"85dbbd0ea2f2da3c48712f1dbbd291131cd0db36d6bb16443a827efca0ba8a12"}
Oct 30 01:30:19.602607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.602588    4425 scope.go:122] "RemoveContainer" containerID="85dbbd0ea2f2da3c48712f1dbbd291131cd0db36d6bb16443a827efca0ba8a12"
Oct 30 01:30:19.602834 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:19.602813    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm_odh-ai-gateway-infra(03eaf7ac-cf67-4da9-b253-3c1994ad57fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm" podUID="03eaf7ac-cf67-4da9-b253-3c1994ad57fa"
Oct 30 01:30:19.609570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.609449    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8"
Oct 30 01:30:19.609570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.609461    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8" event={"ID":"61c894e6-ae30-4978-850a-dfbde9fd69bd","Type":"ContainerDied","Data":"045d1d8268ec42d9d36ad9e6cb92fe2f1722aad6e3877b28447a64c7ca72a2b7"}
Oct 30 01:30:19.639304 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.639219    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wrfw\" (UniqueName: \"kubernetes.io/projected/e2354e0a-cbfa-4fe1-8903-842575a2f208-kube-api-access-9wrfw\") pod \"payload-processing-external-model-d7c4fc775-8nkkc\" (UID: \"e2354e0a-cbfa-4fe1-8903-842575a2f208\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:30:19.639625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.639378    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2354e0a-cbfa-4fe1-8903-842575a2f208-routing-overlay-volume\") pod \"payload-processing-external-model-d7c4fc775-8nkkc\" (UID: \"e2354e0a-cbfa-4fe1-8903-842575a2f208\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:30:19.639625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.639423    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2354e0a-cbfa-4fe1-8903-842575a2f208-provider-credentials\") pod \"payload-processing-external-model-d7c4fc775-8nkkc\" (UID: \"e2354e0a-cbfa-4fe1-8903-842575a2f208\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:30:19.639625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.639482    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2354e0a-cbfa-4fe1-8903-842575a2f208-plugins-config-volume\") pod \"payload-processing-external-model-d7c4fc775-8nkkc\" (UID: \"e2354e0a-cbfa-4fe1-8903-842575a2f208\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:30:19.739601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.739562    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm"]
Oct 30 01:30:19.743666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.740575    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2354e0a-cbfa-4fe1-8903-842575a2f208-routing-overlay-volume\") pod \"payload-processing-external-model-d7c4fc775-8nkkc\" (UID: \"e2354e0a-cbfa-4fe1-8903-842575a2f208\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:30:19.743666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.740655    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2354e0a-cbfa-4fe1-8903-842575a2f208-provider-credentials\") pod \"payload-processing-external-model-d7c4fc775-8nkkc\" (UID: \"e2354e0a-cbfa-4fe1-8903-842575a2f208\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:30:19.743666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.740742    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2354e0a-cbfa-4fe1-8903-842575a2f208-plugins-config-volume\") pod \"payload-processing-external-model-d7c4fc775-8nkkc\" (UID: \"e2354e0a-cbfa-4fe1-8903-842575a2f208\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:30:19.743666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.740920    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wrfw\" (UniqueName: \"kubernetes.io/projected/e2354e0a-cbfa-4fe1-8903-842575a2f208-kube-api-access-9wrfw\") pod \"payload-processing-external-model-d7c4fc775-8nkkc\" (UID: \"e2354e0a-cbfa-4fe1-8903-842575a2f208\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:30:19.743666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.742908    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2354e0a-cbfa-4fe1-8903-842575a2f208-plugins-config-volume\") pod \"payload-processing-external-model-d7c4fc775-8nkkc\" (UID: \"e2354e0a-cbfa-4fe1-8903-842575a2f208\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:30:19.744808 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.744684    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2354e0a-cbfa-4fe1-8903-842575a2f208-routing-overlay-volume\") pod \"payload-processing-external-model-d7c4fc775-8nkkc\" (UID: \"e2354e0a-cbfa-4fe1-8903-842575a2f208\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:30:19.759591 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.759547    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2354e0a-cbfa-4fe1-8903-842575a2f208-provider-credentials\") pod \"payload-processing-external-model-d7c4fc775-8nkkc\" (UID: \"e2354e0a-cbfa-4fe1-8903-842575a2f208\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:30:19.768698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.768641    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wrfw\" (UniqueName: \"kubernetes.io/projected/e2354e0a-cbfa-4fe1-8903-842575a2f208-kube-api-access-9wrfw\") pod \"payload-processing-external-model-d7c4fc775-8nkkc\" (UID: \"e2354e0a-cbfa-4fe1-8903-842575a2f208\") " pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:30:19.841706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.839507    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"]
Oct 30 01:30:19.841706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.840953    4425 cpu_manager.go:414] "containerMap: removing container" podUID="53190760-7148-47da-93f0-1123e5df62e2" containerName="cleanup"
Oct 30 01:30:19.841706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.841010    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53190760-7148-47da-93f0-1123e5df62e2" containerName="cleanup"
Oct 30 01:30:19.841706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.841053    4425 cpu_manager.go:414] "containerMap: removing container" podUID="61c894e6-ae30-4978-850a-dfbde9fd69bd" containerName="cleanup"
Oct 30 01:30:19.841706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.841061    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c894e6-ae30-4978-850a-dfbde9fd69bd" containerName="cleanup"
Oct 30 01:30:19.841706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.841107    4425 cpu_manager.go:414] "containerMap: removing container" podUID="fba425f6-60cc-44b1-89da-533a37750a41" containerName="cleanup"
Oct 30 01:30:19.841706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.841115    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba425f6-60cc-44b1-89da-533a37750a41" containerName="cleanup"
Oct 30 01:30:19.841706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.841417    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd86e619-32bb-4d11-afa2-d4a9d723d886" containerName="cleanup"
Oct 30 01:30:19.849110 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.848772    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:19.849110 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.849011    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:30:19.870629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.870597    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"]
Oct 30 01:30:19.906541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.906491    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct"]
Oct 30 01:30:19.916054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.916034    4425 scope.go:122] "RemoveContainer" containerID="adc634652b6cd08572e5f16c16dc0b399a89ebab5379b6d2a748181a937d908e"
Oct 30 01:30:19.953314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.944777    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:19.953314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.944808    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02874d4b-fe79-4247-b420-369003cd1d23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:19.953314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.944866    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02874d4b-fe79-4247-b420-369003cd1d23-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:19.953314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.944890    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfjj\" (UniqueName: \"kubernetes.io/projected/02874d4b-fe79-4247-b420-369003cd1d23-kube-api-access-tlfjj\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:19.953314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.944937    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:19.953314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.944998    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02874d4b-fe79-4247-b420-369003cd1d23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:19.953314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.945030    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:19.953314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.945094    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:19.953314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.945113    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:19.970874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:19.970846    4425 scope.go:122] "RemoveContainer" containerID="dd48f8d1d1a5954756fdefa022668918f477763902de74f0a526cc6d2fa53ee1"
Oct 30 01:30:20.003455 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.002472    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb"
Oct 30 01:30:20.005791 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.005613    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n"
Oct 30 01:30:20.007824 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.007773    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt"
Oct 30 01:30:20.009549 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.009533    4425 scope.go:122] "RemoveContainer" containerID="7878acfe1ffcf4d2bf61f5cdea21df4d1a7a3f179127278fafedaabcf121a8f6"
Oct 30 01:30:20.046127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.046093    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.046312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.046239    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02874d4b-fe79-4247-b420-369003cd1d23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.046312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.046290    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.046447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.046383    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.046447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.046416    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.046565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.046474    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.046565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.046501    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02874d4b-fe79-4247-b420-369003cd1d23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.046675 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.046583    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02874d4b-fe79-4247-b420-369003cd1d23-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.046675 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.046621    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfjj\" (UniqueName: \"kubernetes.io/projected/02874d4b-fe79-4247-b420-369003cd1d23-kube-api-access-tlfjj\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.048039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.047607    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.048039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.047917    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.048259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.048223    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.048379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.048349    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.048483 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.048399    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02874d4b-fe79-4247-b420-369003cd1d23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.050567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.050325    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.051304 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.051281    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02874d4b-fe79-4247-b420-369003cd1d23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.063446 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.063344    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9"]
Oct 30 01:30:20.069665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.069639    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4107fc-29888730-6mpf9"]
Oct 30 01:30:20.070679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.070657    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02874d4b-fe79-4247-b420-369003cd1d23-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.072469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.072260    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfjj\" (UniqueName: \"kubernetes.io/projected/02874d4b-fe79-4247-b420-369003cd1d23-kube-api-access-tlfjj\") pod \"maas-default-gateway-openshift-default-fcb75c45f-rb6rw\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.147774 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.147739    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8"]
Oct 30 01:30:20.148237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.148208    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1-kube-api-access-xcp9m\" (UniqueName: \"kubernetes.io/projected/4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1-kube-api-access-xcp9m\") pod \"4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1\" (UID: \"4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1\") "
Oct 30 01:30:20.148381 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.148294    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d49b88-2ba6-4ded-a160-806e4e8fed2b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d49b88-2ba6-4ded-a160-806e4e8fed2b-maas-api-service-ca\") pod \"b3d49b88-2ba6-4ded-a160-806e4e8fed2b\" (UID: \"b3d49b88-2ba6-4ded-a160-806e4e8fed2b\") "
Oct 30 01:30:20.148450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.148377    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d49b88-2ba6-4ded-a160-806e4e8fed2b-kube-api-access-2bxz7\" (UniqueName: \"kubernetes.io/projected/b3d49b88-2ba6-4ded-a160-806e4e8fed2b-kube-api-access-2bxz7\") pod \"b3d49b88-2ba6-4ded-a160-806e4e8fed2b\" (UID: \"b3d49b88-2ba6-4ded-a160-806e4e8fed2b\") "
Oct 30 01:30:20.148450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.148432    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107404d2-96de-452e-8e7d-24acb8246db3-kube-api-access-2xpth\" (UniqueName: \"kubernetes.io/projected/107404d2-96de-452e-8e7d-24acb8246db3-kube-api-access-2xpth\") pod \"107404d2-96de-452e-8e7d-24acb8246db3\" (UID: \"107404d2-96de-452e-8e7d-24acb8246db3\") "
Oct 30 01:30:20.148575 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.148560    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107404d2-96de-452e-8e7d-24acb8246db3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/107404d2-96de-452e-8e7d-24acb8246db3-maas-api-service-ca\") pod \"107404d2-96de-452e-8e7d-24acb8246db3\" (UID: \"107404d2-96de-452e-8e7d-24acb8246db3\") "
Oct 30 01:30:20.148650 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.148609    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1-maas-api-service-ca\") pod \"4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1\" (UID: \"4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1\") "
Oct 30 01:30:20.148757 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.148735    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d49b88-2ba6-4ded-a160-806e4e8fed2b-maas-api-service-ca" pod "b3d49b88-2ba6-4ded-a160-806e4e8fed2b" (UID: "b3d49b88-2ba6-4ded-a160-806e4e8fed2b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:20.149345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.149319    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d49b88-2ba6-4ded-a160-806e4e8fed2b-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:20.150548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.150521    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107404d2-96de-452e-8e7d-24acb8246db3-maas-api-service-ca" pod "107404d2-96de-452e-8e7d-24acb8246db3" (UID: "107404d2-96de-452e-8e7d-24acb8246db3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:20.150940 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.150891    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1-maas-api-service-ca" pod "4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1" (UID: "4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:20.151180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.151129    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107404d2-96de-452e-8e7d-24acb8246db3-kube-api-access-2xpth" pod "107404d2-96de-452e-8e7d-24acb8246db3" (UID: "107404d2-96de-452e-8e7d-24acb8246db3"). InnerVolumeSpecName "kube-api-access-2xpth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:20.157399 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.157373    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1-kube-api-access-xcp9m" pod "4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1" (UID: "4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1"). InnerVolumeSpecName "kube-api-access-xcp9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:20.159205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.159184    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888730-zhnd8"]
Oct 30 01:30:20.163309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.163285    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d49b88-2ba6-4ded-a160-806e4e8fed2b-kube-api-access-2bxz7" pod "b3d49b88-2ba6-4ded-a160-806e4e8fed2b" (UID: "b3d49b88-2ba6-4ded-a160-806e4e8fed2b"). InnerVolumeSpecName "kube-api-access-2bxz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:20.167482 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.167463    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct"
Oct 30 01:30:20.217428 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.217378    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"]
Oct 30 01:30:20.252792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.252761    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e2b0e7-6204-402f-a79c-fa8ff22b3a70-kube-api-access-d58t9\" (UniqueName: \"kubernetes.io/projected/47e2b0e7-6204-402f-a79c-fa8ff22b3a70-kube-api-access-d58t9\") pod \"47e2b0e7-6204-402f-a79c-fa8ff22b3a70\" (UID: \"47e2b0e7-6204-402f-a79c-fa8ff22b3a70\") "
Oct 30 01:30:20.253578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.253551    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e2b0e7-6204-402f-a79c-fa8ff22b3a70-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e2b0e7-6204-402f-a79c-fa8ff22b3a70-maas-api-service-ca\") pod \"47e2b0e7-6204-402f-a79c-fa8ff22b3a70\" (UID: \"47e2b0e7-6204-402f-a79c-fa8ff22b3a70\") "
Oct 30 01:30:20.254410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.254388    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bxz7\" (UniqueName: \"kubernetes.io/projected/b3d49b88-2ba6-4ded-a160-806e4e8fed2b-kube-api-access-2bxz7\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:20.254544 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.254528    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xpth\" (UniqueName: \"kubernetes.io/projected/107404d2-96de-452e-8e7d-24acb8246db3-kube-api-access-2xpth\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:20.254639 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.254627    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/107404d2-96de-452e-8e7d-24acb8246db3-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:20.254727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.254715    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:20.254829 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.254816    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcp9m\" (UniqueName: \"kubernetes.io/projected/4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1-kube-api-access-xcp9m\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:20.255294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.255271    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e2b0e7-6204-402f-a79c-fa8ff22b3a70-maas-api-service-ca" pod "47e2b0e7-6204-402f-a79c-fa8ff22b3a70" (UID: "47e2b0e7-6204-402f-a79c-fa8ff22b3a70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:20.255540 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.255516    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e2b0e7-6204-402f-a79c-fa8ff22b3a70-kube-api-access-d58t9" pod "47e2b0e7-6204-402f-a79c-fa8ff22b3a70" (UID: "47e2b0e7-6204-402f-a79c-fa8ff22b3a70"). InnerVolumeSpecName "kube-api-access-d58t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:20.280674 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.280647    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:20.356898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.356849    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d58t9\" (UniqueName: \"kubernetes.io/projected/47e2b0e7-6204-402f-a79c-fa8ff22b3a70-kube-api-access-d58t9\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:20.356898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.356878    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e2b0e7-6204-402f-a79c-fa8ff22b3a70-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:20.479718 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.479635    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth" podUID="600ad629-faf2-49eb-9d14-05f9cf294799" containerName="payload-processing" containerID="cri-o://bacb03000ea169fafb82cad098399326bba0f6f91e2780d60f05d387ca678b9f" gracePeriod=25
Oct 30 01:30:20.622811 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.622782    4425 generic.go:356] "Generic (PLEG): container finished" podID="3699d466-411f-4f43-b62f-e6955340d1da" containerID="5d5f946353141fb0104afbfdecfc40be1b15ad8f64f3e8f4a3d61613cdd3c06e" exitCode=60
Oct 30 01:30:20.623214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.622855    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd" event={"ID":"3699d466-411f-4f43-b62f-e6955340d1da","Type":"ContainerDied","Data":"5d5f946353141fb0104afbfdecfc40be1b15ad8f64f3e8f4a3d61613cdd3c06e"}
Oct 30 01:30:20.623214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.622917    4425 scope.go:122] "RemoveContainer" containerID="00a54dc39418f2348243269bad8b3de89caea2154e18d1f83d351138176c7b11"
Oct 30 01:30:20.623366 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.623348    4425 scope.go:122] "RemoveContainer" containerID="5d5f946353141fb0104afbfdecfc40be1b15ad8f64f3e8f4a3d61613cdd3c06e"
Oct 30 01:30:20.623558 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.623521    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"]
Oct 30 01:30:20.623709 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:20.623691    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd_odh-ai-gateway-infra(3699d466-411f-4f43-b62f-e6955340d1da)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd" podUID="3699d466-411f-4f43-b62f-e6955340d1da"
Oct 30 01:30:20.625308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.624845    4425 generic.go:356] "Generic (PLEG): container finished" podID="600ad629-faf2-49eb-9d14-05f9cf294799" containerID="bacb03000ea169fafb82cad098399326bba0f6f91e2780d60f05d387ca678b9f" exitCode=0
Oct 30 01:30:20.625308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.624911    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth" event={"ID":"600ad629-faf2-49eb-9d14-05f9cf294799","Type":"ContainerDied","Data":"bacb03000ea169fafb82cad098399326bba0f6f91e2780d60f05d387ca678b9f"}
Oct 30 01:30:20.626946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.626925    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt"
Oct 30 01:30:20.626946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.626936    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt" event={"ID":"107404d2-96de-452e-8e7d-24acb8246db3","Type":"ContainerDied","Data":"0e049e1334171af215eecf4c6a5656a627a30a6b2b90ccb450767605faee7b75"}
Oct 30 01:30:20.636173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.631527    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n" event={"ID":"b3d49b88-2ba6-4ded-a160-806e4e8fed2b","Type":"ContainerDied","Data":"9e5a8683b1b3b4865614144bcee0dd0d91efa531a2741ede2cdfff2911b4504e"}
Oct 30 01:30:20.636173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.631638    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n"
Oct 30 01:30:20.637652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.637632    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb"
Oct 30 01:30:20.637773 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.637650    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb" event={"ID":"4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1","Type":"ContainerDied","Data":"a4600020613b0fb72d208a4eb43d9899da3dcd0e9ee3c23254553c80a0acccf0"}
Oct 30 01:30:20.639629 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:20.639544    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02874d4b_fe79_4247_b420_369003cd1d23.slice/crio-95a5b446d73ea76228182068bab0dfe163ad76eb844dfcfc32e7571ba27bc141 WatchSource:0}: Error finding container 95a5b446d73ea76228182068bab0dfe163ad76eb844dfcfc32e7571ba27bc141: Status 404 returned error can't find the container with id 95a5b446d73ea76228182068bab0dfe163ad76eb844dfcfc32e7571ba27bc141
Oct 30 01:30:20.642122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.641360    4425 generic.go:356] "Generic (PLEG): container finished" podID="be3cc419-c375-4b6b-a359-677d816e907b" containerID="9d6c7ba71faf00ee118f19cae6785943ea4fef44bfe8c51871baf778ae746341" exitCode=60
Oct 30 01:30:20.642122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.641456    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7" event={"ID":"be3cc419-c375-4b6b-a359-677d816e907b","Type":"ContainerDied","Data":"9d6c7ba71faf00ee118f19cae6785943ea4fef44bfe8c51871baf778ae746341"}
Oct 30 01:30:20.642122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.641827    4425 scope.go:122] "RemoveContainer" containerID="9d6c7ba71faf00ee118f19cae6785943ea4fef44bfe8c51871baf778ae746341"
Oct 30 01:30:20.642122 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:20.642060    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7_odh-ai-gateway-infra(be3cc419-c375-4b6b-a359-677d816e907b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7" podUID="be3cc419-c375-4b6b-a359-677d816e907b"
Oct 30 01:30:20.647277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.643504    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:30:20.647277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.643583    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:30:20.647277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.643628    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:30:20.652743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.652709    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9"]
Oct 30 01:30:20.654305 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.654286    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct" event={"ID":"47e2b0e7-6204-402f-a79c-fa8ff22b3a70","Type":"ContainerDied","Data":"7c352f094ccc676f315786b3f5fb51d73eefaf02d0e75846f742304c62f51e8e"}
Oct 30 01:30:20.654376 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.654359    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct"
Oct 30 01:30:20.662923 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.662899    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc" event={"ID":"e2354e0a-cbfa-4fe1-8903-842575a2f208","Type":"ContainerStarted","Data":"b9f81c35fab167161626ffd495a1f389a4199bcd5bff0f38bd27f592a64a4097"}
Oct 30 01:30:20.663361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.663330    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:30:20.663449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.663379    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc" event={"ID":"e2354e0a-cbfa-4fe1-8903-842575a2f208","Type":"ContainerStarted","Data":"238bc6f59b294f017ef17f6bae9905e15b46f04719c42bdbe284e6f27a69100f"}
Oct 30 01:30:20.668859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.668459    4425 generic.go:356] "Generic (PLEG): container finished" podID="230424d1-69ed-40c3-8afd-660d905f9d8d" containerID="31e333f4a9155c56e1846a57664340adbe13d0f34b9dcd36b1f1a68b7ddffe5d" exitCode=60
Oct 30 01:30:20.668859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.668486    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862" event={"ID":"230424d1-69ed-40c3-8afd-660d905f9d8d","Type":"ContainerDied","Data":"31e333f4a9155c56e1846a57664340adbe13d0f34b9dcd36b1f1a68b7ddffe5d"}
Oct 30 01:30:20.669366 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.669021    4425 scope.go:122] "RemoveContainer" containerID="31e333f4a9155c56e1846a57664340adbe13d0f34b9dcd36b1f1a68b7ddffe5d"
Oct 30 01:30:20.669468 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:20.669369    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862_odh-ai-gateway-infra(230424d1-69ed-40c3-8afd-660d905f9d8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862" podUID="230424d1-69ed-40c3-8afd-660d905f9d8d"
Oct 30 01:30:20.671603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.671265    4425 generic.go:356] "Generic (PLEG): container finished" podID="c284f618-116f-426d-b25d-fde4ca3c686e" containerID="f4bf8322d1b022d4f932e06092a90ac8930e360729ffd81397bcd9fc7f410143" exitCode=60
Oct 30 01:30:20.671603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.671303    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9" event={"ID":"c284f618-116f-426d-b25d-fde4ca3c686e","Type":"ContainerDied","Data":"f4bf8322d1b022d4f932e06092a90ac8930e360729ffd81397bcd9fc7f410143"}
Oct 30 01:30:20.704201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.704119    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862"]
Oct 30 01:30:20.745837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.745724    4425 scope.go:122] "RemoveContainer" containerID="63d00090671b6000073e92975ec31f81f64678392a3dfa2cc28410c59194407c"
Oct 30 01:30:20.767404 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.767300    4425 scope.go:122] "RemoveContainer" containerID="0edcf5f31617eb25343a3eafd6805369a2780b6e8d2650bed857e4e2a4d9f737"
Oct 30 01:30:20.786659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.786630    4425 scope.go:122] "RemoveContainer" containerID="24824f4e3f5aef315a82ebb26fecb92a6850ef4c3f8ba16f24a7e85f6c63ef79"
Oct 30 01:30:20.788127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.788098    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"
Oct 30 01:30:20.810576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.810270    4425 scope.go:122] "RemoveContainer" containerID="740018d6eafa31dd0d56755582c3803ecce1280a766837af781db9d49f2240b9"
Oct 30 01:30:20.827170 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.827127    4425 scope.go:122] "RemoveContainer" containerID="2f998c6b72312c77ea8468569be36386c97909988e851fb45773e235e8e9499a"
Oct 30 01:30:20.844744 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.844647    4425 scope.go:122] "RemoveContainer" containerID="91f3efcc9801b8778d1f3914497097e45bbe5181097701b9de0c7e4330656613"
Oct 30 01:30:20.845849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.845585    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc" podStartSLOduration=1.845565513 podStartE2EDuration="1.845565513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:20.843931335 +0000 UTC m=+2929.593586838" watchObservedRunningTime="2026-10-30 01:30:20.845565513 +0000 UTC m=+2929.595220824"
Oct 30 01:30:20.871009 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.870712    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx"]
Oct 30 01:30:20.879438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.879126    4425 scope.go:122] "RemoveContainer" containerID="932207fe531f80a91a1eb16d5ddb9ad053e1ed8b1c909a69aaf747b9d7c29c92"
Oct 30 01:30:20.941221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.941130    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n"]
Oct 30 01:30:20.947737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.947655    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm"
Oct 30 01:30:20.948035 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.948013    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-35042a-29888730-hk62n"]
Oct 30 01:30:20.963733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.963711    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9"
Oct 30 01:30:20.967693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.966892    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600ad629-faf2-49eb-9d14-05f9cf294799-provider-credentials\" (UniqueName: \"kubernetes.io/projected/600ad629-faf2-49eb-9d14-05f9cf294799-provider-credentials\") pod \"600ad629-faf2-49eb-9d14-05f9cf294799\" (UID: \"600ad629-faf2-49eb-9d14-05f9cf294799\") "
Oct 30 01:30:20.967693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.966985    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600ad629-faf2-49eb-9d14-05f9cf294799-kube-api-access-t5fqd\" (UniqueName: \"kubernetes.io/projected/600ad629-faf2-49eb-9d14-05f9cf294799-kube-api-access-t5fqd\") pod \"600ad629-faf2-49eb-9d14-05f9cf294799\" (UID: \"600ad629-faf2-49eb-9d14-05f9cf294799\") "
Oct 30 01:30:20.967693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.967110    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/600ad629-faf2-49eb-9d14-05f9cf294799-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600ad629-faf2-49eb-9d14-05f9cf294799-plugins-config-volume\") pod \"600ad629-faf2-49eb-9d14-05f9cf294799\" (UID: \"600ad629-faf2-49eb-9d14-05f9cf294799\") "
Oct 30 01:30:20.967693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.967139    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/600ad629-faf2-49eb-9d14-05f9cf294799-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/600ad629-faf2-49eb-9d14-05f9cf294799-routing-overlay-volume\") pod \"600ad629-faf2-49eb-9d14-05f9cf294799\" (UID: \"600ad629-faf2-49eb-9d14-05f9cf294799\") "
Oct 30 01:30:20.968424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.968085    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600ad629-faf2-49eb-9d14-05f9cf294799-routing-overlay-volume" pod "600ad629-faf2-49eb-9d14-05f9cf294799" (UID: "600ad629-faf2-49eb-9d14-05f9cf294799"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:20.968973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.968953    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600ad629-faf2-49eb-9d14-05f9cf294799-plugins-config-volume" pod "600ad629-faf2-49eb-9d14-05f9cf294799" (UID: "600ad629-faf2-49eb-9d14-05f9cf294799"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:20.972459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.972436    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600ad629-faf2-49eb-9d14-05f9cf294799-kube-api-access-t5fqd" pod "600ad629-faf2-49eb-9d14-05f9cf294799" (UID: "600ad629-faf2-49eb-9d14-05f9cf294799"). InnerVolumeSpecName "kube-api-access-t5fqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:20.980339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:20.980315    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600ad629-faf2-49eb-9d14-05f9cf294799-provider-credentials" pod "600ad629-faf2-49eb-9d14-05f9cf294799" (UID: "600ad629-faf2-49eb-9d14-05f9cf294799"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:21.041953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.041921    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct"]
Oct 30 01:30:21.048259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.048231    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1854c4-29888730-bnsct"]
Oct 30 01:30:21.068255 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.068227    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c284f618-116f-426d-b25d-fde4ca3c686e-kube-api-access-sl7wb\" (UniqueName: \"kubernetes.io/projected/c284f618-116f-426d-b25d-fde4ca3c686e-kube-api-access-sl7wb\") pod \"c284f618-116f-426d-b25d-fde4ca3c686e\" (UID: \"c284f618-116f-426d-b25d-fde4ca3c686e\") "
Oct 30 01:30:21.068837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.068451    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03eaf7ac-cf67-4da9-b253-3c1994ad57fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03eaf7ac-cf67-4da9-b253-3c1994ad57fa-maas-api-service-ca\") pod \"03eaf7ac-cf67-4da9-b253-3c1994ad57fa\" (UID: \"03eaf7ac-cf67-4da9-b253-3c1994ad57fa\") "
Oct 30 01:30:21.068837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.068535    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c284f618-116f-426d-b25d-fde4ca3c686e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c284f618-116f-426d-b25d-fde4ca3c686e-maas-api-service-ca\") pod \"c284f618-116f-426d-b25d-fde4ca3c686e\" (UID: \"c284f618-116f-426d-b25d-fde4ca3c686e\") "
Oct 30 01:30:21.068837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.068596    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03eaf7ac-cf67-4da9-b253-3c1994ad57fa-kube-api-access-5wtxd\" (UniqueName: \"kubernetes.io/projected/03eaf7ac-cf67-4da9-b253-3c1994ad57fa-kube-api-access-5wtxd\") pod \"03eaf7ac-cf67-4da9-b253-3c1994ad57fa\" (UID: \"03eaf7ac-cf67-4da9-b253-3c1994ad57fa\") "
Oct 30 01:30:21.069137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.069111    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/600ad629-faf2-49eb-9d14-05f9cf294799-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.069743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.069712    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5fqd\" (UniqueName: \"kubernetes.io/projected/600ad629-faf2-49eb-9d14-05f9cf294799-kube-api-access-t5fqd\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.069743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.069749    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600ad629-faf2-49eb-9d14-05f9cf294799-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.069913 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.069369    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03eaf7ac-cf67-4da9-b253-3c1994ad57fa-maas-api-service-ca" pod "03eaf7ac-cf67-4da9-b253-3c1994ad57fa" (UID: "03eaf7ac-cf67-4da9-b253-3c1994ad57fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:21.069913 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.069504    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c284f618-116f-426d-b25d-fde4ca3c686e-maas-api-service-ca" pod "c284f618-116f-426d-b25d-fde4ca3c686e" (UID: "c284f618-116f-426d-b25d-fde4ca3c686e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:21.069913 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.069765    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/600ad629-faf2-49eb-9d14-05f9cf294799-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.072135 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.072106    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c284f618-116f-426d-b25d-fde4ca3c686e-kube-api-access-sl7wb" pod "c284f618-116f-426d-b25d-fde4ca3c686e" (UID: "c284f618-116f-426d-b25d-fde4ca3c686e"). InnerVolumeSpecName "kube-api-access-sl7wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:21.072730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.072703    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03eaf7ac-cf67-4da9-b253-3c1994ad57fa-kube-api-access-5wtxd" pod "03eaf7ac-cf67-4da9-b253-3c1994ad57fa" (UID: "03eaf7ac-cf67-4da9-b253-3c1994ad57fa"). InnerVolumeSpecName "kube-api-access-5wtxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:21.073100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.073060    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb"]
Oct 30 01:30:21.079053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.079029    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888730-2sbtb"]
Oct 30 01:30:21.102308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.102284    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx"
Oct 30 01:30:21.104894 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.104866    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt"]
Oct 30 01:30:21.109667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.109645    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888730-wmslt"]
Oct 30 01:30:21.171345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.171311    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03eaf7ac-cf67-4da9-b253-3c1994ad57fa-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.171345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.171347    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c284f618-116f-426d-b25d-fde4ca3c686e-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.171572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.171364    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wtxd\" (UniqueName: \"kubernetes.io/projected/03eaf7ac-cf67-4da9-b253-3c1994ad57fa-kube-api-access-5wtxd\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.171572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.171379    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl7wb\" (UniqueName: \"kubernetes.io/projected/c284f618-116f-426d-b25d-fde4ca3c686e-kube-api-access-sl7wb\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.272760 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.272727    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd93d36a-1693-4cbb-bf97-204d5c295c0d-kube-api-access-8lfgs\" (UniqueName: \"kubernetes.io/projected/fd93d36a-1693-4cbb-bf97-204d5c295c0d-kube-api-access-8lfgs\") pod \"fd93d36a-1693-4cbb-bf97-204d5c295c0d\" (UID: \"fd93d36a-1693-4cbb-bf97-204d5c295c0d\") "
Oct 30 01:30:21.272950 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.272793    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd93d36a-1693-4cbb-bf97-204d5c295c0d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd93d36a-1693-4cbb-bf97-204d5c295c0d-maas-api-service-ca\") pod \"fd93d36a-1693-4cbb-bf97-204d5c295c0d\" (UID: \"fd93d36a-1693-4cbb-bf97-204d5c295c0d\") "
Oct 30 01:30:21.273287 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.273259    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd93d36a-1693-4cbb-bf97-204d5c295c0d-maas-api-service-ca" pod "fd93d36a-1693-4cbb-bf97-204d5c295c0d" (UID: "fd93d36a-1693-4cbb-bf97-204d5c295c0d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:21.273725 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.273699    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd93d36a-1693-4cbb-bf97-204d5c295c0d-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.275379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.275349    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd93d36a-1693-4cbb-bf97-204d5c295c0d-kube-api-access-8lfgs" pod "fd93d36a-1693-4cbb-bf97-204d5c295c0d" (UID: "fd93d36a-1693-4cbb-bf97-204d5c295c0d"). InnerVolumeSpecName "kube-api-access-8lfgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:21.375147 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.375062    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lfgs\" (UniqueName: \"kubernetes.io/projected/fd93d36a-1693-4cbb-bf97-204d5c295c0d-kube-api-access-8lfgs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:21.437715 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.437680    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107404d2-96de-452e-8e7d-24acb8246db3" path="/var/lib/kubelet/pods/107404d2-96de-452e-8e7d-24acb8246db3/volumes"
Oct 30 01:30:21.438192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.438172    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e2b0e7-6204-402f-a79c-fa8ff22b3a70" path="/var/lib/kubelet/pods/47e2b0e7-6204-402f-a79c-fa8ff22b3a70/volumes"
Oct 30 01:30:21.438578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.438565    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1" path="/var/lib/kubelet/pods/4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1/volumes"
Oct 30 01:30:21.439302 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.439288    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53190760-7148-47da-93f0-1123e5df62e2" path="/var/lib/kubelet/pods/53190760-7148-47da-93f0-1123e5df62e2/volumes"
Oct 30 01:30:21.439694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.439680    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c894e6-ae30-4978-850a-dfbde9fd69bd" path="/var/lib/kubelet/pods/61c894e6-ae30-4978-850a-dfbde9fd69bd/volumes"
Oct 30 01:30:21.440074 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.440060    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3d49b88-2ba6-4ded-a160-806e4e8fed2b" path="/var/lib/kubelet/pods/b3d49b88-2ba6-4ded-a160-806e4e8fed2b/volumes"
Oct 30 01:30:21.681701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.681597    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd"]
Oct 30 01:30:21.697138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.696511    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9" event={"ID":"c284f618-116f-426d-b25d-fde4ca3c686e","Type":"ContainerDied","Data":"8b066266decc0607bbe3947eac0e0ca28358d4185b657d15fcd9289b61eca834"}
Oct 30 01:30:21.697138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.696586    4425 scope.go:122] "RemoveContainer" containerID="f4bf8322d1b022d4f932e06092a90ac8930e360729ffd81397bcd9fc7f410143"
Oct 30 01:30:21.697138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.696769    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9"
Oct 30 01:30:21.708547 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.708403    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"
Oct 30 01:30:21.709031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.708706    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth" event={"ID":"600ad629-faf2-49eb-9d14-05f9cf294799","Type":"ContainerDied","Data":"a2bcae866ddb6c9c43762b571478dd5c933787c35b69e745da37dbdd34603d06"}
Oct 30 01:30:21.711523 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.711499    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx" event={"ID":"fd93d36a-1693-4cbb-bf97-204d5c295c0d","Type":"ContainerDied","Data":"a111760e50e13129162662d840307d4de3d1e2faf96a862e8ab1ca0ccfb9ec41"}
Oct 30 01:30:21.711772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.711756    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx"
Oct 30 01:30:21.719265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.719243    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm" event={"ID":"03eaf7ac-cf67-4da9-b253-3c1994ad57fa","Type":"ContainerDied","Data":"fd8b9adc3c38e185be110395601a273cfe6acb8ee2bec9d1aac9278e63687da8"}
Oct 30 01:30:21.719427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.719306    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm"
Oct 30 01:30:21.722537 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.722521    4425 scope.go:122] "RemoveContainer" containerID="bacb03000ea169fafb82cad098399326bba0f6f91e2780d60f05d387ca678b9f"
Oct 30 01:30:21.739604 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.739571    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw" event={"ID":"02874d4b-fe79-4247-b420-369003cd1d23","Type":"ContainerStarted","Data":"2bc7d679e22cd00617cc44577ee880cb0eb20214e4832d00ecf46da9811675ce"}
Oct 30 01:30:21.739770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.739618    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw" event={"ID":"02874d4b-fe79-4247-b420-369003cd1d23","Type":"ContainerStarted","Data":"95a5b446d73ea76228182068bab0dfe163ad76eb844dfcfc32e7571ba27bc141"}
Oct 30 01:30:21.739967 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.739948    4425 scope.go:122] "RemoveContainer" containerID="ed513ebffab0ed8ac33af071c0f0ef25c52147390009e97be1af928626e47ad6"
Oct 30 01:30:21.793024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.792379    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9"]
Oct 30 01:30:21.797929 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.796317    4425 scope.go:122] "RemoveContainer" containerID="85dbbd0ea2f2da3c48712f1dbbd291131cd0db36d6bb16443a827efca0ba8a12"
Oct 30 01:30:21.802215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.800522    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-bc127f-29888730-tq9j9"]
Oct 30 01:30:21.813302 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.813272    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7"]
Oct 30 01:30:21.824232 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.824133    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm"]
Oct 30 01:30:21.832426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.832362    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9da49f-29888730-f8clm"]
Oct 30 01:30:21.891412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.884523    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx"]
Oct 30 01:30:21.893134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.892512    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888730-tq7gx"]
Oct 30 01:30:21.989231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.989114    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"]
Oct 30 01:30:21.992974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.992689    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862"
Oct 30 01:30:21.994259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.993366    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230424d1-69ed-40c3-8afd-660d905f9d8d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230424d1-69ed-40c3-8afd-660d905f9d8d-maas-api-service-ca\") pod \"230424d1-69ed-40c3-8afd-660d905f9d8d\" (UID: \"230424d1-69ed-40c3-8afd-660d905f9d8d\") "
Oct 30 01:30:21.994259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.993589    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230424d1-69ed-40c3-8afd-660d905f9d8d-kube-api-access-qddg4\" (UniqueName: \"kubernetes.io/projected/230424d1-69ed-40c3-8afd-660d905f9d8d-kube-api-access-qddg4\") pod \"230424d1-69ed-40c3-8afd-660d905f9d8d\" (UID: \"230424d1-69ed-40c3-8afd-660d905f9d8d\") "
Oct 30 01:30:21.994259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.993857    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-d7c4fc775-sgwth"]
Oct 30 01:30:21.994259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.994227    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230424d1-69ed-40c3-8afd-660d905f9d8d-maas-api-service-ca" pod "230424d1-69ed-40c3-8afd-660d905f9d8d" (UID: "230424d1-69ed-40c3-8afd-660d905f9d8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:21.996686 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:21.996659    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230424d1-69ed-40c3-8afd-660d905f9d8d-kube-api-access-qddg4" pod "230424d1-69ed-40c3-8afd-660d905f9d8d" (UID: "230424d1-69ed-40c3-8afd-660d905f9d8d"). InnerVolumeSpecName "kube-api-access-qddg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:22.016931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.016880    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw" podStartSLOduration=3.0168623 podStartE2EDuration="3.0168623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:22.011811007 +0000 UTC m=+2930.761466309" watchObservedRunningTime="2026-10-30 01:30:22.0168623 +0000 UTC m=+2930.766517609"
Oct 30 01:30:22.095607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.095567    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qddg4\" (UniqueName: \"kubernetes.io/projected/230424d1-69ed-40c3-8afd-660d905f9d8d-kube-api-access-qddg4\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:22.095607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.095601    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/230424d1-69ed-40c3-8afd-660d905f9d8d-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:22.097057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.097006    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd"
Oct 30 01:30:22.101677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.101652    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7"
Oct 30 01:30:22.196212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.196186    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3699d466-411f-4f43-b62f-e6955340d1da-kube-api-access-mgrl8\" (UniqueName: \"kubernetes.io/projected/3699d466-411f-4f43-b62f-e6955340d1da-kube-api-access-mgrl8\") pod \"3699d466-411f-4f43-b62f-e6955340d1da\" (UID: \"3699d466-411f-4f43-b62f-e6955340d1da\") "
Oct 30 01:30:22.196420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.196402    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be3cc419-c375-4b6b-a359-677d816e907b-kube-api-access-ws8gv\" (UniqueName: \"kubernetes.io/projected/be3cc419-c375-4b6b-a359-677d816e907b-kube-api-access-ws8gv\") pod \"be3cc419-c375-4b6b-a359-677d816e907b\" (UID: \"be3cc419-c375-4b6b-a359-677d816e907b\") "
Oct 30 01:30:22.196576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.196558    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3699d466-411f-4f43-b62f-e6955340d1da-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3699d466-411f-4f43-b62f-e6955340d1da-maas-api-service-ca\") pod \"3699d466-411f-4f43-b62f-e6955340d1da\" (UID: \"3699d466-411f-4f43-b62f-e6955340d1da\") "
Oct 30 01:30:22.196690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.196664    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be3cc419-c375-4b6b-a359-677d816e907b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3cc419-c375-4b6b-a359-677d816e907b-maas-api-service-ca\") pod \"be3cc419-c375-4b6b-a359-677d816e907b\" (UID: \"be3cc419-c375-4b6b-a359-677d816e907b\") "
Oct 30 01:30:22.197044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.197015    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3699d466-411f-4f43-b62f-e6955340d1da-maas-api-service-ca" pod "3699d466-411f-4f43-b62f-e6955340d1da" (UID: "3699d466-411f-4f43-b62f-e6955340d1da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:22.197207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.197139    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3699d466-411f-4f43-b62f-e6955340d1da-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:22.197445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.197415    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3cc419-c375-4b6b-a359-677d816e907b-maas-api-service-ca" pod "be3cc419-c375-4b6b-a359-677d816e907b" (UID: "be3cc419-c375-4b6b-a359-677d816e907b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:22.199183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.199143    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3cc419-c375-4b6b-a359-677d816e907b-kube-api-access-ws8gv" pod "be3cc419-c375-4b6b-a359-677d816e907b" (UID: "be3cc419-c375-4b6b-a359-677d816e907b"). InnerVolumeSpecName "kube-api-access-ws8gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:22.199761 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.199736    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3699d466-411f-4f43-b62f-e6955340d1da-kube-api-access-mgrl8" pod "3699d466-411f-4f43-b62f-e6955340d1da" (UID: "3699d466-411f-4f43-b62f-e6955340d1da"). InnerVolumeSpecName "kube-api-access-mgrl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:22.281288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.281211    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:22.283492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.283191    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rb6rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 01:30:22.283492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.283243    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw" podUID="02874d4b-fe79-4247-b420-369003cd1d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 01:30:22.298721 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.298699    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgrl8\" (UniqueName: \"kubernetes.io/projected/3699d466-411f-4f43-b62f-e6955340d1da-kube-api-access-mgrl8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:22.298879 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.298868    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws8gv\" (UniqueName: \"kubernetes.io/projected/be3cc419-c375-4b6b-a359-677d816e907b-kube-api-access-ws8gv\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:22.298940 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.298932    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3cc419-c375-4b6b-a359-677d816e907b-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:22.748752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.748669    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862"
Oct 30 01:30:22.749235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.748664    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862" event={"ID":"230424d1-69ed-40c3-8afd-660d905f9d8d","Type":"ContainerDied","Data":"02bbc638119701c6f0a6ca9cfe391bdc26f37bce6dbaef579a28ea311a1e15ce"}
Oct 30 01:30:22.749235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.748881    4425 scope.go:122] "RemoveContainer" containerID="31e333f4a9155c56e1846a57664340adbe13d0f34b9dcd36b1f1a68b7ddffe5d"
Oct 30 01:30:22.779928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.779895    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd" event={"ID":"3699d466-411f-4f43-b62f-e6955340d1da","Type":"ContainerDied","Data":"9bd9c6311ae46a68c5d267c009aa7439830d13e81d441bd13453332753ece554"}
Oct 30 01:30:22.780031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.780004    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd"
Oct 30 01:30:22.786508 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.786483    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7"
Oct 30 01:30:22.786619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.786502    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7" event={"ID":"be3cc419-c375-4b6b-a359-677d816e907b","Type":"ContainerDied","Data":"124ad287086f77da2a9d9cfc7548806d8b1bd09a6ee6cefbd0177bbfb55f39d2"}
Oct 30 01:30:22.790417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.790309    4425 scope.go:122] "RemoveContainer" containerID="5d5f946353141fb0104afbfdecfc40be1b15ad8f64f3e8f4a3d61613cdd3c06e"
Oct 30 01:30:22.801920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.801902    4425 scope.go:122] "RemoveContainer" containerID="9d6c7ba71faf00ee118f19cae6785943ea4fef44bfe8c51871baf778ae746341"
Oct 30 01:30:22.827107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.827085    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862"]
Oct 30 01:30:22.835242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.835222    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8ae41e-29888730-pz862"]
Oct 30 01:30:22.894649 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.894627    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd"]
Oct 30 01:30:22.903107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.903083    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888730-c7pjd"]
Oct 30 01:30:22.930643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.930623    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7"]
Oct 30 01:30:22.943172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:22.936591    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-0e1a98-29888730-8bfb7"]
Oct 30 01:30:23.094749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.094695    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t" podUID="2e420fa1-2a6e-4e9f-9daf-97b6d031cc26" containerName="payload-processing" containerID="cri-o://628beffd62d1cdbcc3e63594d886e53ca7366dd39ed6909f9f7ff8f262ebbace" gracePeriod=25
Oct 30 01:30:23.294178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.293775    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rb6rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 01:30:23.294178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.293833    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw" podUID="02874d4b-fe79-4247-b420-369003cd1d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 01:30:23.443231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.442853    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03eaf7ac-cf67-4da9-b253-3c1994ad57fa" path="/var/lib/kubelet/pods/03eaf7ac-cf67-4da9-b253-3c1994ad57fa/volumes"
Oct 30 01:30:23.443713 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.443692    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230424d1-69ed-40c3-8afd-660d905f9d8d" path="/var/lib/kubelet/pods/230424d1-69ed-40c3-8afd-660d905f9d8d/volumes"
Oct 30 01:30:23.444878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.444856    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3699d466-411f-4f43-b62f-e6955340d1da" path="/var/lib/kubelet/pods/3699d466-411f-4f43-b62f-e6955340d1da/volumes"
Oct 30 01:30:23.446057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.446037    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="600ad629-faf2-49eb-9d14-05f9cf294799" path="/var/lib/kubelet/pods/600ad629-faf2-49eb-9d14-05f9cf294799/volumes"
Oct 30 01:30:23.446753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.446724    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be3cc419-c375-4b6b-a359-677d816e907b" path="/var/lib/kubelet/pods/be3cc419-c375-4b6b-a359-677d816e907b/volumes"
Oct 30 01:30:23.447466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.447447    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c284f618-116f-426d-b25d-fde4ca3c686e" path="/var/lib/kubelet/pods/c284f618-116f-426d-b25d-fde4ca3c686e/volumes"
Oct 30 01:30:23.448563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.448544    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd93d36a-1693-4cbb-bf97-204d5c295c0d" path="/var/lib/kubelet/pods/fd93d36a-1693-4cbb-bf97-204d5c295c0d/volumes"
Oct 30 01:30:23.500187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.498778    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:30:23.522669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.521405    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-routing-overlay-volume\") pod \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\" (UID: \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\") "
Oct 30 01:30:23.522669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.521508    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-provider-credentials\") pod \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\" (UID: \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\") "
Oct 30 01:30:23.522669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.521605    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-plugins-config-volume\") pod \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\" (UID: \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\") "
Oct 30 01:30:23.522669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.521704    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-kube-api-access-k45hl\" (UniqueName: \"kubernetes.io/projected/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-kube-api-access-k45hl\") pod \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\" (UID: \"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26\") "
Oct 30 01:30:23.523117 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.522933    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-routing-overlay-volume" pod "2e420fa1-2a6e-4e9f-9daf-97b6d031cc26" (UID: "2e420fa1-2a6e-4e9f-9daf-97b6d031cc26"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:23.523344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.523322    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-plugins-config-volume" pod "2e420fa1-2a6e-4e9f-9daf-97b6d031cc26" (UID: "2e420fa1-2a6e-4e9f-9daf-97b6d031cc26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:23.526734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.526706    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-provider-credentials" pod "2e420fa1-2a6e-4e9f-9daf-97b6d031cc26" (UID: "2e420fa1-2a6e-4e9f-9daf-97b6d031cc26"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:23.527518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.527490    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-kube-api-access-k45hl" pod "2e420fa1-2a6e-4e9f-9daf-97b6d031cc26" (UID: "2e420fa1-2a6e-4e9f-9daf-97b6d031cc26"). InnerVolumeSpecName "kube-api-access-k45hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:23.567177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.560343    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk" podUID="1c880cf0-b76c-4db8-a26c-d7af8345a80b" containerName="payload-processing" containerID="cri-o://d853f8ce036fe9f6f27a9b6e214f6dbe8cd4281027434b2ad41667388d56f055" gracePeriod=25
Oct 30 01:30:23.623433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.623395    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:23.623661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.623641    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:23.623779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.623764    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k45hl\" (UniqueName: \"kubernetes.io/projected/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-kube-api-access-k45hl\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:23.623898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.623883    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:23.806329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.803101    4425 generic.go:356] "Generic (PLEG): container finished" podID="2e420fa1-2a6e-4e9f-9daf-97b6d031cc26" containerID="628beffd62d1cdbcc3e63594d886e53ca7366dd39ed6909f9f7ff8f262ebbace" exitCode=0
Oct 30 01:30:23.806329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.803262    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"
Oct 30 01:30:23.806329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.804097    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t" event={"ID":"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26","Type":"ContainerDied","Data":"628beffd62d1cdbcc3e63594d886e53ca7366dd39ed6909f9f7ff8f262ebbace"}
Oct 30 01:30:23.806329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.804171    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t" event={"ID":"2e420fa1-2a6e-4e9f-9daf-97b6d031cc26","Type":"ContainerDied","Data":"9e07350728cf34249c49ef525d10ba30f3c39a1056944326773cbc3dce07adb4"}
Oct 30 01:30:23.806329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.804204    4425 scope.go:122] "RemoveContainer" containerID="628beffd62d1cdbcc3e63594d886e53ca7366dd39ed6909f9f7ff8f262ebbace"
Oct 30 01:30:23.812101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.811240    4425 generic.go:356] "Generic (PLEG): container finished" podID="1c880cf0-b76c-4db8-a26c-d7af8345a80b" containerID="d853f8ce036fe9f6f27a9b6e214f6dbe8cd4281027434b2ad41667388d56f055" exitCode=0
Oct 30 01:30:23.812101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.812073    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk" event={"ID":"1c880cf0-b76c-4db8-a26c-d7af8345a80b","Type":"ContainerDied","Data":"d853f8ce036fe9f6f27a9b6e214f6dbe8cd4281027434b2ad41667388d56f055"}
Oct 30 01:30:23.834453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.831870    4425 scope.go:122] "RemoveContainer" containerID="628beffd62d1cdbcc3e63594d886e53ca7366dd39ed6909f9f7ff8f262ebbace"
Oct 30 01:30:23.834667 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:23.834605    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"628beffd62d1cdbcc3e63594d886e53ca7366dd39ed6909f9f7ff8f262ebbace\": container with ID starting with 628beffd62d1cdbcc3e63594d886e53ca7366dd39ed6909f9f7ff8f262ebbace not found: ID does not exist" containerID="628beffd62d1cdbcc3e63594d886e53ca7366dd39ed6909f9f7ff8f262ebbace"
Oct 30 01:30:23.834667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.834647    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"628beffd62d1cdbcc3e63594d886e53ca7366dd39ed6909f9f7ff8f262ebbace"} err="failed to get container status \"628beffd62d1cdbcc3e63594d886e53ca7366dd39ed6909f9f7ff8f262ebbace\": rpc error: code = NotFound desc = could not find container \"628beffd62d1cdbcc3e63594d886e53ca7366dd39ed6909f9f7ff8f262ebbace\": container with ID starting with 628beffd62d1cdbcc3e63594d886e53ca7366dd39ed6909f9f7ff8f262ebbace not found: ID does not exist"
Oct 30 01:30:23.889237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.888071    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"]
Oct 30 01:30:23.900797 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.900764    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-66b9xh86t"]
Oct 30 01:30:23.944475 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:23.944445    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"
Oct 30 01:30:24.136945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.136907    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c880cf0-b76c-4db8-a26c-d7af8345a80b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c880cf0-b76c-4db8-a26c-d7af8345a80b-provider-credentials\") pod \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\" (UID: \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\") "
Oct 30 01:30:24.137184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.137108    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c880cf0-b76c-4db8-a26c-d7af8345a80b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c880cf0-b76c-4db8-a26c-d7af8345a80b-routing-overlay-volume\") pod \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\" (UID: \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\") "
Oct 30 01:30:24.137266 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.137180    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c880cf0-b76c-4db8-a26c-d7af8345a80b-kube-api-access-dzfw7\" (UniqueName: \"kubernetes.io/projected/1c880cf0-b76c-4db8-a26c-d7af8345a80b-kube-api-access-dzfw7\") pod \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\" (UID: \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\") "
Oct 30 01:30:24.137322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.137303    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c880cf0-b76c-4db8-a26c-d7af8345a80b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c880cf0-b76c-4db8-a26c-d7af8345a80b-plugins-config-volume\") pod \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\" (UID: \"1c880cf0-b76c-4db8-a26c-d7af8345a80b\") "
Oct 30 01:30:24.137451 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.137427    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c880cf0-b76c-4db8-a26c-d7af8345a80b-routing-overlay-volume" pod "1c880cf0-b76c-4db8-a26c-d7af8345a80b" (UID: "1c880cf0-b76c-4db8-a26c-d7af8345a80b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:24.137864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.137836    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1c880cf0-b76c-4db8-a26c-d7af8345a80b-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:24.137864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.137849    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c880cf0-b76c-4db8-a26c-d7af8345a80b-plugins-config-volume" pod "1c880cf0-b76c-4db8-a26c-d7af8345a80b" (UID: "1c880cf0-b76c-4db8-a26c-d7af8345a80b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:24.139897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.139873    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c880cf0-b76c-4db8-a26c-d7af8345a80b-provider-credentials" pod "1c880cf0-b76c-4db8-a26c-d7af8345a80b" (UID: "1c880cf0-b76c-4db8-a26c-d7af8345a80b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:24.140057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.140041    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c880cf0-b76c-4db8-a26c-d7af8345a80b-kube-api-access-dzfw7" pod "1c880cf0-b76c-4db8-a26c-d7af8345a80b" (UID: "1c880cf0-b76c-4db8-a26c-d7af8345a80b"). InnerVolumeSpecName "kube-api-access-dzfw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:24.239365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.239333    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzfw7\" (UniqueName: \"kubernetes.io/projected/1c880cf0-b76c-4db8-a26c-d7af8345a80b-kube-api-access-dzfw7\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:24.239365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.239361    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c880cf0-b76c-4db8-a26c-d7af8345a80b-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:24.239365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.239370    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1c880cf0-b76c-4db8-a26c-d7af8345a80b-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:24.282477 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.282442    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rb6rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 01:30:24.282665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.282507    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw" podUID="02874d4b-fe79-4247-b420-369003cd1d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 01:30:24.483496 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.483381    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt" podUID="f4defc5b-55df-4d40-ada5-a9463b99e7f3" containerName="payload-processing" containerID="cri-o://95175b8179b90efe27ee987e4c011c661e16f9cb1c0f791cc159387551c747bd" gracePeriod=25
Oct 30 01:30:24.798933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.798907    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:24.822304 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.822268    4425 generic.go:356] "Generic (PLEG): container finished" podID="f4defc5b-55df-4d40-ada5-a9463b99e7f3" containerID="95175b8179b90efe27ee987e4c011c661e16f9cb1c0f791cc159387551c747bd" exitCode=0
Oct 30 01:30:24.822640 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.822318    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"
Oct 30 01:30:24.822640 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.822356    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt" event={"ID":"f4defc5b-55df-4d40-ada5-a9463b99e7f3","Type":"ContainerDied","Data":"95175b8179b90efe27ee987e4c011c661e16f9cb1c0f791cc159387551c747bd"}
Oct 30 01:30:24.822640 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.822411    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt" event={"ID":"f4defc5b-55df-4d40-ada5-a9463b99e7f3","Type":"ContainerDied","Data":"effd20b55e253caa7c1dfb7f90089f614f54e3dca004221289401e3c77548b6e"}
Oct 30 01:30:24.822640 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.822440    4425 scope.go:122] "RemoveContainer" containerID="95175b8179b90efe27ee987e4c011c661e16f9cb1c0f791cc159387551c747bd"
Oct 30 01:30:24.824388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.824370    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"
Oct 30 01:30:24.824511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.824389    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk" event={"ID":"1c880cf0-b76c-4db8-a26c-d7af8345a80b","Type":"ContainerDied","Data":"0995f74bc6a6013b8aaa01da76437258f60ba1454d05c9f5783efb5786713476"}
Oct 30 01:30:24.834008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.833985    4425 scope.go:122] "RemoveContainer" containerID="95175b8179b90efe27ee987e4c011c661e16f9cb1c0f791cc159387551c747bd"
Oct 30 01:30:24.834336 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:24.834314    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95175b8179b90efe27ee987e4c011c661e16f9cb1c0f791cc159387551c747bd\": container with ID starting with 95175b8179b90efe27ee987e4c011c661e16f9cb1c0f791cc159387551c747bd not found: ID does not exist" containerID="95175b8179b90efe27ee987e4c011c661e16f9cb1c0f791cc159387551c747bd"
Oct 30 01:30:24.834531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.834341    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95175b8179b90efe27ee987e4c011c661e16f9cb1c0f791cc159387551c747bd"} err="failed to get container status \"95175b8179b90efe27ee987e4c011c661e16f9cb1c0f791cc159387551c747bd\": rpc error: code = NotFound desc = could not find container \"95175b8179b90efe27ee987e4c011c661e16f9cb1c0f791cc159387551c747bd\": container with ID starting with 95175b8179b90efe27ee987e4c011c661e16f9cb1c0f791cc159387551c747bd not found: ID does not exist"
Oct 30 01:30:24.834531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.834358    4425 scope.go:122] "RemoveContainer" containerID="d853f8ce036fe9f6f27a9b6e214f6dbe8cd4281027434b2ad41667388d56f055"
Oct 30 01:30:24.846352 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.846329    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4defc5b-55df-4d40-ada5-a9463b99e7f3-kube-api-access-nth42\" (UniqueName: \"kubernetes.io/projected/f4defc5b-55df-4d40-ada5-a9463b99e7f3-kube-api-access-nth42\") pod \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\" (UID: \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\") "
Oct 30 01:30:24.846499 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.846444    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4defc5b-55df-4d40-ada5-a9463b99e7f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4defc5b-55df-4d40-ada5-a9463b99e7f3-plugins-config-volume\") pod \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\" (UID: \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\") "
Oct 30 01:30:24.846565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.846540    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4defc5b-55df-4d40-ada5-a9463b99e7f3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4defc5b-55df-4d40-ada5-a9463b99e7f3-routing-overlay-volume\") pod \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\" (UID: \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\") "
Oct 30 01:30:24.846637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.846626    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4defc5b-55df-4d40-ada5-a9463b99e7f3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4defc5b-55df-4d40-ada5-a9463b99e7f3-provider-credentials\") pod \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\" (UID: \"f4defc5b-55df-4d40-ada5-a9463b99e7f3\") "
Oct 30 01:30:24.846897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.846872    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4defc5b-55df-4d40-ada5-a9463b99e7f3-plugins-config-volume" pod "f4defc5b-55df-4d40-ada5-a9463b99e7f3" (UID: "f4defc5b-55df-4d40-ada5-a9463b99e7f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:24.846982 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.846918    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4defc5b-55df-4d40-ada5-a9463b99e7f3-routing-overlay-volume" pod "f4defc5b-55df-4d40-ada5-a9463b99e7f3" (UID: "f4defc5b-55df-4d40-ada5-a9463b99e7f3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:24.847375 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.847350    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4defc5b-55df-4d40-ada5-a9463b99e7f3-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:24.847748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.847380    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4defc5b-55df-4d40-ada5-a9463b99e7f3-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:24.883182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.881348    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4defc5b-55df-4d40-ada5-a9463b99e7f3-provider-credentials" pod "f4defc5b-55df-4d40-ada5-a9463b99e7f3" (UID: "f4defc5b-55df-4d40-ada5-a9463b99e7f3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:24.890733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.890710    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4defc5b-55df-4d40-ada5-a9463b99e7f3-kube-api-access-nth42" pod "f4defc5b-55df-4d40-ada5-a9463b99e7f3" (UID: "f4defc5b-55df-4d40-ada5-a9463b99e7f3"). InnerVolumeSpecName "kube-api-access-nth42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:24.903715 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.903687    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"]
Oct 30 01:30:24.907485 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.907466    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-6f9dqfgxk"]
Oct 30 01:30:24.948403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.948379    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4defc5b-55df-4d40-ada5-a9463b99e7f3-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:24.948498 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:24.948407    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nth42\" (UniqueName: \"kubernetes.io/projected/f4defc5b-55df-4d40-ada5-a9463b99e7f3-kube-api-access-nth42\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:25.178538 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.178506    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"]
Oct 30 01:30:25.182403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.182376    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cdcb67694-5tpqt"]
Oct 30 01:30:25.281635 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.281601    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rb6rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 01:30:25.281798 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.281654    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw" podUID="02874d4b-fe79-4247-b420-369003cd1d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 01:30:25.426860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.426777    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c880cf0-b76c-4db8-a26c-d7af8345a80b" path="/var/lib/kubelet/pods/1c880cf0-b76c-4db8-a26c-d7af8345a80b/volumes"
Oct 30 01:30:25.427533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.427511    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e420fa1-2a6e-4e9f-9daf-97b6d031cc26" path="/var/lib/kubelet/pods/2e420fa1-2a6e-4e9f-9daf-97b6d031cc26/volumes"
Oct 30 01:30:25.427967 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.427951    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4defc5b-55df-4d40-ada5-a9463b99e7f3" path="/var/lib/kubelet/pods/f4defc5b-55df-4d40-ada5-a9463b99e7f3/volumes"
Oct 30 01:30:25.526961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.526930    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"]
Oct 30 01:30:25.527689 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527674    4425 cpu_manager.go:414] "containerMap: removing container" podUID="600ad629-faf2-49eb-9d14-05f9cf294799" containerName="payload-processing"
Oct 30 01:30:25.527739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527691    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600ad629-faf2-49eb-9d14-05f9cf294799" containerName="payload-processing"
Oct 30 01:30:25.527739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527707    4425 cpu_manager.go:414] "containerMap: removing container" podUID="be3cc419-c375-4b6b-a359-677d816e907b" containerName="cleanup"
Oct 30 01:30:25.527739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527714    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be3cc419-c375-4b6b-a359-677d816e907b" containerName="cleanup"
Oct 30 01:30:25.527739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527720    4425 cpu_manager.go:414] "containerMap: removing container" podUID="107404d2-96de-452e-8e7d-24acb8246db3" containerName="cleanup"
Oct 30 01:30:25.527739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527725    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107404d2-96de-452e-8e7d-24acb8246db3" containerName="cleanup"
Oct 30 01:30:25.527739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527733    4425 cpu_manager.go:414] "containerMap: removing container" podUID="47e2b0e7-6204-402f-a79c-fa8ff22b3a70" containerName="cleanup"
Oct 30 01:30:25.527739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527738    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e2b0e7-6204-402f-a79c-fa8ff22b3a70" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527747    4425 cpu_manager.go:414] "containerMap: removing container" podUID="fd93d36a-1693-4cbb-bf97-204d5c295c0d" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527751    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd93d36a-1693-4cbb-bf97-204d5c295c0d" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527761    4425 cpu_manager.go:414] "containerMap: removing container" podUID="03eaf7ac-cf67-4da9-b253-3c1994ad57fa" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527766    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03eaf7ac-cf67-4da9-b253-3c1994ad57fa" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527778    4425 cpu_manager.go:414] "containerMap: removing container" podUID="f4defc5b-55df-4d40-ada5-a9463b99e7f3" containerName="payload-processing"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527783    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4defc5b-55df-4d40-ada5-a9463b99e7f3" containerName="payload-processing"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527791    4425 cpu_manager.go:414] "containerMap: removing container" podUID="47e2b0e7-6204-402f-a79c-fa8ff22b3a70" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527795    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e2b0e7-6204-402f-a79c-fa8ff22b3a70" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527804    4425 cpu_manager.go:414] "containerMap: removing container" podUID="be3cc419-c375-4b6b-a359-677d816e907b" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527809    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be3cc419-c375-4b6b-a359-677d816e907b" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527820    4425 cpu_manager.go:414] "containerMap: removing container" podUID="47e2b0e7-6204-402f-a79c-fa8ff22b3a70" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527827    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e2b0e7-6204-402f-a79c-fa8ff22b3a70" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527835    4425 cpu_manager.go:414] "containerMap: removing container" podUID="107404d2-96de-452e-8e7d-24acb8246db3" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527839    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107404d2-96de-452e-8e7d-24acb8246db3" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527846    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c284f618-116f-426d-b25d-fde4ca3c686e" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527850    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c284f618-116f-426d-b25d-fde4ca3c686e" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527860    4425 cpu_manager.go:414] "containerMap: removing container" podUID="4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527866    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527873    4425 cpu_manager.go:414] "containerMap: removing container" podUID="230424d1-69ed-40c3-8afd-660d905f9d8d" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527878    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230424d1-69ed-40c3-8afd-660d905f9d8d" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527887    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c284f618-116f-426d-b25d-fde4ca3c686e" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527892    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c284f618-116f-426d-b25d-fde4ca3c686e" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527902    4425 cpu_manager.go:414] "containerMap: removing container" podUID="1c880cf0-b76c-4db8-a26c-d7af8345a80b" containerName="payload-processing"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527906    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c880cf0-b76c-4db8-a26c-d7af8345a80b" containerName="payload-processing"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527924    4425 cpu_manager.go:414] "containerMap: removing container" podUID="2e420fa1-2a6e-4e9f-9daf-97b6d031cc26" containerName="payload-processing"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527928    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e420fa1-2a6e-4e9f-9daf-97b6d031cc26" containerName="payload-processing"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527936    4425 cpu_manager.go:414] "containerMap: removing container" podUID="230424d1-69ed-40c3-8afd-660d905f9d8d" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527941    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230424d1-69ed-40c3-8afd-660d905f9d8d" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527947    4425 cpu_manager.go:414] "containerMap: removing container" podUID="3699d466-411f-4f43-b62f-e6955340d1da" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527951    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3699d466-411f-4f43-b62f-e6955340d1da" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527958    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b3d49b88-2ba6-4ded-a160-806e4e8fed2b" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527962    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d49b88-2ba6-4ded-a160-806e4e8fed2b" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527968    4425 cpu_manager.go:414] "containerMap: removing container" podUID="4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527972    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527981    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b3d49b88-2ba6-4ded-a160-806e4e8fed2b" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527985    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d49b88-2ba6-4ded-a160-806e4e8fed2b" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527993    4425 cpu_manager.go:414] "containerMap: removing container" podUID="3699d466-411f-4f43-b62f-e6955340d1da" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.527997    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3699d466-411f-4f43-b62f-e6955340d1da" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528004    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b3d49b88-2ba6-4ded-a160-806e4e8fed2b" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528008    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d49b88-2ba6-4ded-a160-806e4e8fed2b" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528015    4425 cpu_manager.go:414] "containerMap: removing container" podUID="4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1" containerName="cleanup"
Oct 30 01:30:25.528006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528020    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528029    4425 cpu_manager.go:414] "containerMap: removing container" podUID="03eaf7ac-cf67-4da9-b253-3c1994ad57fa" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528033    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03eaf7ac-cf67-4da9-b253-3c1994ad57fa" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528040    4425 cpu_manager.go:414] "containerMap: removing container" podUID="3699d466-411f-4f43-b62f-e6955340d1da" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528045    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3699d466-411f-4f43-b62f-e6955340d1da" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528054    4425 cpu_manager.go:414] "containerMap: removing container" podUID="fd93d36a-1693-4cbb-bf97-204d5c295c0d" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528058    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd93d36a-1693-4cbb-bf97-204d5c295c0d" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528174    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="230424d1-69ed-40c3-8afd-660d905f9d8d" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528183    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3699d466-411f-4f43-b62f-e6955340d1da" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528189    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd93d36a-1693-4cbb-bf97-204d5c295c0d" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528195    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="107404d2-96de-452e-8e7d-24acb8246db3" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528200    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d49b88-2ba6-4ded-a160-806e4e8fed2b" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528206    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528211    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d49b88-2ba6-4ded-a160-806e4e8fed2b" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528219    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3699d466-411f-4f43-b62f-e6955340d1da" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528223    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d49b88-2ba6-4ded-a160-806e4e8fed2b" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528231    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="230424d1-69ed-40c3-8afd-660d905f9d8d" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528236    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="be3cc419-c375-4b6b-a359-677d816e907b" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528241    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528247    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="03eaf7ac-cf67-4da9-b253-3c1994ad57fa" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528253    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c284f618-116f-426d-b25d-fde4ca3c686e" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528258    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="600ad629-faf2-49eb-9d14-05f9cf294799" containerName="payload-processing"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528264    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="be3cc419-c375-4b6b-a359-677d816e907b" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528268    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="107404d2-96de-452e-8e7d-24acb8246db3" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528274    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e2b0e7-6204-402f-a79c-fa8ff22b3a70" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528280    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd93d36a-1693-4cbb-bf97-204d5c295c0d" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528287    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4defc5b-55df-4d40-ada5-a9463b99e7f3" containerName="payload-processing"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528293    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="be3cc419-c375-4b6b-a359-677d816e907b" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528299    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e2b0e7-6204-402f-a79c-fa8ff22b3a70" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528303    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="107404d2-96de-452e-8e7d-24acb8246db3" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528309    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c284f618-116f-426d-b25d-fde4ca3c686e" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528317    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c284f618-116f-426d-b25d-fde4ca3c686e" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528323    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c880cf0-b76c-4db8-a26c-d7af8345a80b" containerName="payload-processing"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528330    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="03eaf7ac-cf67-4da9-b253-3c1994ad57fa" containerName="cleanup"
Oct 30 01:30:25.529260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.528337    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e420fa1-2a6e-4e9f-9daf-97b6d031cc26" containerName="payload-processing"
Oct 30 01:30:25.533615 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.533599    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:30:25.546866 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.546804    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"agc-cred-75b298c27b49\""
Oct 30 01:30:25.547040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.547017    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"routing-overlay\""
Oct 30 01:30:25.547040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.547026    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"openshift-service-ca.crt\""
Oct 30 01:30:25.547233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.547044    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"kube-root-ca.crt\""
Oct 30 01:30:25.547233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.547219    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ae1820\""
Oct 30 01:30:25.547233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.547227    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"payload-processing-external-model-e2e--53969a13-dockercfg-24pcd\""
Oct 30 01:30:25.583209 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.562610    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"]
Oct 30 01:30:25.583209 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.583069    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7877a3-3659-4043-bc54-ffdd6e9f2369-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s\" (UID: \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:30:25.583209 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.583125    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d7877a3-3659-4043-bc54-ffdd6e9f2369-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s\" (UID: \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:30:25.583509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.583334    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d7877a3-3659-4043-bc54-ffdd6e9f2369-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s\" (UID: \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:30:25.583509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.583405    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqbsc\" (UniqueName: \"kubernetes.io/projected/9d7877a3-3659-4043-bc54-ffdd6e9f2369-kube-api-access-hqbsc\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s\" (UID: \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:30:25.685334 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.685246    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7877a3-3659-4043-bc54-ffdd6e9f2369-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s\" (UID: \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:30:25.685334 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.685298    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d7877a3-3659-4043-bc54-ffdd6e9f2369-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s\" (UID: \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:30:25.685515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.685349    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d7877a3-3659-4043-bc54-ffdd6e9f2369-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s\" (UID: \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:30:25.685515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.685382    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqbsc\" (UniqueName: \"kubernetes.io/projected/9d7877a3-3659-4043-bc54-ffdd6e9f2369-kube-api-access-hqbsc\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s\" (UID: \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:30:25.686008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.685982    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d7877a3-3659-4043-bc54-ffdd6e9f2369-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s\" (UID: \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:30:25.686108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.686092    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7877a3-3659-4043-bc54-ffdd6e9f2369-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s\" (UID: \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:30:25.687831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.687810    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d7877a3-3659-4043-bc54-ffdd6e9f2369-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s\" (UID: \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:30:25.707956 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.707926    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqbsc\" (UniqueName: \"kubernetes.io/projected/9d7877a3-3659-4043-bc54-ffdd6e9f2369-kube-api-access-hqbsc\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s\" (UID: \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:30:25.848260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:25.848231    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:30:26.198142 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.198108    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"]
Oct 30 01:30:26.281139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.281105    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rb6rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 01:30:26.281315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.281184    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw" podUID="02874d4b-fe79-4247-b420-369003cd1d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 01:30:26.803137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.803098    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"]
Oct 30 01:30:26.803974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.803951    4425 cpu_manager.go:414] "containerMap: removing container" podUID="fd93d36a-1693-4cbb-bf97-204d5c295c0d" containerName="cleanup"
Oct 30 01:30:26.803974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.803972    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd93d36a-1693-4cbb-bf97-204d5c295c0d" containerName="cleanup"
Oct 30 01:30:26.804113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.803987    4425 cpu_manager.go:414] "containerMap: removing container" podUID="03eaf7ac-cf67-4da9-b253-3c1994ad57fa" containerName="cleanup"
Oct 30 01:30:26.804113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.803993    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03eaf7ac-cf67-4da9-b253-3c1994ad57fa" containerName="cleanup"
Oct 30 01:30:26.804113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.804006    4425 cpu_manager.go:414] "containerMap: removing container" podUID="be3cc419-c375-4b6b-a359-677d816e907b" containerName="cleanup"
Oct 30 01:30:26.804113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.804011    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be3cc419-c375-4b6b-a359-677d816e907b" containerName="cleanup"
Oct 30 01:30:26.804113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.804020    4425 cpu_manager.go:414] "containerMap: removing container" podUID="107404d2-96de-452e-8e7d-24acb8246db3" containerName="cleanup"
Oct 30 01:30:26.804113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.804025    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107404d2-96de-452e-8e7d-24acb8246db3" containerName="cleanup"
Oct 30 01:30:26.804113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.804039    4425 cpu_manager.go:414] "containerMap: removing container" podUID="230424d1-69ed-40c3-8afd-660d905f9d8d" containerName="cleanup"
Oct 30 01:30:26.804113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.804043    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230424d1-69ed-40c3-8afd-660d905f9d8d" containerName="cleanup"
Oct 30 01:30:26.804113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.804055    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c284f618-116f-426d-b25d-fde4ca3c686e" containerName="cleanup"
Oct 30 01:30:26.804113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.804060    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c284f618-116f-426d-b25d-fde4ca3c686e" containerName="cleanup"
Oct 30 01:30:26.804550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.804211    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e2b0e7-6204-402f-a79c-fa8ff22b3a70" containerName="cleanup"
Oct 30 01:30:26.804550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.804223    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="03eaf7ac-cf67-4da9-b253-3c1994ad57fa" containerName="cleanup"
Oct 30 01:30:26.804550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.804231    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd93d36a-1693-4cbb-bf97-204d5c295c0d" containerName="cleanup"
Oct 30 01:30:26.804550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.804238    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3699d466-411f-4f43-b62f-e6955340d1da" containerName="cleanup"
Oct 30 01:30:26.804550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.804245    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="230424d1-69ed-40c3-8afd-660d905f9d8d" containerName="cleanup"
Oct 30 01:30:26.804550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.804251    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ded5d10-ee2b-4ef7-a4a7-9aceb229a7c1" containerName="cleanup"
Oct 30 01:30:26.808335 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.808317    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:30:26.821302 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.821284    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"kube-root-ca.crt\""
Oct 30 01:30:26.823966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.823946    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"agc-cred-5ffddfb0a62f\""
Oct 30 01:30:26.828487 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.828467    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"]
Oct 30 01:30:26.848280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.847894    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s" event={"ID":"9d7877a3-3659-4043-bc54-ffdd6e9f2369","Type":"ContainerStarted","Data":"0f5fec64245f1ef503b78f1ff7376385f596ecd6af1ecbb2906642ae4ff37005"}
Oct 30 01:30:26.848280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.847969    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s" event={"ID":"9d7877a3-3659-4043-bc54-ffdd6e9f2369","Type":"ContainerStarted","Data":"a974ef2b53524c0821006daef03d958a7e18554eaaff795bd1b8fe5d465efeae"}
Oct 30 01:30:26.848280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.848087    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:30:26.901889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.901806    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s" podStartSLOduration=1.9017908289999998 podStartE2EDuration="1.901790829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:26.899567063 +0000 UTC m=+2935.649222372" watchObservedRunningTime="2026-10-30 01:30:26.901790829 +0000 UTC m=+2935.651446137"
Oct 30 01:30:26.902501 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.902472    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bdf5\" (UniqueName: \"kubernetes.io/projected/75600e6d-1dcf-4c52-b072-31666c67f51e-kube-api-access-6bdf5\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n\" (UID: \"75600e6d-1dcf-4c52-b072-31666c67f51e\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:30:26.902899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.902867    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75600e6d-1dcf-4c52-b072-31666c67f51e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n\" (UID: \"75600e6d-1dcf-4c52-b072-31666c67f51e\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:30:26.903054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.902959    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75600e6d-1dcf-4c52-b072-31666c67f51e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n\" (UID: \"75600e6d-1dcf-4c52-b072-31666c67f51e\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:30:26.903054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.903020    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75600e6d-1dcf-4c52-b072-31666c67f51e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n\" (UID: \"75600e6d-1dcf-4c52-b072-31666c67f51e\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:30:26.921105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.921080    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"openshift-service-ca.crt\""
Oct 30 01:30:26.921261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.921243    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a47b9a\""
Oct 30 01:30:26.921376 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:26.921362    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a47b9a\"/\"routing-overlay\""
Oct 30 01:30:27.005426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:27.005397    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bdf5\" (UniqueName: \"kubernetes.io/projected/75600e6d-1dcf-4c52-b072-31666c67f51e-kube-api-access-6bdf5\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n\" (UID: \"75600e6d-1dcf-4c52-b072-31666c67f51e\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:30:27.005575 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:27.005556    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75600e6d-1dcf-4c52-b072-31666c67f51e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n\" (UID: \"75600e6d-1dcf-4c52-b072-31666c67f51e\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:30:27.005633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:27.005614    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75600e6d-1dcf-4c52-b072-31666c67f51e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n\" (UID: \"75600e6d-1dcf-4c52-b072-31666c67f51e\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:30:27.005685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:27.005656    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75600e6d-1dcf-4c52-b072-31666c67f51e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n\" (UID: \"75600e6d-1dcf-4c52-b072-31666c67f51e\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:30:27.006536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:27.006509    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75600e6d-1dcf-4c52-b072-31666c67f51e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n\" (UID: \"75600e6d-1dcf-4c52-b072-31666c67f51e\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:30:27.006669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:27.006594    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75600e6d-1dcf-4c52-b072-31666c67f51e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n\" (UID: \"75600e6d-1dcf-4c52-b072-31666c67f51e\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:30:27.008106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:27.008085    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75600e6d-1dcf-4c52-b072-31666c67f51e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n\" (UID: \"75600e6d-1dcf-4c52-b072-31666c67f51e\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:30:27.026326 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:27.026295    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bdf5\" (UniqueName: \"kubernetes.io/projected/75600e6d-1dcf-4c52-b072-31666c67f51e-kube-api-access-6bdf5\") pod \"payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n\" (UID: \"75600e6d-1dcf-4c52-b072-31666c67f51e\") " pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:30:27.124002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:27.123870    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:30:27.281724 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:27.281688    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rb6rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 01:30:27.281891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:27.281739    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw" podUID="02874d4b-fe79-4247-b420-369003cd1d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 01:30:27.441775 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:27.441747    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"]
Oct 30 01:30:27.442908 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:27.442879    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75600e6d_1dcf_4c52_b072_31666c67f51e.slice/crio-b54ea15dc80d44147fd41eff5f6f8012775376079539b18edafa447c9a191cf5 WatchSource:0}: Error finding container b54ea15dc80d44147fd41eff5f6f8012775376079539b18edafa447c9a191cf5: Status 404 returned error can't find the container with id b54ea15dc80d44147fd41eff5f6f8012775376079539b18edafa447c9a191cf5
Oct 30 01:30:27.860605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:27.860567    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n" event={"ID":"75600e6d-1dcf-4c52-b072-31666c67f51e","Type":"ContainerStarted","Data":"229180e69a8030fdef34548540e475cf2649477f01e84c96cb639828bdf9bfd7"}
Oct 30 01:30:27.861048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:27.860634    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n" event={"ID":"75600e6d-1dcf-4c52-b072-31666c67f51e","Type":"ContainerStarted","Data":"b54ea15dc80d44147fd41eff5f6f8012775376079539b18edafa447c9a191cf5"}
Oct 30 01:30:27.900955 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:27.900898    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n" podStartSLOduration=1.90088284 podStartE2EDuration="1.90088284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:27.897857696 +0000 UTC m=+2936.647513017" watchObservedRunningTime="2026-10-30 01:30:27.90088284 +0000 UTC m=+2936.650538174"
Oct 30 01:30:28.281532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:28.281492    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-rb6rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 01:30:28.281707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:28.281556    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw" podUID="02874d4b-fe79-4247-b420-369003cd1d23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 01:30:28.871785 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:28.871740    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:30:29.667811 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:29.667779    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:29.668420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:29.668390    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:29.669469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:29.669449    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:30:31.170320 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:31.170275    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:30:31.743806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:31.743778    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:30:31.823896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:31.823862    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"]
Oct 30 01:30:34.948855 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:34.948826    4425 scope.go:122] "RemoveContainer" containerID="78ef4e410587a000a0c46cd5d192c61181bb8e3da112e49f098803613a9fa1f6"
Oct 30 01:30:35.029758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:35.029734    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"]
Oct 30 01:30:35.037865 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:35.037841    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a770f91a-kxgw7"]
Oct 30 01:30:35.425314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:35.425281    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6a04a93-2a01-43a4-b392-f3779c455217" path="/var/lib/kubelet/pods/d6a04a93-2a01-43a4-b392-f3779c455217/volumes"
Oct 30 01:30:36.824518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:36.824460    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n" podUID="d435c726-6e32-4624-86ec-fa093d531c40" containerName="payload-processing" containerID="cri-o://ac2b3961a85e773253681e5a26a40824176021adb175792c5461516fc1ea6085" gracePeriod=25
Oct 30 01:30:36.955107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:36.955077    4425 generic.go:356] "Generic (PLEG): container finished" podID="d435c726-6e32-4624-86ec-fa093d531c40" containerID="ac2b3961a85e773253681e5a26a40824176021adb175792c5461516fc1ea6085" exitCode=0
Oct 30 01:30:36.955237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:36.955178    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n" event={"ID":"d435c726-6e32-4624-86ec-fa093d531c40","Type":"ContainerDied","Data":"ac2b3961a85e773253681e5a26a40824176021adb175792c5461516fc1ea6085"}
Oct 30 01:30:37.105111 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:37.105091    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:30:37.267499 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:37.267465    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d435c726-6e32-4624-86ec-fa093d531c40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d435c726-6e32-4624-86ec-fa093d531c40-plugins-config-volume\") pod \"d435c726-6e32-4624-86ec-fa093d531c40\" (UID: \"d435c726-6e32-4624-86ec-fa093d531c40\") "
Oct 30 01:30:37.267706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:37.267690    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d435c726-6e32-4624-86ec-fa093d531c40-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d435c726-6e32-4624-86ec-fa093d531c40-routing-overlay-volume\") pod \"d435c726-6e32-4624-86ec-fa093d531c40\" (UID: \"d435c726-6e32-4624-86ec-fa093d531c40\") "
Oct 30 01:30:37.267789 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:37.267727    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d435c726-6e32-4624-86ec-fa093d531c40-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d435c726-6e32-4624-86ec-fa093d531c40-provider-credentials\") pod \"d435c726-6e32-4624-86ec-fa093d531c40\" (UID: \"d435c726-6e32-4624-86ec-fa093d531c40\") "
Oct 30 01:30:37.267789 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:37.267772    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d435c726-6e32-4624-86ec-fa093d531c40-kube-api-access-ggxrt\" (UniqueName: \"kubernetes.io/projected/d435c726-6e32-4624-86ec-fa093d531c40-kube-api-access-ggxrt\") pod \"d435c726-6e32-4624-86ec-fa093d531c40\" (UID: \"d435c726-6e32-4624-86ec-fa093d531c40\") "
Oct 30 01:30:37.268040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:37.267997    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d435c726-6e32-4624-86ec-fa093d531c40-plugins-config-volume" pod "d435c726-6e32-4624-86ec-fa093d531c40" (UID: "d435c726-6e32-4624-86ec-fa093d531c40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:37.268188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:37.268040    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d435c726-6e32-4624-86ec-fa093d531c40-routing-overlay-volume" pod "d435c726-6e32-4624-86ec-fa093d531c40" (UID: "d435c726-6e32-4624-86ec-fa093d531c40"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:37.268281 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:37.268267    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d435c726-6e32-4624-86ec-fa093d531c40-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:37.268325 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:37.268284    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d435c726-6e32-4624-86ec-fa093d531c40-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:37.269918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:37.269900    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d435c726-6e32-4624-86ec-fa093d531c40-provider-credentials" pod "d435c726-6e32-4624-86ec-fa093d531c40" (UID: "d435c726-6e32-4624-86ec-fa093d531c40"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:37.269991 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:37.269922    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d435c726-6e32-4624-86ec-fa093d531c40-kube-api-access-ggxrt" pod "d435c726-6e32-4624-86ec-fa093d531c40" (UID: "d435c726-6e32-4624-86ec-fa093d531c40"). InnerVolumeSpecName "kube-api-access-ggxrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:37.370060 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:37.369999    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d435c726-6e32-4624-86ec-fa093d531c40-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:37.370060 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:37.370024    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggxrt\" (UniqueName: \"kubernetes.io/projected/d435c726-6e32-4624-86ec-fa093d531c40-kube-api-access-ggxrt\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:30:37.555702 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:30:37.555668    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:30:37.864691 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:37.864663    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:30:37.964516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:37.964481    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"
Oct 30 01:30:37.964516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:37.964503    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n" event={"ID":"d435c726-6e32-4624-86ec-fa093d531c40","Type":"ContainerDied","Data":"315d2009a95a3feb24710e4f1e0d5519112f3f1e1f3cf83e5c30a197560a476d"}
Oct 30 01:30:37.964719 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:37.964570    4425 scope.go:122] "RemoveContainer" containerID="ac2b3961a85e773253681e5a26a40824176021adb175792c5461516fc1ea6085"
Oct 30 01:30:38.032392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:38.032357    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"]
Oct 30 01:30:38.038953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:38.038925    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f6fbbf746-8j44n"]
Oct 30 01:30:39.425090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:39.425057    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d435c726-6e32-4624-86ec-fa093d531c40" path="/var/lib/kubelet/pods/d435c726-6e32-4624-86ec-fa093d531c40/volumes"
Oct 30 01:30:39.885173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:39.885129    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:30:43.027427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.027395    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"]
Oct 30 01:30:43.028223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.028207    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d435c726-6e32-4624-86ec-fa093d531c40" containerName="payload-processing"
Oct 30 01:30:43.028223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.028223    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d435c726-6e32-4624-86ec-fa093d531c40" containerName="payload-processing"
Oct 30 01:30:43.028360 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.028351    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d435c726-6e32-4624-86ec-fa093d531c40" containerName="payload-processing"
Oct 30 01:30:43.034236 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.034219    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"
Oct 30 01:30:43.046914 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.046885    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-5e8b4d\"/\"e2e-usage-sim-055d3c-cert\""
Oct 30 01:30:43.047042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.046960    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-5e8b4d\"/\"default-dockercfg-mrnqm\""
Oct 30 01:30:43.050689 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.050665    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"]
Oct 30 01:30:43.055753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.055731    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-5e8b4d\"/\"openshift-service-ca.crt\""
Oct 30 01:30:43.055753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.055743    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-5e8b4d\"/\"kube-root-ca.crt\""
Oct 30 01:30:43.140308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.140283    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c-tls\") pod \"e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx\" (UID: \"ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c\") " pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"
Oct 30 01:30:43.140430 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.140384    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk29q\" (UniqueName: \"kubernetes.io/projected/ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c-kube-api-access-dk29q\") pod \"e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx\" (UID: \"ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c\") " pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"
Oct 30 01:30:43.242348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.242321    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c-tls\") pod \"e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx\" (UID: \"ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c\") " pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"
Oct 30 01:30:43.242469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.242381    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk29q\" (UniqueName: \"kubernetes.io/projected/ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c-kube-api-access-dk29q\") pod \"e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx\" (UID: \"ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c\") " pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"
Oct 30 01:30:43.244814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.244794    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c-tls\") pod \"e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx\" (UID: \"ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c\") " pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"
Oct 30 01:30:43.263756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.263727    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk29q\" (UniqueName: \"kubernetes.io/projected/ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c-kube-api-access-dk29q\") pod \"e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx\" (UID: \"ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c\") " pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"
Oct 30 01:30:43.347600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.347526    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"
Oct 30 01:30:43.652122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:43.652092    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"]
Oct 30 01:30:43.652582 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:43.652557    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba9d3959_08ef_4607_bf1e_4e29c3ec3d0c.slice/crio-591906ae6f4795965504b5955eb7773158ce874405db9c14801dd1533e0feda4 WatchSource:0}: Error finding container 591906ae6f4795965504b5955eb7773158ce874405db9c14801dd1533e0feda4: Status 404 returned error can't find the container with id 591906ae6f4795965504b5955eb7773158ce874405db9c14801dd1533e0feda4
Oct 30 01:30:44.013028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:44.012996    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx" event={"ID":"ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c","Type":"ContainerStarted","Data":"9c76dcedd3e972584b7bd4f7a2ea81ab8aa4a19dab0132860b0ff7d8f1708bda"}
Oct 30 01:30:44.013233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:44.013042    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx" event={"ID":"ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c","Type":"ContainerStarted","Data":"591906ae6f4795965504b5955eb7773158ce874405db9c14801dd1533e0feda4"}
Oct 30 01:30:44.013307 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:44.013273    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"
Oct 30 01:30:44.017903 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:44.017880    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"
Oct 30 01:30:44.048754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:44.048702    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx" podStartSLOduration=1.04868382 podStartE2EDuration="1.04868382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:44.047307056 +0000 UTC m=+2952.796962365" watchObservedRunningTime="2026-10-30 01:30:44.04868382 +0000 UTC m=+2952.798339130"
Oct 30 01:30:49.893800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:49.893758    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"]
Oct 30 01:30:49.898938 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:49.898915    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:49.928192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:49.928167    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"]
Oct 30 01:30:50.026446 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.026410    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.026627 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.026453    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81dd80d6-4327-4622-937a-39c52afd0fef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.026627 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.026557    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.026627 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.026608    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.026769 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.026671    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.026769 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.026728    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81dd80d6-4327-4622-937a-39c52afd0fef-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.026884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.026867    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.026932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.026902    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81dd80d6-4327-4622-937a-39c52afd0fef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.026984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.026965    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2wlm\" (UniqueName: \"kubernetes.io/projected/81dd80d6-4327-4622-937a-39c52afd0fef-kube-api-access-r2wlm\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.128685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.128653    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.128867 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.128697    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81dd80d6-4327-4622-937a-39c52afd0fef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.128867 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.128730    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2wlm\" (UniqueName: \"kubernetes.io/projected/81dd80d6-4327-4622-937a-39c52afd0fef-kube-api-access-r2wlm\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.128867 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.128782    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.128867 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.128808    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81dd80d6-4327-4622-937a-39c52afd0fef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.129125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.128871    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.129125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.128920    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.129125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.128945    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.129125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.128981    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81dd80d6-4327-4622-937a-39c52afd0fef-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.129391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.129254    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.129391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.129336    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.129513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.129376    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.129513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.129485    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.129600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.129528    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81dd80d6-4327-4622-937a-39c52afd0fef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.131213 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.131192    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.131357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.131338    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81dd80d6-4327-4622-937a-39c52afd0fef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.149542 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.149483    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81dd80d6-4327-4622-937a-39c52afd0fef-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.149697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.149678    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2wlm\" (UniqueName: \"kubernetes.io/projected/81dd80d6-4327-4622-937a-39c52afd0fef-kube-api-access-r2wlm\") pod \"maas-default-gateway-openshift-default-fcb75c45f-kdm6l\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.215359 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.215317    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:50.544916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.544884    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"]
Oct 30 01:30:50.546742 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:50.546716    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81dd80d6_4327_4622_937a_39c52afd0fef.slice/crio-daf830440a9fd65e1e78ce4024acd47f6703db5b68f749dfdcefd6c19e3a8eb7 WatchSource:0}: Error finding container daf830440a9fd65e1e78ce4024acd47f6703db5b68f749dfdcefd6c19e3a8eb7: Status 404 returned error can't find the container with id daf830440a9fd65e1e78ce4024acd47f6703db5b68f749dfdcefd6c19e3a8eb7
Oct 30 01:30:50.549425 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.549393    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:30:50.549540 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.549459    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:30:50.549540 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:50.549490    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:30:51.087401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.087358    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l" event={"ID":"81dd80d6-4327-4622-937a-39c52afd0fef","Type":"ContainerStarted","Data":"a431e23bc8d7f2b885d05ebec4840e68539984606077db3234af4e3a8bd0bc56"}
Oct 30 01:30:51.088053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.088020    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l" event={"ID":"81dd80d6-4327-4622-937a-39c52afd0fef","Type":"ContainerStarted","Data":"daf830440a9fd65e1e78ce4024acd47f6703db5b68f749dfdcefd6c19e3a8eb7"}
Oct 30 01:30:51.140468 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.140406    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l" podStartSLOduration=2.140387304 podStartE2EDuration="2.140387304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:51.135768934 +0000 UTC m=+2959.885424245" watchObservedRunningTime="2026-10-30 01:30:51.140387304 +0000 UTC m=+2959.890042606"
Oct 30 01:30:51.216321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.216283    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:51.218273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.218228    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-kdm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 30 01:30:51.218427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.218329    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l" podUID="81dd80d6-4327-4622-937a-39c52afd0fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 30 01:30:51.220875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.220850    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"]
Oct 30 01:30:51.225792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.225774    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:30:51.238690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.238658    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-499482\"/\"agc-cred-ce13ccbf6751\""
Oct 30 01:30:51.238995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.238833    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-499482\"/\"kube-root-ca.crt\""
Oct 30 01:30:51.238995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.238911    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-499482\"/\"payload-processing-external-model-e2e--b4b1dd99-dockercfg-9scwk\""
Oct 30 01:30:51.238995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.238956    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-499482\"/\"openshift-service-ca.crt\""
Oct 30 01:30:51.323176 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.308946    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-499482\"/\"payload-processing-external-model-plugins-e2e-extmod-a-499482\""
Oct 30 01:30:51.323176 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.309068    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-499482\"/\"routing-overlay\""
Oct 30 01:30:51.323176 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.317877    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"]
Oct 30 01:30:51.419935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.412886    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5077895f-c887-4fac-a91e-562c5311ece8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-57c726bdp\" (UID: \"5077895f-c887-4fac-a91e-562c5311ece8\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:30:51.419935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.412963    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d6sg\" (UniqueName: \"kubernetes.io/projected/5077895f-c887-4fac-a91e-562c5311ece8-kube-api-access-6d6sg\") pod \"payload-processing-external-model-e2e-extmod-a-499482-57c726bdp\" (UID: \"5077895f-c887-4fac-a91e-562c5311ece8\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:30:51.419935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.413092    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5077895f-c887-4fac-a91e-562c5311ece8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-499482-57c726bdp\" (UID: \"5077895f-c887-4fac-a91e-562c5311ece8\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:30:51.419935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.413139    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5077895f-c887-4fac-a91e-562c5311ece8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-57c726bdp\" (UID: \"5077895f-c887-4fac-a91e-562c5311ece8\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:30:51.514775 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.514737    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5077895f-c887-4fac-a91e-562c5311ece8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-499482-57c726bdp\" (UID: \"5077895f-c887-4fac-a91e-562c5311ece8\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:30:51.514958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.514788    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5077895f-c887-4fac-a91e-562c5311ece8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-57c726bdp\" (UID: \"5077895f-c887-4fac-a91e-562c5311ece8\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:30:51.514958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.514911    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5077895f-c887-4fac-a91e-562c5311ece8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-57c726bdp\" (UID: \"5077895f-c887-4fac-a91e-562c5311ece8\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:30:51.514958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.514942    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d6sg\" (UniqueName: \"kubernetes.io/projected/5077895f-c887-4fac-a91e-562c5311ece8-kube-api-access-6d6sg\") pod \"payload-processing-external-model-e2e-extmod-a-499482-57c726bdp\" (UID: \"5077895f-c887-4fac-a91e-562c5311ece8\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:30:51.515518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.515493    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5077895f-c887-4fac-a91e-562c5311ece8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-57c726bdp\" (UID: \"5077895f-c887-4fac-a91e-562c5311ece8\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:30:51.515658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.515638    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5077895f-c887-4fac-a91e-562c5311ece8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-499482-57c726bdp\" (UID: \"5077895f-c887-4fac-a91e-562c5311ece8\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:30:51.517462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.517435    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5077895f-c887-4fac-a91e-562c5311ece8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-499482-57c726bdp\" (UID: \"5077895f-c887-4fac-a91e-562c5311ece8\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:30:51.536054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.536030    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d6sg\" (UniqueName: \"kubernetes.io/projected/5077895f-c887-4fac-a91e-562c5311ece8-kube-api-access-6d6sg\") pod \"payload-processing-external-model-e2e-extmod-a-499482-57c726bdp\" (UID: \"5077895f-c887-4fac-a91e-562c5311ece8\") " pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:30:51.544020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.543999    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:30:51.886496 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:51.886458    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5077895f_c887_4fac_a91e_562c5311ece8.slice/crio-d6ba7968ee3079b4c84250234b3fdca510e7a135d50653d24efdf137d66f3a75 WatchSource:0}: Error finding container d6ba7968ee3079b4c84250234b3fdca510e7a135d50653d24efdf137d66f3a75: Status 404 returned error can't find the container with id d6ba7968ee3079b4c84250234b3fdca510e7a135d50653d24efdf137d66f3a75
Oct 30 01:30:51.887677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:51.887647    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"]
Oct 30 01:30:52.095956 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.095923    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp" event={"ID":"5077895f-c887-4fac-a91e-562c5311ece8","Type":"ContainerStarted","Data":"3ac7b90c9f73f21fed9b632cd3dbada536c6ba08ce1fe4d5f8160ab86e062338"}
Oct 30 01:30:52.096372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.095979    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp" event={"ID":"5077895f-c887-4fac-a91e-562c5311ece8","Type":"ContainerStarted","Data":"d6ba7968ee3079b4c84250234b3fdca510e7a135d50653d24efdf137d66f3a75"}
Oct 30 01:30:52.096372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.096007    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:30:52.136911 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.136815    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp" podStartSLOduration=1.136801543 podStartE2EDuration="1.136801543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:52.131794022 +0000 UTC m=+2960.881449332" watchObservedRunningTime="2026-10-30 01:30:52.136801543 +0000 UTC m=+2960.886456852"
Oct 30 01:30:52.180832 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.180798    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"]
Oct 30 01:30:52.185805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.185781    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:30:52.201844 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.201817    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dd80cb\"/\"agc-cred-d99b3f64a2d5\""
Oct 30 01:30:52.201994 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.201913    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dd80cb\"/\"routing-overlay\""
Oct 30 01:30:52.202078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.201988    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dd80cb\"/\"openshift-service-ca.crt\""
Oct 30 01:30:52.202078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.202012    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dd80cb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-dd80cb\""
Oct 30 01:30:52.202240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.202092    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-dd80cb\"/\"kube-root-ca.crt\""
Oct 30 01:30:52.202240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.202128    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-dd80cb\"/\"payload-processing-external-model-e2e--871060f7-dockercfg-7bfq8\""
Oct 30 01:30:52.206727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.206704    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"]
Oct 30 01:30:52.216682 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.216656    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-kdm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 30 01:30:52.216786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.216701    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l" podUID="81dd80d6-4327-4622-937a-39c52afd0fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 30 01:30:52.326292 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.326256    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad95246c-631e-459f-aef2-83d8303f6593-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54\" (UID: \"ad95246c-631e-459f-aef2-83d8303f6593\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:30:52.326517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.326405    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad95246c-631e-459f-aef2-83d8303f6593-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54\" (UID: \"ad95246c-631e-459f-aef2-83d8303f6593\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:30:52.326517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.326504    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad95246c-631e-459f-aef2-83d8303f6593-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54\" (UID: \"ad95246c-631e-459f-aef2-83d8303f6593\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:30:52.326626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.326587    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9q9c\" (UniqueName: \"kubernetes.io/projected/ad95246c-631e-459f-aef2-83d8303f6593-kube-api-access-h9q9c\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54\" (UID: \"ad95246c-631e-459f-aef2-83d8303f6593\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:30:52.428464 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.428364    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad95246c-631e-459f-aef2-83d8303f6593-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54\" (UID: \"ad95246c-631e-459f-aef2-83d8303f6593\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:30:52.428464 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.428438    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9q9c\" (UniqueName: \"kubernetes.io/projected/ad95246c-631e-459f-aef2-83d8303f6593-kube-api-access-h9q9c\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54\" (UID: \"ad95246c-631e-459f-aef2-83d8303f6593\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:30:52.428678 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.428567    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad95246c-631e-459f-aef2-83d8303f6593-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54\" (UID: \"ad95246c-631e-459f-aef2-83d8303f6593\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:30:52.428725 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.428699    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad95246c-631e-459f-aef2-83d8303f6593-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54\" (UID: \"ad95246c-631e-459f-aef2-83d8303f6593\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:30:52.429283 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.429252    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad95246c-631e-459f-aef2-83d8303f6593-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54\" (UID: \"ad95246c-631e-459f-aef2-83d8303f6593\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:30:52.429429 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.429405    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad95246c-631e-459f-aef2-83d8303f6593-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54\" (UID: \"ad95246c-631e-459f-aef2-83d8303f6593\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:30:52.430941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.430918    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad95246c-631e-459f-aef2-83d8303f6593-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54\" (UID: \"ad95246c-631e-459f-aef2-83d8303f6593\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:30:52.467183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.465904    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9q9c\" (UniqueName: \"kubernetes.io/projected/ad95246c-631e-459f-aef2-83d8303f6593-kube-api-access-h9q9c\") pod \"payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54\" (UID: \"ad95246c-631e-459f-aef2-83d8303f6593\") " pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:30:52.501666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.501634    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:30:52.837896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:52.837862    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"]
Oct 30 01:30:52.838102 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:52.838080    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad95246c_631e_459f_aef2_83d8303f6593.slice/crio-2bd9a2d2d0810a4f034c8aa7e8bf772e55e71242aac040314aa76db9ca2072b5 WatchSource:0}: Error finding container 2bd9a2d2d0810a4f034c8aa7e8bf772e55e71242aac040314aa76db9ca2072b5: Status 404 returned error can't find the container with id 2bd9a2d2d0810a4f034c8aa7e8bf772e55e71242aac040314aa76db9ca2072b5
Oct 30 01:30:53.105233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:53.105123    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54" event={"ID":"ad95246c-631e-459f-aef2-83d8303f6593","Type":"ContainerStarted","Data":"9fa6308b4394b01643353a3d4b7414107c599e8cf409f570b0ac720ea14e493e"}
Oct 30 01:30:53.105678 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:53.105568    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:30:53.105678 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:53.105610    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54" event={"ID":"ad95246c-631e-459f-aef2-83d8303f6593","Type":"ContainerStarted","Data":"2bd9a2d2d0810a4f034c8aa7e8bf772e55e71242aac040314aa76db9ca2072b5"}
Oct 30 01:30:53.142964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:53.142920    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54" podStartSLOduration=1.142907531 podStartE2EDuration="1.142907531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:53.141549214 +0000 UTC m=+2961.891204547" watchObservedRunningTime="2026-10-30 01:30:53.142907531 +0000 UTC m=+2961.892562839"
Oct 30 01:30:53.216324 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:53.216278    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-kdm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 30 01:30:53.216473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:53.216339    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l" podUID="81dd80d6-4327-4622-937a-39c52afd0fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 30 01:30:54.216460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:54.216421    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-kdm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 30 01:30:54.216942 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:54.216473    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l" podUID="81dd80d6-4327-4622-937a-39c52afd0fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 30 01:30:55.216524 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:55.216485    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-kdm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 30 01:30:55.217029 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:55.216541    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l" podUID="81dd80d6-4327-4622-937a-39c52afd0fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 30 01:30:56.216705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:56.216668    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-kdm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 30 01:30:56.217234 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:56.216727    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l" podUID="81dd80d6-4327-4622-937a-39c52afd0fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 30 01:30:56.834759 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:56.834693    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"]
Oct 30 01:30:56.840316 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:56.840293    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:30:56.859605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:56.859578    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"]
Oct 30 01:30:56.864331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:56.864308    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"openshift-service-ca.crt\""
Oct 30 01:30:56.864478 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:56.864372    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"kube-root-ca.crt\""
Oct 30 01:30:56.864478 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:56.864396    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"routing-overlay\""
Oct 30 01:30:56.864478 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:56.864309    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c7bba9\""
Oct 30 01:30:56.864627 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:56.864497    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"agc-cred-6fb94fa6b5f6\""
Oct 30 01:30:56.864627 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:56.864516    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"payload-processing-external-model-e2e--42876bb9-dockercfg-snpkn\""
Oct 30 01:30:56.940921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:56.940873    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/049bac41-91c6-4773-8cf0-7c8532987208-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt\" (UID: \"049bac41-91c6-4773-8cf0-7c8532987208\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:30:56.941067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:56.940936    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/049bac41-91c6-4773-8cf0-7c8532987208-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt\" (UID: \"049bac41-91c6-4773-8cf0-7c8532987208\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:30:56.941109 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:56.941074    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049bac41-91c6-4773-8cf0-7c8532987208-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt\" (UID: \"049bac41-91c6-4773-8cf0-7c8532987208\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:30:56.941144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:56.941132    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m86vd\" (UniqueName: \"kubernetes.io/projected/049bac41-91c6-4773-8cf0-7c8532987208-kube-api-access-m86vd\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt\" (UID: \"049bac41-91c6-4773-8cf0-7c8532987208\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:30:57.042595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.042554    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049bac41-91c6-4773-8cf0-7c8532987208-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt\" (UID: \"049bac41-91c6-4773-8cf0-7c8532987208\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:30:57.042804 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.042667    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m86vd\" (UniqueName: \"kubernetes.io/projected/049bac41-91c6-4773-8cf0-7c8532987208-kube-api-access-m86vd\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt\" (UID: \"049bac41-91c6-4773-8cf0-7c8532987208\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:30:57.042804 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.042780    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/049bac41-91c6-4773-8cf0-7c8532987208-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt\" (UID: \"049bac41-91c6-4773-8cf0-7c8532987208\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:30:57.042933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.042807    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/049bac41-91c6-4773-8cf0-7c8532987208-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt\" (UID: \"049bac41-91c6-4773-8cf0-7c8532987208\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:30:57.043748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.043720    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/049bac41-91c6-4773-8cf0-7c8532987208-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt\" (UID: \"049bac41-91c6-4773-8cf0-7c8532987208\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:30:57.046612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.044444    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049bac41-91c6-4773-8cf0-7c8532987208-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt\" (UID: \"049bac41-91c6-4773-8cf0-7c8532987208\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:30:57.052352 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.051690    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/049bac41-91c6-4773-8cf0-7c8532987208-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt\" (UID: \"049bac41-91c6-4773-8cf0-7c8532987208\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:30:57.065905 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.065877    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m86vd\" (UniqueName: \"kubernetes.io/projected/049bac41-91c6-4773-8cf0-7c8532987208-kube-api-access-m86vd\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt\" (UID: \"049bac41-91c6-4773-8cf0-7c8532987208\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:30:57.157059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.156984    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:30:57.218275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.216336    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-kdm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 30 01:30:57.218275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.216405    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l" podUID="81dd80d6-4327-4622-937a-39c52afd0fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 30 01:30:57.469765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.469737    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"]
Oct 30 01:30:57.471069 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:57.471044    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049bac41_91c6_4773_8cf0_7c8532987208.slice/crio-7f90bb61cdbfd894ddffd2c839d2f78b33b5eb87215c45c0c70f015cf0e324f9 WatchSource:0}: Error finding container 7f90bb61cdbfd894ddffd2c839d2f78b33b5eb87215c45c0c70f015cf0e324f9: Status 404 returned error can't find the container with id 7f90bb61cdbfd894ddffd2c839d2f78b33b5eb87215c45c0c70f015cf0e324f9
Oct 30 01:30:57.832079 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.832046    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"]
Oct 30 01:30:57.837015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.836993    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:30:57.854886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.854861    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"agc-cred-5c911d45fa08\""
Oct 30 01:30:57.855006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.854924    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"payload-processing-external-model-e2e--33153ae7-dockercfg-6jxz7\""
Oct 30 01:30:57.855006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.854861    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"openshift-service-ca.crt\""
Oct 30 01:30:57.855097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.855016    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"routing-overlay\""
Oct 30 01:30:57.855276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.855240    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"payload-processing-external-model-plugins-e2e-extmod-b-94ea1e\""
Oct 30 01:30:57.855276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.855268    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-94ea1e\"/\"kube-root-ca.crt\""
Oct 30 01:30:57.857737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.857709    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"]
Oct 30 01:30:57.954881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.954836    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5be75a55-0fde-421f-a978-47264d6545b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv\" (UID: \"5be75a55-0fde-421f-a978-47264d6545b6\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:30:57.955421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.955393    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be75a55-0fde-421f-a978-47264d6545b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv\" (UID: \"5be75a55-0fde-421f-a978-47264d6545b6\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:30:57.955600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.955571    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5be75a55-0fde-421f-a978-47264d6545b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv\" (UID: \"5be75a55-0fde-421f-a978-47264d6545b6\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:30:57.955832 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:57.955804    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtztm\" (UniqueName: \"kubernetes.io/projected/5be75a55-0fde-421f-a978-47264d6545b6-kube-api-access-jtztm\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv\" (UID: \"5be75a55-0fde-421f-a978-47264d6545b6\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:30:58.057210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:58.057151    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be75a55-0fde-421f-a978-47264d6545b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv\" (UID: \"5be75a55-0fde-421f-a978-47264d6545b6\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:30:58.057416 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:58.057274    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5be75a55-0fde-421f-a978-47264d6545b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv\" (UID: \"5be75a55-0fde-421f-a978-47264d6545b6\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:30:58.057416 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:58.057401    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtztm\" (UniqueName: \"kubernetes.io/projected/5be75a55-0fde-421f-a978-47264d6545b6-kube-api-access-jtztm\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv\" (UID: \"5be75a55-0fde-421f-a978-47264d6545b6\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:30:58.057604 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:58.057466    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5be75a55-0fde-421f-a978-47264d6545b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv\" (UID: \"5be75a55-0fde-421f-a978-47264d6545b6\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:30:58.058344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:58.058318    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be75a55-0fde-421f-a978-47264d6545b6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv\" (UID: \"5be75a55-0fde-421f-a978-47264d6545b6\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:30:58.058444 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:58.058391    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5be75a55-0fde-421f-a978-47264d6545b6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv\" (UID: \"5be75a55-0fde-421f-a978-47264d6545b6\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:30:58.060174 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:58.060129    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5be75a55-0fde-421f-a978-47264d6545b6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv\" (UID: \"5be75a55-0fde-421f-a978-47264d6545b6\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:30:58.078905 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:58.078883    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtztm\" (UniqueName: \"kubernetes.io/projected/5be75a55-0fde-421f-a978-47264d6545b6-kube-api-access-jtztm\") pod \"payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv\" (UID: \"5be75a55-0fde-421f-a978-47264d6545b6\") " pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:30:58.149763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:58.149694    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt" event={"ID":"049bac41-91c6-4773-8cf0-7c8532987208","Type":"ContainerStarted","Data":"da301bc6076b6869aee772adcda22e427541ff14cfc6ec60f6ed72a881aec05e"}
Oct 30 01:30:58.149763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:58.149745    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt" event={"ID":"049bac41-91c6-4773-8cf0-7c8532987208","Type":"ContainerStarted","Data":"7f90bb61cdbfd894ddffd2c839d2f78b33b5eb87215c45c0c70f015cf0e324f9"}
Oct 30 01:30:58.149952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:58.149935    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:30:58.151178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:58.151132    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:30:58.188686 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:58.188636    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt" podStartSLOduration=2.188621208 podStartE2EDuration="2.188621208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:58.185422876 +0000 UTC m=+2966.935078222" watchObservedRunningTime="2026-10-30 01:30:58.188621208 +0000 UTC m=+2966.938276577"
Oct 30 01:30:58.217100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:58.216754    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-kdm6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 30 01:30:58.217100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:58.216847    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l" podUID="81dd80d6-4327-4622-937a-39c52afd0fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 30 01:30:58.471144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:58.471116    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"]
Oct 30 01:30:58.472144 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:30:58.472113    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be75a55_0fde_421f_a978_47264d6545b6.slice/crio-a75971e4c71e94c0d87b59d41bea5851feb57510e2873d9e23cab4c6ca370e07 WatchSource:0}: Error finding container a75971e4c71e94c0d87b59d41bea5851feb57510e2873d9e23cab4c6ca370e07: Status 404 returned error can't find the container with id a75971e4c71e94c0d87b59d41bea5851feb57510e2873d9e23cab4c6ca370e07
Oct 30 01:30:59.161413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:59.161362    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv" event={"ID":"5be75a55-0fde-421f-a978-47264d6545b6","Type":"ContainerStarted","Data":"95eae26e454727222c9d33332760c3671b918f4134ee05f360439ba6edd4aa42"}
Oct 30 01:30:59.161621 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:59.161432    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv" event={"ID":"5be75a55-0fde-421f-a978-47264d6545b6","Type":"ContainerStarted","Data":"a75971e4c71e94c0d87b59d41bea5851feb57510e2873d9e23cab4c6ca370e07"}
Oct 30 01:30:59.161694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:59.161664    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:30:59.201321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:59.201272    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv" podStartSLOduration=2.2012585590000002 podStartE2EDuration="2.201258559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:59.198911993 +0000 UTC m=+2967.948567307" watchObservedRunningTime="2026-10-30 01:30:59.201258559 +0000 UTC m=+2967.950913867"
Oct 30 01:30:59.386706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:59.386675    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:59.387191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:59.387169    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:30:59.387879 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:30:59.387859    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:31:03.108770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:03.108731    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:31:04.125688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:04.125656    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:31:07.557343 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:31:07.557308    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:31:09.165767 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:09.165737    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:31:10.171461 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:10.171430    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:31:15.424817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:15.424780    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-kdm6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:31:15.425439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:15.425405    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l" podUID="81dd80d6-4327-4622-937a-39c52afd0fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:31:16.383254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:16.383210    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"]
Oct 30 01:31:16.856027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:16.855993    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"]
Oct 30 01:31:17.592550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:17.592516    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"]
Oct 30 01:31:17.599031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:17.598999    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"
Oct 30 01:31:17.616712 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:17.616686    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"]
Oct 30 01:31:17.620003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:17.619975    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:31:17.620144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:17.620066    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:31:17.675255 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:17.675218    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01b43a26-79d4-4136-8735-bd75b5897c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed55ee2-j6ddp\" (UID: \"01b43a26-79d4-4136-8735-bd75b5897c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"
Oct 30 01:31:17.675423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:17.675329    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01b43a26-79d4-4136-8735-bd75b5897c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed55ee2-j6ddp\" (UID: \"01b43a26-79d4-4136-8735-bd75b5897c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"
Oct 30 01:31:17.675649 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:17.675628    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8wv8\" (UniqueName: \"kubernetes.io/projected/01b43a26-79d4-4136-8735-bd75b5897c07-kube-api-access-z8wv8\") pod \"maas-api-revoke-sub-9ed55ee2-j6ddp\" (UID: \"01b43a26-79d4-4136-8735-bd75b5897c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"
Oct 30 01:31:17.776942 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:17.776909    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01b43a26-79d4-4136-8735-bd75b5897c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed55ee2-j6ddp\" (UID: \"01b43a26-79d4-4136-8735-bd75b5897c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"
Oct 30 01:31:17.777141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:17.776966    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01b43a26-79d4-4136-8735-bd75b5897c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed55ee2-j6ddp\" (UID: \"01b43a26-79d4-4136-8735-bd75b5897c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"
Oct 30 01:31:17.777265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:17.777209    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8wv8\" (UniqueName: \"kubernetes.io/projected/01b43a26-79d4-4136-8735-bd75b5897c07-kube-api-access-z8wv8\") pod \"maas-api-revoke-sub-9ed55ee2-j6ddp\" (UID: \"01b43a26-79d4-4136-8735-bd75b5897c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"
Oct 30 01:31:17.777890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:17.777863    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01b43a26-79d4-4136-8735-bd75b5897c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed55ee2-j6ddp\" (UID: \"01b43a26-79d4-4136-8735-bd75b5897c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"
Oct 30 01:31:17.779864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:17.779844    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01b43a26-79d4-4136-8735-bd75b5897c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed55ee2-j6ddp\" (UID: \"01b43a26-79d4-4136-8735-bd75b5897c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"
Oct 30 01:31:17.818391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:17.818362    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8wv8\" (UniqueName: \"kubernetes.io/projected/01b43a26-79d4-4136-8735-bd75b5897c07-kube-api-access-z8wv8\") pod \"maas-api-revoke-sub-9ed55ee2-j6ddp\" (UID: \"01b43a26-79d4-4136-8735-bd75b5897c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"
Oct 30 01:31:17.918826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:17.918768    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"
Oct 30 01:31:18.227820 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:18.227791    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"]
Oct 30 01:31:18.231067 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:31:18.228657    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b43a26_79d4_4136_8735_bd75b5897c07.slice/crio-db1e57810d5772391a1702bc6b7102a4bf78acab8771d03caa59a665b07396af WatchSource:0}: Error finding container db1e57810d5772391a1702bc6b7102a4bf78acab8771d03caa59a665b07396af: Status 404 returned error can't find the container with id db1e57810d5772391a1702bc6b7102a4bf78acab8771d03caa59a665b07396af
Oct 30 01:31:18.380740 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:18.380709    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp" event={"ID":"01b43a26-79d4-4136-8735-bd75b5897c07","Type":"ContainerStarted","Data":"db1e57810d5772391a1702bc6b7102a4bf78acab8771d03caa59a665b07396af"}
Oct 30 01:31:19.389833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:19.389801    4425 generic.go:356] "Generic (PLEG): container finished" podID="01b43a26-79d4-4136-8735-bd75b5897c07" containerID="c8f57c3fc79f9837bccfdb9f2f7cb42e0438cd0a31a17f05c5fccd3c989e8e29" exitCode=0
Oct 30 01:31:19.390281 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:19.389896    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp" event={"ID":"01b43a26-79d4-4136-8735-bd75b5897c07","Type":"ContainerDied","Data":"c8f57c3fc79f9837bccfdb9f2f7cb42e0438cd0a31a17f05c5fccd3c989e8e29"}
Oct 30 01:31:20.556618 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:20.556591    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"
Oct 30 01:31:20.611348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:20.611320    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01b43a26-79d4-4136-8735-bd75b5897c07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01b43a26-79d4-4136-8735-bd75b5897c07-maas-api-xks-ca\") pod \"01b43a26-79d4-4136-8735-bd75b5897c07\" (UID: \"01b43a26-79d4-4136-8735-bd75b5897c07\") "
Oct 30 01:31:20.611468 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:20.611417    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b43a26-79d4-4136-8735-bd75b5897c07-kube-api-access-z8wv8\" (UniqueName: \"kubernetes.io/projected/01b43a26-79d4-4136-8735-bd75b5897c07-kube-api-access-z8wv8\") pod \"01b43a26-79d4-4136-8735-bd75b5897c07\" (UID: \"01b43a26-79d4-4136-8735-bd75b5897c07\") "
Oct 30 01:31:20.611567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:20.611552    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01b43a26-79d4-4136-8735-bd75b5897c07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01b43a26-79d4-4136-8735-bd75b5897c07-maas-api-openshift-ca\") pod \"01b43a26-79d4-4136-8735-bd75b5897c07\" (UID: \"01b43a26-79d4-4136-8735-bd75b5897c07\") "
Oct 30 01:31:20.611942 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:20.611913    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b43a26-79d4-4136-8735-bd75b5897c07-maas-api-openshift-ca" pod "01b43a26-79d4-4136-8735-bd75b5897c07" (UID: "01b43a26-79d4-4136-8735-bd75b5897c07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:20.612376 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:20.612351    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01b43a26-79d4-4136-8735-bd75b5897c07-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:20.613939 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:20.613904    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b43a26-79d4-4136-8735-bd75b5897c07-maas-api-xks-ca" pod "01b43a26-79d4-4136-8735-bd75b5897c07" (UID: "01b43a26-79d4-4136-8735-bd75b5897c07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:31:20.614075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:20.613970    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b43a26-79d4-4136-8735-bd75b5897c07-kube-api-access-z8wv8" pod "01b43a26-79d4-4136-8735-bd75b5897c07" (UID: "01b43a26-79d4-4136-8735-bd75b5897c07"). InnerVolumeSpecName "kube-api-access-z8wv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:20.714200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:20.714098    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01b43a26-79d4-4136-8735-bd75b5897c07-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:20.714200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:20.714132    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8wv8\" (UniqueName: \"kubernetes.io/projected/01b43a26-79d4-4136-8735-bd75b5897c07-kube-api-access-z8wv8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:21.384558 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.384518    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv" podUID="5be75a55-0fde-421f-a978-47264d6545b6" containerName="payload-processing" containerID="cri-o://95eae26e454727222c9d33332760c3671b918f4134ee05f360439ba6edd4aa42" gracePeriod=25
Oct 30 01:31:21.410810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.410782    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"
Oct 30 01:31:21.410949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.410824    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp" event={"ID":"01b43a26-79d4-4136-8735-bd75b5897c07","Type":"ContainerDied","Data":"db1e57810d5772391a1702bc6b7102a4bf78acab8771d03caa59a665b07396af"}
Oct 30 01:31:21.410949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.410871    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db1e57810d5772391a1702bc6b7102a4bf78acab8771d03caa59a665b07396af"
Oct 30 01:31:21.657644 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.657580    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:31:21.726835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.726803    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be75a55-0fde-421f-a978-47264d6545b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be75a55-0fde-421f-a978-47264d6545b6-plugins-config-volume\") pod \"5be75a55-0fde-421f-a978-47264d6545b6\" (UID: \"5be75a55-0fde-421f-a978-47264d6545b6\") "
Oct 30 01:31:21.727040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.726912    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be75a55-0fde-421f-a978-47264d6545b6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5be75a55-0fde-421f-a978-47264d6545b6-routing-overlay-volume\") pod \"5be75a55-0fde-421f-a978-47264d6545b6\" (UID: \"5be75a55-0fde-421f-a978-47264d6545b6\") "
Oct 30 01:31:21.727040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.726947    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be75a55-0fde-421f-a978-47264d6545b6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5be75a55-0fde-421f-a978-47264d6545b6-provider-credentials\") pod \"5be75a55-0fde-421f-a978-47264d6545b6\" (UID: \"5be75a55-0fde-421f-a978-47264d6545b6\") "
Oct 30 01:31:21.727040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.726979    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be75a55-0fde-421f-a978-47264d6545b6-kube-api-access-jtztm\" (UniqueName: \"kubernetes.io/projected/5be75a55-0fde-421f-a978-47264d6545b6-kube-api-access-jtztm\") pod \"5be75a55-0fde-421f-a978-47264d6545b6\" (UID: \"5be75a55-0fde-421f-a978-47264d6545b6\") "
Oct 30 01:31:21.727310 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.727288    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be75a55-0fde-421f-a978-47264d6545b6-plugins-config-volume" pod "5be75a55-0fde-421f-a978-47264d6545b6" (UID: "5be75a55-0fde-421f-a978-47264d6545b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:21.727393 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.727331    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be75a55-0fde-421f-a978-47264d6545b6-routing-overlay-volume" pod "5be75a55-0fde-421f-a978-47264d6545b6" (UID: "5be75a55-0fde-421f-a978-47264d6545b6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:21.727762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.727738    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5be75a55-0fde-421f-a978-47264d6545b6-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:21.727867 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.727765    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5be75a55-0fde-421f-a978-47264d6545b6-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:21.729763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.729739    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be75a55-0fde-421f-a978-47264d6545b6-provider-credentials" pod "5be75a55-0fde-421f-a978-47264d6545b6" (UID: "5be75a55-0fde-421f-a978-47264d6545b6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:21.729868 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.729784    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be75a55-0fde-421f-a978-47264d6545b6-kube-api-access-jtztm" pod "5be75a55-0fde-421f-a978-47264d6545b6" (UID: "5be75a55-0fde-421f-a978-47264d6545b6"). InnerVolumeSpecName "kube-api-access-jtztm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:21.828856 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.828825    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5be75a55-0fde-421f-a978-47264d6545b6-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:21.828856 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.828851    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtztm\" (UniqueName: \"kubernetes.io/projected/5be75a55-0fde-421f-a978-47264d6545b6-kube-api-access-jtztm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:21.856851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:21.856814    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt" podUID="049bac41-91c6-4773-8cf0-7c8532987208" containerName="payload-processing" containerID="cri-o://da301bc6076b6869aee772adcda22e427541ff14cfc6ec60f6ed72a881aec05e" gracePeriod=25
Oct 30 01:31:22.097976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.097951    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:31:22.232965 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.232877    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049bac41-91c6-4773-8cf0-7c8532987208-provider-credentials\" (UniqueName: \"kubernetes.io/projected/049bac41-91c6-4773-8cf0-7c8532987208-provider-credentials\") pod \"049bac41-91c6-4773-8cf0-7c8532987208\" (UID: \"049bac41-91c6-4773-8cf0-7c8532987208\") "
Oct 30 01:31:22.233141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.233001    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/049bac41-91c6-4773-8cf0-7c8532987208-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049bac41-91c6-4773-8cf0-7c8532987208-plugins-config-volume\") pod \"049bac41-91c6-4773-8cf0-7c8532987208\" (UID: \"049bac41-91c6-4773-8cf0-7c8532987208\") "
Oct 30 01:31:22.233141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.233067    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049bac41-91c6-4773-8cf0-7c8532987208-kube-api-access-m86vd\" (UniqueName: \"kubernetes.io/projected/049bac41-91c6-4773-8cf0-7c8532987208-kube-api-access-m86vd\") pod \"049bac41-91c6-4773-8cf0-7c8532987208\" (UID: \"049bac41-91c6-4773-8cf0-7c8532987208\") "
Oct 30 01:31:22.233329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.233226    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/049bac41-91c6-4773-8cf0-7c8532987208-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/049bac41-91c6-4773-8cf0-7c8532987208-routing-overlay-volume\") pod \"049bac41-91c6-4773-8cf0-7c8532987208\" (UID: \"049bac41-91c6-4773-8cf0-7c8532987208\") "
Oct 30 01:31:22.233527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.233496    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049bac41-91c6-4773-8cf0-7c8532987208-plugins-config-volume" pod "049bac41-91c6-4773-8cf0-7c8532987208" (UID: "049bac41-91c6-4773-8cf0-7c8532987208"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:22.233667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.233583    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049bac41-91c6-4773-8cf0-7c8532987208-routing-overlay-volume" pod "049bac41-91c6-4773-8cf0-7c8532987208" (UID: "049bac41-91c6-4773-8cf0-7c8532987208"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:22.233902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.233880    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049bac41-91c6-4773-8cf0-7c8532987208-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:22.233976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.233907    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/049bac41-91c6-4773-8cf0-7c8532987208-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:22.234994 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.234974    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049bac41-91c6-4773-8cf0-7c8532987208-provider-credentials" pod "049bac41-91c6-4773-8cf0-7c8532987208" (UID: "049bac41-91c6-4773-8cf0-7c8532987208"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:22.235304 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.235289    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049bac41-91c6-4773-8cf0-7c8532987208-kube-api-access-m86vd" pod "049bac41-91c6-4773-8cf0-7c8532987208" (UID: "049bac41-91c6-4773-8cf0-7c8532987208"). InnerVolumeSpecName "kube-api-access-m86vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:22.334933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.334901    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/049bac41-91c6-4773-8cf0-7c8532987208-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:22.334933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.334932    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m86vd\" (UniqueName: \"kubernetes.io/projected/049bac41-91c6-4773-8cf0-7c8532987208-kube-api-access-m86vd\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:22.420308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.420276    4425 generic.go:356] "Generic (PLEG): container finished" podID="049bac41-91c6-4773-8cf0-7c8532987208" containerID="da301bc6076b6869aee772adcda22e427541ff14cfc6ec60f6ed72a881aec05e" exitCode=0
Oct 30 01:31:22.420308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.420312    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"
Oct 30 01:31:22.420574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.420331    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt" event={"ID":"049bac41-91c6-4773-8cf0-7c8532987208","Type":"ContainerDied","Data":"da301bc6076b6869aee772adcda22e427541ff14cfc6ec60f6ed72a881aec05e"}
Oct 30 01:31:22.420574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.420382    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt" event={"ID":"049bac41-91c6-4773-8cf0-7c8532987208","Type":"ContainerDied","Data":"7f90bb61cdbfd894ddffd2c839d2f78b33b5eb87215c45c0c70f015cf0e324f9"}
Oct 30 01:31:22.420574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.420412    4425 scope.go:122] "RemoveContainer" containerID="da301bc6076b6869aee772adcda22e427541ff14cfc6ec60f6ed72a881aec05e"
Oct 30 01:31:22.422226 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.422190    4425 generic.go:356] "Generic (PLEG): container finished" podID="5be75a55-0fde-421f-a978-47264d6545b6" containerID="95eae26e454727222c9d33332760c3671b918f4134ee05f360439ba6edd4aa42" exitCode=0
Oct 30 01:31:22.422335 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.422227    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv" event={"ID":"5be75a55-0fde-421f-a978-47264d6545b6","Type":"ContainerDied","Data":"95eae26e454727222c9d33332760c3671b918f4134ee05f360439ba6edd4aa42"}
Oct 30 01:31:22.422335 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.422267    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv" event={"ID":"5be75a55-0fde-421f-a978-47264d6545b6","Type":"ContainerDied","Data":"a75971e4c71e94c0d87b59d41bea5851feb57510e2873d9e23cab4c6ca370e07"}
Oct 30 01:31:22.422335 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.422326    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"
Oct 30 01:31:22.432946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.432842    4425 scope.go:122] "RemoveContainer" containerID="da301bc6076b6869aee772adcda22e427541ff14cfc6ec60f6ed72a881aec05e"
Oct 30 01:31:22.433249 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:31:22.433219    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da301bc6076b6869aee772adcda22e427541ff14cfc6ec60f6ed72a881aec05e\": container with ID starting with da301bc6076b6869aee772adcda22e427541ff14cfc6ec60f6ed72a881aec05e not found: ID does not exist" containerID="da301bc6076b6869aee772adcda22e427541ff14cfc6ec60f6ed72a881aec05e"
Oct 30 01:31:22.433373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.433259    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da301bc6076b6869aee772adcda22e427541ff14cfc6ec60f6ed72a881aec05e"} err="failed to get container status \"da301bc6076b6869aee772adcda22e427541ff14cfc6ec60f6ed72a881aec05e\": rpc error: code = NotFound desc = could not find container \"da301bc6076b6869aee772adcda22e427541ff14cfc6ec60f6ed72a881aec05e\": container with ID starting with da301bc6076b6869aee772adcda22e427541ff14cfc6ec60f6ed72a881aec05e not found: ID does not exist"
Oct 30 01:31:22.433373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.433283    4425 scope.go:122] "RemoveContainer" containerID="95eae26e454727222c9d33332760c3671b918f4134ee05f360439ba6edd4aa42"
Oct 30 01:31:22.443194 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.443150    4425 scope.go:122] "RemoveContainer" containerID="95eae26e454727222c9d33332760c3671b918f4134ee05f360439ba6edd4aa42"
Oct 30 01:31:22.443439 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:31:22.443421    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95eae26e454727222c9d33332760c3671b918f4134ee05f360439ba6edd4aa42\": container with ID starting with 95eae26e454727222c9d33332760c3671b918f4134ee05f360439ba6edd4aa42 not found: ID does not exist" containerID="95eae26e454727222c9d33332760c3671b918f4134ee05f360439ba6edd4aa42"
Oct 30 01:31:22.443502 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.443445    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95eae26e454727222c9d33332760c3671b918f4134ee05f360439ba6edd4aa42"} err="failed to get container status \"95eae26e454727222c9d33332760c3671b918f4134ee05f360439ba6edd4aa42\": rpc error: code = NotFound desc = could not find container \"95eae26e454727222c9d33332760c3671b918f4134ee05f360439ba6edd4aa42\": container with ID starting with 95eae26e454727222c9d33332760c3671b918f4134ee05f360439ba6edd4aa42 not found: ID does not exist"
Oct 30 01:31:22.479565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.479543    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"]
Oct 30 01:31:22.489006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.488960    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76w5hqt"]
Oct 30 01:31:22.513499 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.513476    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"]
Oct 30 01:31:22.517331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:22.517310    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-94ea1e/payload-processing-external-model-e2e-extmod-b-94ea1e-6459drlxv"]
Oct 30 01:31:23.427550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:23.427521    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="049bac41-91c6-4773-8cf0-7c8532987208" path="/var/lib/kubelet/pods/049bac41-91c6-4773-8cf0-7c8532987208/volumes"
Oct 30 01:31:23.428189 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:23.428138    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5be75a55-0fde-421f-a978-47264d6545b6" path="/var/lib/kubelet/pods/5be75a55-0fde-421f-a978-47264d6545b6/volumes"
Oct 30 01:31:28.391547 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.391515    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"]
Oct 30 01:31:28.392303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.392289    4425 cpu_manager.go:414] "containerMap: removing container" podUID="5be75a55-0fde-421f-a978-47264d6545b6" containerName="payload-processing"
Oct 30 01:31:28.392370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.392304    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be75a55-0fde-421f-a978-47264d6545b6" containerName="payload-processing"
Oct 30 01:31:28.392370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.392340    4425 cpu_manager.go:414] "containerMap: removing container" podUID="049bac41-91c6-4773-8cf0-7c8532987208" containerName="payload-processing"
Oct 30 01:31:28.392370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.392346    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049bac41-91c6-4773-8cf0-7c8532987208" containerName="payload-processing"
Oct 30 01:31:28.392370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.392360    4425 cpu_manager.go:414] "containerMap: removing container" podUID="01b43a26-79d4-4136-8735-bd75b5897c07" containerName="revoke-keys"
Oct 30 01:31:28.392370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.392365    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b43a26-79d4-4136-8735-bd75b5897c07" containerName="revoke-keys"
Oct 30 01:31:28.392546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.392494    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b43a26-79d4-4136-8735-bd75b5897c07" containerName="revoke-keys"
Oct 30 01:31:28.392546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.392507    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be75a55-0fde-421f-a978-47264d6545b6" containerName="payload-processing"
Oct 30 01:31:28.392546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.392520    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="049bac41-91c6-4773-8cf0-7c8532987208" containerName="payload-processing"
Oct 30 01:31:28.397803 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.397784    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"
Oct 30 01:31:28.410783 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.410756    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:31:28.420530 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.420507    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:31:28.426144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.426118    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"]
Oct 30 01:31:28.506331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.506300    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-959aec9b-flzql\" (UID: \"98dbf13a-0e47-45ab-869f-8d6122e5a7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"
Oct 30 01:31:28.506331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.506331    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxckq\" (UniqueName: \"kubernetes.io/projected/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-kube-api-access-vxckq\") pod \"maas-api-revoke-sub-959aec9b-flzql\" (UID: \"98dbf13a-0e47-45ab-869f-8d6122e5a7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"
Oct 30 01:31:28.506553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.506419    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-959aec9b-flzql\" (UID: \"98dbf13a-0e47-45ab-869f-8d6122e5a7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"
Oct 30 01:31:28.608393 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.608359    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-959aec9b-flzql\" (UID: \"98dbf13a-0e47-45ab-869f-8d6122e5a7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"
Oct 30 01:31:28.608616 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.608596    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-959aec9b-flzql\" (UID: \"98dbf13a-0e47-45ab-869f-8d6122e5a7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"
Oct 30 01:31:28.608666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.608630    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxckq\" (UniqueName: \"kubernetes.io/projected/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-kube-api-access-vxckq\") pod \"maas-api-revoke-sub-959aec9b-flzql\" (UID: \"98dbf13a-0e47-45ab-869f-8d6122e5a7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"
Oct 30 01:31:28.609274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.609249    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-959aec9b-flzql\" (UID: \"98dbf13a-0e47-45ab-869f-8d6122e5a7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"
Oct 30 01:31:28.610753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.610732    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-959aec9b-flzql\" (UID: \"98dbf13a-0e47-45ab-869f-8d6122e5a7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"
Oct 30 01:31:28.629721 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.629696    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxckq\" (UniqueName: \"kubernetes.io/projected/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-kube-api-access-vxckq\") pod \"maas-api-revoke-sub-959aec9b-flzql\" (UID: \"98dbf13a-0e47-45ab-869f-8d6122e5a7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"
Oct 30 01:31:28.712140 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:28.712088    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"
Oct 30 01:31:29.021226 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:29.020584    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"]
Oct 30 01:31:29.480306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:29.480269    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql" event={"ID":"98dbf13a-0e47-45ab-869f-8d6122e5a7e7","Type":"ContainerStarted","Data":"be5b37d2a6cf55bc4cedcf7159153bebb1b3ceec1290dc6dcd2895fb678749aa"}
Oct 30 01:31:29.480657 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:29.480323    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql" event={"ID":"98dbf13a-0e47-45ab-869f-8d6122e5a7e7","Type":"ContainerStarted","Data":"87546a074e986b8abf51f0b08a4ab124ea18ad3c3d426c62bfcf5fbc3bec7dc5"}
Oct 30 01:31:29.519870 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:29.519818    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql" podStartSLOduration=1.5198048979999998 podStartE2EDuration="1.519804898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:29.515460905 +0000 UTC m=+2998.265116215" watchObservedRunningTime="2026-10-30 01:31:29.519804898 +0000 UTC m=+2998.269460206"
Oct 30 01:31:30.489727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:30.489644    4425 generic.go:356] "Generic (PLEG): container finished" podID="98dbf13a-0e47-45ab-869f-8d6122e5a7e7" containerID="be5b37d2a6cf55bc4cedcf7159153bebb1b3ceec1290dc6dcd2895fb678749aa" exitCode=0
Oct 30 01:31:30.490107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:30.489734    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql" event={"ID":"98dbf13a-0e47-45ab-869f-8d6122e5a7e7","Type":"ContainerDied","Data":"be5b37d2a6cf55bc4cedcf7159153bebb1b3ceec1290dc6dcd2895fb678749aa"}
Oct 30 01:31:31.667725 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:31.667700    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"
Oct 30 01:31:31.849746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:31.849711    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-maas-api-xks-ca\") pod \"98dbf13a-0e47-45ab-869f-8d6122e5a7e7\" (UID: \"98dbf13a-0e47-45ab-869f-8d6122e5a7e7\") "
Oct 30 01:31:31.849918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:31.849831    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-kube-api-access-vxckq\" (UniqueName: \"kubernetes.io/projected/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-kube-api-access-vxckq\") pod \"98dbf13a-0e47-45ab-869f-8d6122e5a7e7\" (UID: \"98dbf13a-0e47-45ab-869f-8d6122e5a7e7\") "
Oct 30 01:31:31.849963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:31.849950    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-maas-api-openshift-ca\") pod \"98dbf13a-0e47-45ab-869f-8d6122e5a7e7\" (UID: \"98dbf13a-0e47-45ab-869f-8d6122e5a7e7\") "
Oct 30 01:31:31.850327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:31.850298    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-maas-api-openshift-ca" pod "98dbf13a-0e47-45ab-869f-8d6122e5a7e7" (UID: "98dbf13a-0e47-45ab-869f-8d6122e5a7e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:31.850549 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:31.850534    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:31.851802 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:31.851777    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-maas-api-xks-ca" pod "98dbf13a-0e47-45ab-869f-8d6122e5a7e7" (UID: "98dbf13a-0e47-45ab-869f-8d6122e5a7e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:31:31.851945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:31.851925    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-kube-api-access-vxckq" pod "98dbf13a-0e47-45ab-869f-8d6122e5a7e7" (UID: "98dbf13a-0e47-45ab-869f-8d6122e5a7e7"). InnerVolumeSpecName "kube-api-access-vxckq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:31.952290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:31.952260    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxckq\" (UniqueName: \"kubernetes.io/projected/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-kube-api-access-vxckq\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:31.952290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:31.952291    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98dbf13a-0e47-45ab-869f-8d6122e5a7e7-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:32.506903 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:32.506859    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql" event={"ID":"98dbf13a-0e47-45ab-869f-8d6122e5a7e7","Type":"ContainerDied","Data":"87546a074e986b8abf51f0b08a4ab124ea18ad3c3d426c62bfcf5fbc3bec7dc5"}
Oct 30 01:31:32.506903 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:32.506883    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"
Oct 30 01:31:32.506903 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:32.506906    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87546a074e986b8abf51f0b08a4ab124ea18ad3c3d426c62bfcf5fbc3bec7dc5"
Oct 30 01:31:35.241289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:35.241257    4425 scope.go:122] "RemoveContainer" containerID="5e4ed70b47ba1ecdfd19bb942f24eb9aef5364871385b73d617b01d974b69a25"
Oct 30 01:31:37.556745 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:31:37.556708    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:31:40.151465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.151424    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"]
Oct 30 01:31:40.152250 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.152232    4425 cpu_manager.go:414] "containerMap: removing container" podUID="98dbf13a-0e47-45ab-869f-8d6122e5a7e7" containerName="revoke-keys"
Oct 30 01:31:40.152311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.152252    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98dbf13a-0e47-45ab-869f-8d6122e5a7e7" containerName="revoke-keys"
Oct 30 01:31:40.152379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.152370    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="98dbf13a-0e47-45ab-869f-8d6122e5a7e7" containerName="revoke-keys"
Oct 30 01:31:40.157400 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.157381    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"
Oct 30 01:31:40.176671 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.176647    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"]
Oct 30 01:31:40.265887 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.265850    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2r99\" (UniqueName: \"kubernetes.io/projected/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-kube-api-access-r2r99\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74\" (UID: \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"
Oct 30 01:31:40.266061 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.266011    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74\" (UID: \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"
Oct 30 01:31:40.266108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.266061    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74\" (UID: \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"
Oct 30 01:31:40.266171 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.266122    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74\" (UID: \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"
Oct 30 01:31:40.367064 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.367034    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74\" (UID: \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"
Oct 30 01:31:40.367277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.367080    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74\" (UID: \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"
Oct 30 01:31:40.367277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.367122    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74\" (UID: \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"
Oct 30 01:31:40.367435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.367317    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2r99\" (UniqueName: \"kubernetes.io/projected/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-kube-api-access-r2r99\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74\" (UID: \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"
Oct 30 01:31:40.367986 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.367913    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74\" (UID: \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"
Oct 30 01:31:40.367986 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.367921    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74\" (UID: \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"
Oct 30 01:31:40.369933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.369909    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74\" (UID: \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"
Oct 30 01:31:40.393502 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.393478    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2r99\" (UniqueName: \"kubernetes.io/projected/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-kube-api-access-r2r99\") pod \"payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74\" (UID: \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\") " pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"
Oct 30 01:31:40.472297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.472222    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"
Oct 30 01:31:40.782374 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.782326    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"]
Oct 30 01:31:40.825810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.825775    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"]
Oct 30 01:31:40.833819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:40.833790    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"]
Oct 30 01:31:41.596240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:41.596207    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74" event={"ID":"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25","Type":"ContainerStarted","Data":"396fbbe2ccd25ae0b833a26dd500aea1054c9bfc3ff46f901f1447bc4d0e7640"}
Oct 30 01:31:41.596645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:41.596259    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74" event={"ID":"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25","Type":"ContainerStarted","Data":"13209e59b3867ddcd8e784e1627c6f57a7d6c40f804b9651854b70849ac902e7"}
Oct 30 01:31:41.596645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:41.596280    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"
Oct 30 01:31:41.635564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:41.635522    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74" podStartSLOduration=1.635509949 podStartE2EDuration="1.635509949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:41.634067198 +0000 UTC m=+3010.383722506" watchObservedRunningTime="2026-10-30 01:31:41.635509949 +0000 UTC m=+3010.385165301"
Oct 30 01:31:45.041328 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:45.041247    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"]
Oct 30 01:31:45.052013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:45.051973    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-683f64b3-5lxcg"]
Oct 30 01:31:45.425556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:45.425478    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3c6775-eb0a-4cdf-8bb4-9982eab056b8" path="/var/lib/kubelet/pods/6b3c6775-eb0a-4cdf-8bb4-9982eab056b8/volumes"
Oct 30 01:31:45.826630 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:45.826590    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql" podUID="37f9e186-d092-44a7-8d83-2c1f9e96388a" containerName="payload-processing" containerID="cri-o://ccc2e5a4e3158806b9ed6916f12d325c56f538b67703874b3c2b5d3db439c995" gracePeriod=25
Oct 30 01:31:46.125591 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.125561    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:31:46.251928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.251892    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f9e186-d092-44a7-8d83-2c1f9e96388a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f9e186-d092-44a7-8d83-2c1f9e96388a-plugins-config-volume\") pod \"37f9e186-d092-44a7-8d83-2c1f9e96388a\" (UID: \"37f9e186-d092-44a7-8d83-2c1f9e96388a\") "
Oct 30 01:31:46.252096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.251973    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f9e186-d092-44a7-8d83-2c1f9e96388a-kube-api-access-5k9hf\" (UniqueName: \"kubernetes.io/projected/37f9e186-d092-44a7-8d83-2c1f9e96388a-kube-api-access-5k9hf\") pod \"37f9e186-d092-44a7-8d83-2c1f9e96388a\" (UID: \"37f9e186-d092-44a7-8d83-2c1f9e96388a\") "
Oct 30 01:31:46.252096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.252032    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f9e186-d092-44a7-8d83-2c1f9e96388a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37f9e186-d092-44a7-8d83-2c1f9e96388a-routing-overlay-volume\") pod \"37f9e186-d092-44a7-8d83-2c1f9e96388a\" (UID: \"37f9e186-d092-44a7-8d83-2c1f9e96388a\") "
Oct 30 01:31:46.252263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.252226    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f9e186-d092-44a7-8d83-2c1f9e96388a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/37f9e186-d092-44a7-8d83-2c1f9e96388a-provider-credentials\") pod \"37f9e186-d092-44a7-8d83-2c1f9e96388a\" (UID: \"37f9e186-d092-44a7-8d83-2c1f9e96388a\") "
Oct 30 01:31:46.252418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.252392    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f9e186-d092-44a7-8d83-2c1f9e96388a-plugins-config-volume" pod "37f9e186-d092-44a7-8d83-2c1f9e96388a" (UID: "37f9e186-d092-44a7-8d83-2c1f9e96388a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:46.252475 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.252452    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f9e186-d092-44a7-8d83-2c1f9e96388a-routing-overlay-volume" pod "37f9e186-d092-44a7-8d83-2c1f9e96388a" (UID: "37f9e186-d092-44a7-8d83-2c1f9e96388a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:46.252959 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.252936    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f9e186-d092-44a7-8d83-2c1f9e96388a-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:46.253059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.252961    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37f9e186-d092-44a7-8d83-2c1f9e96388a-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:46.254264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.254242    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f9e186-d092-44a7-8d83-2c1f9e96388a-provider-credentials" pod "37f9e186-d092-44a7-8d83-2c1f9e96388a" (UID: "37f9e186-d092-44a7-8d83-2c1f9e96388a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:46.254352 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.254332    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f9e186-d092-44a7-8d83-2c1f9e96388a-kube-api-access-5k9hf" pod "37f9e186-d092-44a7-8d83-2c1f9e96388a" (UID: "37f9e186-d092-44a7-8d83-2c1f9e96388a"). InnerVolumeSpecName "kube-api-access-5k9hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:46.354880 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.354804    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k9hf\" (UniqueName: \"kubernetes.io/projected/37f9e186-d092-44a7-8d83-2c1f9e96388a-kube-api-access-5k9hf\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:46.354880 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.354832    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37f9e186-d092-44a7-8d83-2c1f9e96388a-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:46.596777 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.596738    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74" podUID="9c3e4798-52b6-4a4b-bf06-6c5aece3eb25" containerName="payload-processing" containerID="cri-o://396fbbe2ccd25ae0b833a26dd500aea1054c9bfc3ff46f901f1447bc4d0e7640" gracePeriod=25
Oct 30 01:31:46.637769 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.637741    4425 generic.go:356] "Generic (PLEG): container finished" podID="37f9e186-d092-44a7-8d83-2c1f9e96388a" containerID="ccc2e5a4e3158806b9ed6916f12d325c56f538b67703874b3c2b5d3db439c995" exitCode=0
Oct 30 01:31:46.637899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.637780    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"
Oct 30 01:31:46.637899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.637819    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql" event={"ID":"37f9e186-d092-44a7-8d83-2c1f9e96388a","Type":"ContainerDied","Data":"ccc2e5a4e3158806b9ed6916f12d325c56f538b67703874b3c2b5d3db439c995"}
Oct 30 01:31:46.637899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.637871    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql" event={"ID":"37f9e186-d092-44a7-8d83-2c1f9e96388a","Type":"ContainerDied","Data":"a0af1369bc8bdc6333d6363ce1ab232ad9438603196014c9a43df9750c2c0354"}
Oct 30 01:31:46.638037 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.637899    4425 scope.go:122] "RemoveContainer" containerID="ccc2e5a4e3158806b9ed6916f12d325c56f538b67703874b3c2b5d3db439c995"
Oct 30 01:31:46.712756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.712733    4425 scope.go:122] "RemoveContainer" containerID="ccc2e5a4e3158806b9ed6916f12d325c56f538b67703874b3c2b5d3db439c995"
Oct 30 01:31:46.713083 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:31:46.713062    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc2e5a4e3158806b9ed6916f12d325c56f538b67703874b3c2b5d3db439c995\": container with ID starting with ccc2e5a4e3158806b9ed6916f12d325c56f538b67703874b3c2b5d3db439c995 not found: ID does not exist" containerID="ccc2e5a4e3158806b9ed6916f12d325c56f538b67703874b3c2b5d3db439c995"
Oct 30 01:31:46.713137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.713091    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc2e5a4e3158806b9ed6916f12d325c56f538b67703874b3c2b5d3db439c995"} err="failed to get container status \"ccc2e5a4e3158806b9ed6916f12d325c56f538b67703874b3c2b5d3db439c995\": rpc error: code = NotFound desc = could not find container \"ccc2e5a4e3158806b9ed6916f12d325c56f538b67703874b3c2b5d3db439c995\": container with ID starting with ccc2e5a4e3158806b9ed6916f12d325c56f538b67703874b3c2b5d3db439c995 not found: ID does not exist"
Oct 30 01:31:46.720134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.720111    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"]
Oct 30 01:31:46.729198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.729165    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-cfb7w76ql"]
Oct 30 01:31:46.808117 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.808087    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:31:46.883367 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.883334    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"
Oct 30 01:31:46.962419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.962371    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-plugins-config-volume\") pod \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\" (UID: \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\") "
Oct 30 01:31:46.962641 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.962619    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-provider-credentials\") pod \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\" (UID: \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\") "
Oct 30 01:31:46.962733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.962673    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-routing-overlay-volume\") pod \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\" (UID: \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\") "
Oct 30 01:31:46.962797 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.962746    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-plugins-config-volume" pod "9c3e4798-52b6-4a4b-bf06-6c5aece3eb25" (UID: "9c3e4798-52b6-4a4b-bf06-6c5aece3eb25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:46.962797 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.962759    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-kube-api-access-r2r99\" (UniqueName: \"kubernetes.io/projected/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-kube-api-access-r2r99\") pod \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\" (UID: \"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25\") "
Oct 30 01:31:46.963096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.963066    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-routing-overlay-volume" pod "9c3e4798-52b6-4a4b-bf06-6c5aece3eb25" (UID: "9c3e4798-52b6-4a4b-bf06-6c5aece3eb25"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:46.963739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.963506    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:46.963739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.963532    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:46.973184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.973148    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-provider-credentials" pod "9c3e4798-52b6-4a4b-bf06-6c5aece3eb25" (UID: "9c3e4798-52b6-4a4b-bf06-6c5aece3eb25"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:46.975764 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:46.975742    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-kube-api-access-r2r99" pod "9c3e4798-52b6-4a4b-bf06-6c5aece3eb25" (UID: "9c3e4798-52b6-4a4b-bf06-6c5aece3eb25"). InnerVolumeSpecName "kube-api-access-r2r99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:47.064640 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:47.064606    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2r99\" (UniqueName: \"kubernetes.io/projected/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-kube-api-access-r2r99\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:47.064640 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:47.064637    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:47.424795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:47.424764    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37f9e186-d092-44a7-8d83-2c1f9e96388a" path="/var/lib/kubelet/pods/37f9e186-d092-44a7-8d83-2c1f9e96388a/volumes"
Oct 30 01:31:47.647809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:47.647775    4425 generic.go:356] "Generic (PLEG): container finished" podID="9c3e4798-52b6-4a4b-bf06-6c5aece3eb25" containerID="396fbbe2ccd25ae0b833a26dd500aea1054c9bfc3ff46f901f1447bc4d0e7640" exitCode=0
Oct 30 01:31:47.647992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:47.647816    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"
Oct 30 01:31:47.647992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:47.647858    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74" event={"ID":"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25","Type":"ContainerDied","Data":"396fbbe2ccd25ae0b833a26dd500aea1054c9bfc3ff46f901f1447bc4d0e7640"}
Oct 30 01:31:47.647992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:47.647931    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74" event={"ID":"9c3e4798-52b6-4a4b-bf06-6c5aece3eb25","Type":"ContainerDied","Data":"13209e59b3867ddcd8e784e1627c6f57a7d6c40f804b9651854b70849ac902e7"}
Oct 30 01:31:47.647992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:47.647961    4425 scope.go:122] "RemoveContainer" containerID="396fbbe2ccd25ae0b833a26dd500aea1054c9bfc3ff46f901f1447bc4d0e7640"
Oct 30 01:31:47.661070 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:47.661053    4425 scope.go:122] "RemoveContainer" containerID="396fbbe2ccd25ae0b833a26dd500aea1054c9bfc3ff46f901f1447bc4d0e7640"
Oct 30 01:31:47.661405 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:31:47.661377    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396fbbe2ccd25ae0b833a26dd500aea1054c9bfc3ff46f901f1447bc4d0e7640\": container with ID starting with 396fbbe2ccd25ae0b833a26dd500aea1054c9bfc3ff46f901f1447bc4d0e7640 not found: ID does not exist" containerID="396fbbe2ccd25ae0b833a26dd500aea1054c9bfc3ff46f901f1447bc4d0e7640"
Oct 30 01:31:47.661471 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:47.661408    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396fbbe2ccd25ae0b833a26dd500aea1054c9bfc3ff46f901f1447bc4d0e7640"} err="failed to get container status \"396fbbe2ccd25ae0b833a26dd500aea1054c9bfc3ff46f901f1447bc4d0e7640\": rpc error: code = NotFound desc = could not find container \"396fbbe2ccd25ae0b833a26dd500aea1054c9bfc3ff46f901f1447bc4d0e7640\": container with ID starting with 396fbbe2ccd25ae0b833a26dd500aea1054c9bfc3ff46f901f1447bc4d0e7640 not found: ID does not exist"
Oct 30 01:31:47.732347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:47.732208    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"]
Oct 30 01:31:47.734208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:47.733144    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2f820e/payload-processing-external-model-e2e-extmod-a-2f820e-5cf7qxs74"]
Oct 30 01:31:49.430594 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:49.430551    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c3e4798-52b6-4a4b-bf06-6c5aece3eb25" path="/var/lib/kubelet/pods/9c3e4798-52b6-4a4b-bf06-6c5aece3eb25/volumes"
Oct 30 01:31:53.131882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.131849    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"]
Oct 30 01:31:53.132646 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.132631    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9c3e4798-52b6-4a4b-bf06-6c5aece3eb25" containerName="payload-processing"
Oct 30 01:31:53.132695 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.132647    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3e4798-52b6-4a4b-bf06-6c5aece3eb25" containerName="payload-processing"
Oct 30 01:31:53.132695 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.132658    4425 cpu_manager.go:414] "containerMap: removing container" podUID="37f9e186-d092-44a7-8d83-2c1f9e96388a" containerName="payload-processing"
Oct 30 01:31:53.132695 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.132663    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f9e186-d092-44a7-8d83-2c1f9e96388a" containerName="payload-processing"
Oct 30 01:31:53.132800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.132783    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3e4798-52b6-4a4b-bf06-6c5aece3eb25" containerName="payload-processing"
Oct 30 01:31:53.132800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.132797    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f9e186-d092-44a7-8d83-2c1f9e96388a" containerName="payload-processing"
Oct 30 01:31:53.138249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.138226    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"
Oct 30 01:31:53.165015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.164988    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:31:53.165195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.165018    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:31:53.169524 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.169497    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"]
Oct 30 01:31:53.289471 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.289434    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt6xp\" (UniqueName: \"kubernetes.io/projected/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-kube-api-access-bt6xp\") pod \"maas-api-revoke-sub-e4b26e33-q2zfr\" (UID: \"88e278fe-f9a9-4a71-b02d-4ef30ed8c049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"
Oct 30 01:31:53.289655 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.289613    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4b26e33-q2zfr\" (UID: \"88e278fe-f9a9-4a71-b02d-4ef30ed8c049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"
Oct 30 01:31:53.289732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.289680    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4b26e33-q2zfr\" (UID: \"88e278fe-f9a9-4a71-b02d-4ef30ed8c049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"
Oct 30 01:31:53.391423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.391332    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4b26e33-q2zfr\" (UID: \"88e278fe-f9a9-4a71-b02d-4ef30ed8c049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"
Oct 30 01:31:53.391592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.391445    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4b26e33-q2zfr\" (UID: \"88e278fe-f9a9-4a71-b02d-4ef30ed8c049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"
Oct 30 01:31:53.391592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.391569    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6xp\" (UniqueName: \"kubernetes.io/projected/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-kube-api-access-bt6xp\") pod \"maas-api-revoke-sub-e4b26e33-q2zfr\" (UID: \"88e278fe-f9a9-4a71-b02d-4ef30ed8c049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"
Oct 30 01:31:53.392224 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.392200    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4b26e33-q2zfr\" (UID: \"88e278fe-f9a9-4a71-b02d-4ef30ed8c049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"
Oct 30 01:31:53.394042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.394016    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4b26e33-q2zfr\" (UID: \"88e278fe-f9a9-4a71-b02d-4ef30ed8c049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"
Oct 30 01:31:53.414492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.414467    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt6xp\" (UniqueName: \"kubernetes.io/projected/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-kube-api-access-bt6xp\") pod \"maas-api-revoke-sub-e4b26e33-q2zfr\" (UID: \"88e278fe-f9a9-4a71-b02d-4ef30ed8c049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"
Oct 30 01:31:53.453134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.453109    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"
Oct 30 01:31:53.765546 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:31:53.765516    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e278fe_f9a9_4a71_b02d_4ef30ed8c049.slice/crio-67cf93db0c4894b1d8db543bdc73dedb60778ea0401fe5440c1b8242d9fa42b0 WatchSource:0}: Error finding container 67cf93db0c4894b1d8db543bdc73dedb60778ea0401fe5440c1b8242d9fa42b0: Status 404 returned error can't find the container with id 67cf93db0c4894b1d8db543bdc73dedb60778ea0401fe5440c1b8242d9fa42b0
Oct 30 01:31:53.766600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:53.766574    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"]
Oct 30 01:31:54.033517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:54.033428    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"]
Oct 30 01:31:54.040055 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:54.040031    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f19b72b-9jnj8"]
Oct 30 01:31:54.736586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:54.736548    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr" event={"ID":"88e278fe-f9a9-4a71-b02d-4ef30ed8c049","Type":"ContainerStarted","Data":"8105e7d297222d6e5f2a95713f43709acecc5d0fe1bf201655f7e4c18ea2ca86"}
Oct 30 01:31:54.736997 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:54.736615    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr" event={"ID":"88e278fe-f9a9-4a71-b02d-4ef30ed8c049","Type":"ContainerStarted","Data":"67cf93db0c4894b1d8db543bdc73dedb60778ea0401fe5440c1b8242d9fa42b0"}
Oct 30 01:31:54.796585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:54.796535    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr" podStartSLOduration=1.796521117 podStartE2EDuration="1.796521117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:54.792080277 +0000 UTC m=+3023.541735586" watchObservedRunningTime="2026-10-30 01:31:54.796521117 +0000 UTC m=+3023.546176426"
Oct 30 01:31:55.424706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:55.424662    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c21980-39ac-404a-b60e-03bb753a335e" path="/var/lib/kubelet/pods/43c21980-39ac-404a-b60e-03bb753a335e/volumes"
Oct 30 01:31:55.745816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:55.745725    4425 generic.go:356] "Generic (PLEG): container finished" podID="88e278fe-f9a9-4a71-b02d-4ef30ed8c049" containerID="8105e7d297222d6e5f2a95713f43709acecc5d0fe1bf201655f7e4c18ea2ca86" exitCode=0
Oct 30 01:31:55.746214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:55.745812    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr" event={"ID":"88e278fe-f9a9-4a71-b02d-4ef30ed8c049","Type":"ContainerDied","Data":"8105e7d297222d6e5f2a95713f43709acecc5d0fe1bf201655f7e4c18ea2ca86"}
Oct 30 01:31:56.920739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:56.920714    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"
Oct 30 01:31:57.040383 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:57.040353    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-maas-api-xks-ca\") pod \"88e278fe-f9a9-4a71-b02d-4ef30ed8c049\" (UID: \"88e278fe-f9a9-4a71-b02d-4ef30ed8c049\") "
Oct 30 01:31:57.040726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:57.040484    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-kube-api-access-bt6xp\" (UniqueName: \"kubernetes.io/projected/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-kube-api-access-bt6xp\") pod \"88e278fe-f9a9-4a71-b02d-4ef30ed8c049\" (UID: \"88e278fe-f9a9-4a71-b02d-4ef30ed8c049\") "
Oct 30 01:31:57.040726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:57.040504    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-maas-api-openshift-ca\") pod \"88e278fe-f9a9-4a71-b02d-4ef30ed8c049\" (UID: \"88e278fe-f9a9-4a71-b02d-4ef30ed8c049\") "
Oct 30 01:31:57.040970 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:57.040944    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-maas-api-openshift-ca" pod "88e278fe-f9a9-4a71-b02d-4ef30ed8c049" (UID: "88e278fe-f9a9-4a71-b02d-4ef30ed8c049"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:57.042483 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:57.042459    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-maas-api-xks-ca" pod "88e278fe-f9a9-4a71-b02d-4ef30ed8c049" (UID: "88e278fe-f9a9-4a71-b02d-4ef30ed8c049"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:31:57.042567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:57.042538    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-kube-api-access-bt6xp" pod "88e278fe-f9a9-4a71-b02d-4ef30ed8c049" (UID: "88e278fe-f9a9-4a71-b02d-4ef30ed8c049"). InnerVolumeSpecName "kube-api-access-bt6xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:57.142033 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:57.142003    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt6xp\" (UniqueName: \"kubernetes.io/projected/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-kube-api-access-bt6xp\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:57.142033 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:57.142030    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:57.142197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:57.142044    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e278fe-f9a9-4a71-b02d-4ef30ed8c049-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:31:57.763402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:57.763368    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr" event={"ID":"88e278fe-f9a9-4a71-b02d-4ef30ed8c049","Type":"ContainerDied","Data":"67cf93db0c4894b1d8db543bdc73dedb60778ea0401fe5440c1b8242d9fa42b0"}
Oct 30 01:31:57.763402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:57.763387    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"
Oct 30 01:31:57.763598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:31:57.763416    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67cf93db0c4894b1d8db543bdc73dedb60778ea0401fe5440c1b8242d9fa42b0"
Oct 30 01:32:05.026487 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.026444    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"]
Oct 30 01:32:05.027509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.027485    4425 cpu_manager.go:414] "containerMap: removing container" podUID="88e278fe-f9a9-4a71-b02d-4ef30ed8c049" containerName="revoke-keys"
Oct 30 01:32:05.027509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.027508    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e278fe-f9a9-4a71-b02d-4ef30ed8c049" containerName="revoke-keys"
Oct 30 01:32:05.027718 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.027704    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e278fe-f9a9-4a71-b02d-4ef30ed8c049" containerName="revoke-keys"
Oct 30 01:32:05.032390 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.032147    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"
Oct 30 01:32:05.054456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.054432    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"]
Oct 30 01:32:05.057354 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.057325    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-415c6b\""
Oct 30 01:32:05.103678 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.103648    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"]
Oct 30 01:32:05.108464 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.108442    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"
Oct 30 01:32:05.125070 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.125039    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-415c6b-dockercfg-njqlt\""
Oct 30 01:32:05.129953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.129928    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"]
Oct 30 01:32:05.143694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.143666    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rwjp\" (UniqueName: \"kubernetes.io/projected/11991a7b-d429-4f6e-a8a5-bc8352924d46-kube-api-access-6rwjp\") pod \"payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq\" (UID: \"11991a7b-d429-4f6e-a8a5-bc8352924d46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"
Oct 30 01:32:05.143781 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.143737    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979\" (UID: \"01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"
Oct 30 01:32:05.143961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.143936    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11991a7b-d429-4f6e-a8a5-bc8352924d46-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq\" (UID: \"11991a7b-d429-4f6e-a8a5-bc8352924d46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"
Oct 30 01:32:05.144033 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.143989    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xngpx\" (UniqueName: \"kubernetes.io/projected/01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9-kube-api-access-xngpx\") pod \"payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979\" (UID: \"01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"
Oct 30 01:32:05.245113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.245073    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11991a7b-d429-4f6e-a8a5-bc8352924d46-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq\" (UID: \"11991a7b-d429-4f6e-a8a5-bc8352924d46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"
Oct 30 01:32:05.245371 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.245123    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xngpx\" (UniqueName: \"kubernetes.io/projected/01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9-kube-api-access-xngpx\") pod \"payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979\" (UID: \"01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"
Oct 30 01:32:05.245371 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.245304    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rwjp\" (UniqueName: \"kubernetes.io/projected/11991a7b-d429-4f6e-a8a5-bc8352924d46-kube-api-access-6rwjp\") pod \"payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq\" (UID: \"11991a7b-d429-4f6e-a8a5-bc8352924d46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"
Oct 30 01:32:05.245371 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.245341    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979\" (UID: \"01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"
Oct 30 01:32:05.246069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.246045    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11991a7b-d429-4f6e-a8a5-bc8352924d46-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq\" (UID: \"11991a7b-d429-4f6e-a8a5-bc8352924d46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"
Oct 30 01:32:05.246134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.246092    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979\" (UID: \"01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"
Oct 30 01:32:05.272554 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.272526    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rwjp\" (UniqueName: \"kubernetes.io/projected/11991a7b-d429-4f6e-a8a5-bc8352924d46-kube-api-access-6rwjp\") pod \"payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq\" (UID: \"11991a7b-d429-4f6e-a8a5-bc8352924d46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"
Oct 30 01:32:05.273323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.273290    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xngpx\" (UniqueName: \"kubernetes.io/projected/01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9-kube-api-access-xngpx\") pod \"payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979\" (UID: \"01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"
Oct 30 01:32:05.346885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.346796    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"
Oct 30 01:32:05.424143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.423818    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"
Oct 30 01:32:05.431479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.431452    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"]
Oct 30 01:32:05.652588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.652555    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"]
Oct 30 01:32:05.653363 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:32:05.653331    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11991a7b_d429_4f6e_a8a5_bc8352924d46.slice/crio-2e667fb41a0db07f50067efa9b8c10f76a1b33cb5ef4ef74cc379e54587cf9a9 WatchSource:0}: Error finding container 2e667fb41a0db07f50067efa9b8c10f76a1b33cb5ef4ef74cc379e54587cf9a9: Status 404 returned error can't find the container with id 2e667fb41a0db07f50067efa9b8c10f76a1b33cb5ef4ef74cc379e54587cf9a9
Oct 30 01:32:05.739224 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.738833    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"]
Oct 30 01:32:05.739224 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:32:05.739116    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fbdfda_10b6_4b8d_9c44_9a5ac177e7b9.slice/crio-ec4045428d6e787222c9d330287df3378a872d984b573e3f29a3dc916e97bb6a WatchSource:0}: Error finding container ec4045428d6e787222c9d330287df3378a872d984b573e3f29a3dc916e97bb6a: Status 404 returned error can't find the container with id ec4045428d6e787222c9d330287df3378a872d984b573e3f29a3dc916e97bb6a
Oct 30 01:32:05.835488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.835455    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979" event={"ID":"01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9","Type":"ContainerStarted","Data":"ec4045428d6e787222c9d330287df3378a872d984b573e3f29a3dc916e97bb6a"}
Oct 30 01:32:05.837421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.837392    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq" event={"ID":"11991a7b-d429-4f6e-a8a5-bc8352924d46","Type":"ContainerStarted","Data":"e195f4fd8555d89721334a389cad5f9b24853ce17b405a88d880a7c892c1caee"}
Oct 30 01:32:05.837567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.837446    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq" event={"ID":"11991a7b-d429-4f6e-a8a5-bc8352924d46","Type":"ContainerStarted","Data":"2e667fb41a0db07f50067efa9b8c10f76a1b33cb5ef4ef74cc379e54587cf9a9"}
Oct 30 01:32:05.837731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.837713    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"
Oct 30 01:32:05.880882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:05.880833    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq" podStartSLOduration=1.8808183569999999 podStartE2EDuration="1.880818357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:05.875598663 +0000 UTC m=+3034.625253972" watchObservedRunningTime="2026-10-30 01:32:05.880818357 +0000 UTC m=+3034.630473666"
Oct 30 01:32:06.847577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:06.847534    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979" event={"ID":"01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9","Type":"ContainerStarted","Data":"61c877f0fd6203e13d36eee730dd3f4dbcf751de605fa15ae331ed8724095b46"}
Oct 30 01:32:06.847997 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:06.847629    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"
Oct 30 01:32:06.891546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:06.891486    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979" podStartSLOduration=1.891473613 podStartE2EDuration="1.891473613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:06.889322323 +0000 UTC m=+3035.638977634" watchObservedRunningTime="2026-10-30 01:32:06.891473613 +0000 UTC m=+3035.641128923"
Oct 30 01:32:07.561029 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:32:07.560990    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:32:10.432737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.432694    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc" podUID="e2354e0a-cbfa-4fe1-8903-842575a2f208" containerName="payload-processing" containerID="cri-o://b9f81c35fab167161626ffd495a1f389a4199bcd5bff0f38bd27f592a64a4097" gracePeriod=25
Oct 30 01:32:10.717104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.717078    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:32:10.821858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.821819    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2354e0a-cbfa-4fe1-8903-842575a2f208-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2354e0a-cbfa-4fe1-8903-842575a2f208-provider-credentials\") pod \"e2354e0a-cbfa-4fe1-8903-842575a2f208\" (UID: \"e2354e0a-cbfa-4fe1-8903-842575a2f208\") "
Oct 30 01:32:10.822036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.821897    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2354e0a-cbfa-4fe1-8903-842575a2f208-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2354e0a-cbfa-4fe1-8903-842575a2f208-routing-overlay-volume\") pod \"e2354e0a-cbfa-4fe1-8903-842575a2f208\" (UID: \"e2354e0a-cbfa-4fe1-8903-842575a2f208\") "
Oct 30 01:32:10.822036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.821933    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2354e0a-cbfa-4fe1-8903-842575a2f208-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2354e0a-cbfa-4fe1-8903-842575a2f208-plugins-config-volume\") pod \"e2354e0a-cbfa-4fe1-8903-842575a2f208\" (UID: \"e2354e0a-cbfa-4fe1-8903-842575a2f208\") "
Oct 30 01:32:10.822036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.822009    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2354e0a-cbfa-4fe1-8903-842575a2f208-kube-api-access-9wrfw\" (UniqueName: \"kubernetes.io/projected/e2354e0a-cbfa-4fe1-8903-842575a2f208-kube-api-access-9wrfw\") pod \"e2354e0a-cbfa-4fe1-8903-842575a2f208\" (UID: \"e2354e0a-cbfa-4fe1-8903-842575a2f208\") "
Oct 30 01:32:10.828062 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.826971    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2354e0a-cbfa-4fe1-8903-842575a2f208-plugins-config-volume" pod "e2354e0a-cbfa-4fe1-8903-842575a2f208" (UID: "e2354e0a-cbfa-4fe1-8903-842575a2f208"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:10.828062 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.827124    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2354e0a-cbfa-4fe1-8903-842575a2f208-kube-api-access-9wrfw" pod "e2354e0a-cbfa-4fe1-8903-842575a2f208" (UID: "e2354e0a-cbfa-4fe1-8903-842575a2f208"). InnerVolumeSpecName "kube-api-access-9wrfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:10.829811 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.829786    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2354e0a-cbfa-4fe1-8903-842575a2f208-routing-overlay-volume" pod "e2354e0a-cbfa-4fe1-8903-842575a2f208" (UID: "e2354e0a-cbfa-4fe1-8903-842575a2f208"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:32:10.832062 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.832012    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2354e0a-cbfa-4fe1-8903-842575a2f208-provider-credentials" pod "e2354e0a-cbfa-4fe1-8903-842575a2f208" (UID: "e2354e0a-cbfa-4fe1-8903-842575a2f208"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:32:10.882918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.882883    4425 generic.go:356] "Generic (PLEG): container finished" podID="e2354e0a-cbfa-4fe1-8903-842575a2f208" containerID="b9f81c35fab167161626ffd495a1f389a4199bcd5bff0f38bd27f592a64a4097" exitCode=0
Oct 30 01:32:10.883146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.882939    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"
Oct 30 01:32:10.883146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.882967    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc" event={"ID":"e2354e0a-cbfa-4fe1-8903-842575a2f208","Type":"ContainerDied","Data":"b9f81c35fab167161626ffd495a1f389a4199bcd5bff0f38bd27f592a64a4097"}
Oct 30 01:32:10.883146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.883027    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc" event={"ID":"e2354e0a-cbfa-4fe1-8903-842575a2f208","Type":"ContainerDied","Data":"238bc6f59b294f017ef17f6bae9905e15b46f04719c42bdbe284e6f27a69100f"}
Oct 30 01:32:10.883146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.883051    4425 scope.go:122] "RemoveContainer" containerID="b9f81c35fab167161626ffd495a1f389a4199bcd5bff0f38bd27f592a64a4097"
Oct 30 01:32:10.896396 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.896378    4425 scope.go:122] "RemoveContainer" containerID="b9f81c35fab167161626ffd495a1f389a4199bcd5bff0f38bd27f592a64a4097"
Oct 30 01:32:10.896654 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:32:10.896635    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f81c35fab167161626ffd495a1f389a4199bcd5bff0f38bd27f592a64a4097\": container with ID starting with b9f81c35fab167161626ffd495a1f389a4199bcd5bff0f38bd27f592a64a4097 not found: ID does not exist" containerID="b9f81c35fab167161626ffd495a1f389a4199bcd5bff0f38bd27f592a64a4097"
Oct 30 01:32:10.896712 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.896661    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f81c35fab167161626ffd495a1f389a4199bcd5bff0f38bd27f592a64a4097"} err="failed to get container status \"b9f81c35fab167161626ffd495a1f389a4199bcd5bff0f38bd27f592a64a4097\": rpc error: code = NotFound desc = could not find container \"b9f81c35fab167161626ffd495a1f389a4199bcd5bff0f38bd27f592a64a4097\": container with ID starting with b9f81c35fab167161626ffd495a1f389a4199bcd5bff0f38bd27f592a64a4097 not found: ID does not exist"
Oct 30 01:32:10.923971 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.923946    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2354e0a-cbfa-4fe1-8903-842575a2f208-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:32:10.923971 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.923970    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2354e0a-cbfa-4fe1-8903-842575a2f208-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:32:10.924108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.923981    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2354e0a-cbfa-4fe1-8903-842575a2f208-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:32:10.924108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.923989    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wrfw\" (UniqueName: \"kubernetes.io/projected/e2354e0a-cbfa-4fe1-8903-842575a2f208-kube-api-access-9wrfw\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:32:10.973549 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.973497    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"]
Oct 30 01:32:10.977761 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:10.977738    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-d7c4fc775-8nkkc"]
Oct 30 01:32:11.426148 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:11.426116    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2354e0a-cbfa-4fe1-8903-842575a2f208" path="/var/lib/kubelet/pods/e2354e0a-cbfa-4fe1-8903-842575a2f208/volumes"
Oct 30 01:32:13.483562 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.483533    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"]
Oct 30 01:32:13.484542 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.484518    4425 cpu_manager.go:414] "containerMap: removing container" podUID="e2354e0a-cbfa-4fe1-8903-842575a2f208" containerName="payload-processing"
Oct 30 01:32:13.484542 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.484540    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2354e0a-cbfa-4fe1-8903-842575a2f208" containerName="payload-processing"
Oct 30 01:32:13.484737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.484715    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2354e0a-cbfa-4fe1-8903-842575a2f208" containerName="payload-processing"
Oct 30 01:32:13.489650 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.489632    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"
Oct 30 01:32:13.513973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.513944    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"]
Oct 30 01:32:13.518622 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.518599    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-bd287f\""
Oct 30 01:32:13.523411 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.523389    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"]
Oct 30 01:32:13.528365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.528342    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"
Oct 30 01:32:13.553443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.553407    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-bd287f-dockercfg-2c2d2\""
Oct 30 01:32:13.559598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.559576    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"]
Oct 30 01:32:13.567498 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.567473    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc566\" (UniqueName: \"kubernetes.io/projected/29432241-060e-48e7-bb45-578a8391964b-kube-api-access-pc566\") pod \"payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh\" (UID: \"29432241-060e-48e7-bb45-578a8391964b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"
Oct 30 01:32:13.567802 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.567775    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29432241-060e-48e7-bb45-578a8391964b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh\" (UID: \"29432241-060e-48e7-bb45-578a8391964b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"
Oct 30 01:32:13.673199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.673066    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29432241-060e-48e7-bb45-578a8391964b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh\" (UID: \"29432241-060e-48e7-bb45-578a8391964b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"
Oct 30 01:32:13.673412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.673209    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc566\" (UniqueName: \"kubernetes.io/projected/29432241-060e-48e7-bb45-578a8391964b-kube-api-access-pc566\") pod \"payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh\" (UID: \"29432241-060e-48e7-bb45-578a8391964b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"
Oct 30 01:32:13.673412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.673245    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rskr7\" (UniqueName: \"kubernetes.io/projected/863be03f-aa38-4aeb-8f71-9ebd0685c391-kube-api-access-rskr7\") pod \"payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js\" (UID: \"863be03f-aa38-4aeb-8f71-9ebd0685c391\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"
Oct 30 01:32:13.673412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.673405    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/863be03f-aa38-4aeb-8f71-9ebd0685c391-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js\" (UID: \"863be03f-aa38-4aeb-8f71-9ebd0685c391\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"
Oct 30 01:32:13.682179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.674484    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29432241-060e-48e7-bb45-578a8391964b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh\" (UID: \"29432241-060e-48e7-bb45-578a8391964b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"
Oct 30 01:32:13.702890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.702827    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc566\" (UniqueName: \"kubernetes.io/projected/29432241-060e-48e7-bb45-578a8391964b-kube-api-access-pc566\") pod \"payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh\" (UID: \"29432241-060e-48e7-bb45-578a8391964b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"
Oct 30 01:32:13.774502 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.774467    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rskr7\" (UniqueName: \"kubernetes.io/projected/863be03f-aa38-4aeb-8f71-9ebd0685c391-kube-api-access-rskr7\") pod \"payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js\" (UID: \"863be03f-aa38-4aeb-8f71-9ebd0685c391\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"
Oct 30 01:32:13.774694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.774575    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/863be03f-aa38-4aeb-8f71-9ebd0685c391-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js\" (UID: \"863be03f-aa38-4aeb-8f71-9ebd0685c391\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"
Oct 30 01:32:13.775288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.775268    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/863be03f-aa38-4aeb-8f71-9ebd0685c391-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js\" (UID: \"863be03f-aa38-4aeb-8f71-9ebd0685c391\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"
Oct 30 01:32:13.796969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.796941    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rskr7\" (UniqueName: \"kubernetes.io/projected/863be03f-aa38-4aeb-8f71-9ebd0685c391-kube-api-access-rskr7\") pod \"payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js\" (UID: \"863be03f-aa38-4aeb-8f71-9ebd0685c391\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"
Oct 30 01:32:13.804977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.804956    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"
Oct 30 01:32:13.844582 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:13.844550    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"
Oct 30 01:32:14.125429 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:14.124909    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"]
Oct 30 01:32:14.128431 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:32:14.128403    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29432241_060e_48e7_bb45_578a8391964b.slice/crio-29b6086d123e8e4b5f8968695b1969bac36cf5322c97865549a3b279be8512e3 WatchSource:0}: Error finding container 29b6086d123e8e4b5f8968695b1969bac36cf5322c97865549a3b279be8512e3: Status 404 returned error can't find the container with id 29b6086d123e8e4b5f8968695b1969bac36cf5322c97865549a3b279be8512e3
Oct 30 01:32:14.174553 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:32:14.174525    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863be03f_aa38_4aeb_8f71_9ebd0685c391.slice/crio-b6bbbeb93d8a2ad19d20f8af32d129d19f369739590a85e660bb9a3cce177529 WatchSource:0}: Error finding container b6bbbeb93d8a2ad19d20f8af32d129d19f369739590a85e660bb9a3cce177529: Status 404 returned error can't find the container with id b6bbbeb93d8a2ad19d20f8af32d129d19f369739590a85e660bb9a3cce177529
Oct 30 01:32:14.175281 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:14.175255    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"]
Oct 30 01:32:14.925861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:14.925823    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh" event={"ID":"29432241-060e-48e7-bb45-578a8391964b","Type":"ContainerStarted","Data":"a5346c749393159b6e5526971cd495b693553885d1d391e7a79f750fcecf97d3"}
Oct 30 01:32:14.926410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:14.925885    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh" event={"ID":"29432241-060e-48e7-bb45-578a8391964b","Type":"ContainerStarted","Data":"29b6086d123e8e4b5f8968695b1969bac36cf5322c97865549a3b279be8512e3"}
Oct 30 01:32:14.926410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:14.925916    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"
Oct 30 01:32:14.927792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:14.927760    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js" event={"ID":"863be03f-aa38-4aeb-8f71-9ebd0685c391","Type":"ContainerStarted","Data":"a4564e34ba0fff98d71b70936012c2e1545ea8a243a4c9f6e1f8b2d6f6423e8d"}
Oct 30 01:32:14.927949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:14.927807    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js" event={"ID":"863be03f-aa38-4aeb-8f71-9ebd0685c391","Type":"ContainerStarted","Data":"b6bbbeb93d8a2ad19d20f8af32d129d19f369739590a85e660bb9a3cce177529"}
Oct 30 01:32:14.927949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:14.927848    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"
Oct 30 01:32:14.962429 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:14.962384    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh" podStartSLOduration=1.962371692 podStartE2EDuration="1.962371692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:14.961895097 +0000 UTC m=+3043.711550405" watchObservedRunningTime="2026-10-30 01:32:14.962371692 +0000 UTC m=+3043.712027000"
Oct 30 01:32:14.998265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:14.998220    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js" podStartSLOduration=1.9982072720000001 podStartE2EDuration="1.998207272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:14.996789274 +0000 UTC m=+3043.746444582" watchObservedRunningTime="2026-10-30 01:32:14.998207272 +0000 UTC m=+3043.747862581"
Oct 30 01:32:16.851436 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:16.851401    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"
Oct 30 01:32:17.860489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:17.860455    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"
Oct 30 01:32:25.938473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:25.938425    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"
Oct 30 01:32:25.938895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:25.938508    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"
Oct 30 01:32:33.688241 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:33.688207    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:32:35.112201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:35.112098    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"]
Oct 30 01:32:35.113521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:35.113480    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7530d38e-q29fk"]
Oct 30 01:32:35.426350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:35.426273    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e582e9-cc6a-4396-ad0d-960cbe9b89f1" path="/var/lib/kubelet/pods/a2e582e9-cc6a-4396-ad0d-960cbe9b89f1/volumes"
Oct 30 01:32:37.555667 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:32:37.555627    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:32:49.093231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:49.093192    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"]
Oct 30 01:32:49.106274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:49.106248    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8988efd-zw842"]
Oct 30 01:32:49.425208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:49.425111    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0994f410-4586-44ef-a171-2e3b946e60f0" path="/var/lib/kubelet/pods/0994f410-4586-44ef-a171-2e3b946e60f0/volumes"
Oct 30 01:32:52.373232 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:52.373190    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"]
Oct 30 01:32:52.380437 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:52.380414    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:32:52.405220 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:52.405189    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-415c6b-serving-cert\""
Oct 30 01:32:52.409819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:52.409794    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"]
Oct 30 01:32:52.475285 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:52.475254    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdrjp\" (UniqueName: \"kubernetes.io/projected/2994479c-7570-40e8-b898-e7e5404f22c1-kube-api-access-mdrjp\") pod \"maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp\" (UID: \"2994479c-7570-40e8-b898-e7e5404f22c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:32:52.475458 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:52.475368    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2994479c-7570-40e8-b898-e7e5404f22c1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp\" (UID: \"2994479c-7570-40e8-b898-e7e5404f22c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:32:52.475458 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:52.475389    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2994479c-7570-40e8-b898-e7e5404f22c1-maas-api-tls\") pod \"maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp\" (UID: \"2994479c-7570-40e8-b898-e7e5404f22c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:32:52.475576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:52.475556    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2994479c-7570-40e8-b898-e7e5404f22c1-metrics-certs\") pod \"maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp\" (UID: \"2994479c-7570-40e8-b898-e7e5404f22c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:32:52.576519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:52.576480    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdrjp\" (UniqueName: \"kubernetes.io/projected/2994479c-7570-40e8-b898-e7e5404f22c1-kube-api-access-mdrjp\") pod \"maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp\" (UID: \"2994479c-7570-40e8-b898-e7e5404f22c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:32:52.576714 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:52.576600    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2994479c-7570-40e8-b898-e7e5404f22c1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp\" (UID: \"2994479c-7570-40e8-b898-e7e5404f22c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:32:52.576714 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:52.576625    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2994479c-7570-40e8-b898-e7e5404f22c1-maas-api-tls\") pod \"maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp\" (UID: \"2994479c-7570-40e8-b898-e7e5404f22c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:32:52.576811 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:52.576773    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2994479c-7570-40e8-b898-e7e5404f22c1-metrics-certs\") pod \"maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp\" (UID: \"2994479c-7570-40e8-b898-e7e5404f22c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:32:52.577339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:52.577319    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2994479c-7570-40e8-b898-e7e5404f22c1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp\" (UID: \"2994479c-7570-40e8-b898-e7e5404f22c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:32:52.579402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:52.579379    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2994479c-7570-40e8-b898-e7e5404f22c1-metrics-certs\") pod \"maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp\" (UID: \"2994479c-7570-40e8-b898-e7e5404f22c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:32:52.579502 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:52.579422    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2994479c-7570-40e8-b898-e7e5404f22c1-maas-api-tls\") pod \"maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp\" (UID: \"2994479c-7570-40e8-b898-e7e5404f22c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:32:52.597730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:52.597705    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdrjp\" (UniqueName: \"kubernetes.io/projected/2994479c-7570-40e8-b898-e7e5404f22c1-kube-api-access-mdrjp\") pod \"maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp\" (UID: \"2994479c-7570-40e8-b898-e7e5404f22c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:32:52.695246 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:52.695136    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:32:53.005205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:53.005165    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"]
Oct 30 01:32:53.005702 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:32:53.005677    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2994479c_7570_40e8_b898_e7e5404f22c1.slice/crio-0d62a9f4e1f4fe655fee3d7e88aedbb8c2ce6b580493183a26966ca3382b4371 WatchSource:0}: Error finding container 0d62a9f4e1f4fe655fee3d7e88aedbb8c2ce6b580493183a26966ca3382b4371: Status 404 returned error can't find the container with id 0d62a9f4e1f4fe655fee3d7e88aedbb8c2ce6b580493183a26966ca3382b4371
Oct 30 01:32:53.007266 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:53.007247    4425 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:32:53.283078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:53.283037    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp" event={"ID":"2994479c-7570-40e8-b898-e7e5404f22c1","Type":"ContainerStarted","Data":"0d62a9f4e1f4fe655fee3d7e88aedbb8c2ce6b580493183a26966ca3382b4371"}
Oct 30 01:32:54.293641 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:54.293598    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp" event={"ID":"2994479c-7570-40e8-b898-e7e5404f22c1","Type":"ContainerStarted","Data":"f7b46a68b1d2c42459a3d0fc2657d2526c525ac7ad7dbb1efdf09a59339e26f7"}
Oct 30 01:32:54.294096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:54.293689    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:32:54.337764 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:32:54.337702    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp" podStartSLOduration=1.889174245 podStartE2EDuration="2.337684866s" totalImagesPullingTime="448.510621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:54.333026823 +0000 UTC m=+3083.082682133" watchObservedRunningTime="2026-10-30 01:32:54.337684866 +0000 UTC m=+3083.087340173"
Oct 30 01:33:00.309126 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:00.309095    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:33:01.527246 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:01.527205    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"]
Oct 30 01:33:01.532322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:01.532299    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:33:01.545305 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:01.545280    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-bd287f-serving-cert\""
Oct 30 01:33:01.560667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:01.560631    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"]
Oct 30 01:33:01.695205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:01.695138    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npn9b\" (UniqueName: \"kubernetes.io/projected/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-kube-api-access-npn9b\") pod \"maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw\" (UID: \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:33:01.695370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:01.695212    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-metrics-certs\") pod \"maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw\" (UID: \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:33:01.695370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:01.695325    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw\" (UID: \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:33:01.695482 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:01.695369    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-maas-api-tls\") pod \"maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw\" (UID: \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:33:01.796763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:01.796669    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npn9b\" (UniqueName: \"kubernetes.io/projected/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-kube-api-access-npn9b\") pod \"maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw\" (UID: \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:33:01.796763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:01.796710    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-metrics-certs\") pod \"maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw\" (UID: \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:33:01.796763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:01.796752    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw\" (UID: \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:33:01.797045 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:01.796780    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-maas-api-tls\") pod \"maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw\" (UID: \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:33:01.797551 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:01.797525    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw\" (UID: \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:33:01.799515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:01.799492    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-maas-api-tls\") pod \"maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw\" (UID: \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:33:01.799629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:01.799540    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-metrics-certs\") pod \"maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw\" (UID: \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:33:01.818320 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:01.818290    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npn9b\" (UniqueName: \"kubernetes.io/projected/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-kube-api-access-npn9b\") pod \"maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw\" (UID: \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:33:01.847819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:01.847784    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:33:02.166984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:02.166949    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"]
Oct 30 01:33:02.361365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:02.361329    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw" event={"ID":"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe","Type":"ContainerStarted","Data":"e28479b1674e020d1b18472185d04bb4a6d50674916088b47ae3116bc21a4b6c"}
Oct 30 01:33:03.371441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:03.371403    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw" event={"ID":"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe","Type":"ContainerStarted","Data":"51b6e1817bedcc27b730e8380c435a21ec12edf56e225bccbfc74c9f47262a6a"}
Oct 30 01:33:03.371927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:03.371487    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:33:03.416722 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:03.416665    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw" podStartSLOduration=2.041932935 podStartE2EDuration="2.416635209s" totalImagesPullingTime="374.702274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:03.412988112 +0000 UTC m=+3092.162643431" watchObservedRunningTime="2026-10-30 01:33:03.416635209 +0000 UTC m=+3092.166290519"
Oct 30 01:33:07.560954 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:33:07.560912    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:33:09.414643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:09.414609    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:33:11.032027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:11.031996    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"]
Oct 30 01:33:11.040733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:11.040706    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81e9e8ce-g5gh9"]
Oct 30 01:33:11.426106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:11.426026    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="788a53b3-4edf-4be5-89d4-9ae57c3b817f" path="/var/lib/kubelet/pods/788a53b3-4edf-4be5-89d4-9ae57c3b817f/volumes"
Oct 30 01:33:25.288137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:25.288060    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:33:35.341123 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:35.341081    4425 scope.go:122] "RemoveContainer" containerID="9dffa0fc7eb747f9c977f70870d812f745adbb7d2309a90ea772f4d0a45f87cf"
Oct 30 01:33:35.358118 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:35.358095    4425 scope.go:122] "RemoveContainer" containerID="e441f1828d45cc197dcac0f49c8da10bc94ab42397dfcdb9e2e8b32c964e7f48"
Oct 30 01:33:35.368025 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:35.368008    4425 scope.go:122] "RemoveContainer" containerID="6d96e22fbaf8a799b10348abdeeddafff1ccf38037d7201af95716d113b7f7ec"
Oct 30 01:33:37.556366 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:33:37.556330    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:33:53.755050 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:53.755007    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"]
Oct 30 01:33:53.762224 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:53.762200    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"
Oct 30 01:33:53.785686 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:53.785662    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:33:53.785799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:53.785764    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:33:53.804916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:53.804882    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"]
Oct 30 01:33:53.885717 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:53.885682    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84652b6a-91b8-472b-b17a-b715a34134ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c62fa63e-9749s\" (UID: \"84652b6a-91b8-472b-b17a-b715a34134ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"
Oct 30 01:33:53.886017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:53.885989    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2kx2\" (UniqueName: \"kubernetes.io/projected/84652b6a-91b8-472b-b17a-b715a34134ca-kube-api-access-s2kx2\") pod \"maas-api-revoke-sub-c62fa63e-9749s\" (UID: \"84652b6a-91b8-472b-b17a-b715a34134ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"
Oct 30 01:33:53.886017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:53.886024    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84652b6a-91b8-472b-b17a-b715a34134ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c62fa63e-9749s\" (UID: \"84652b6a-91b8-472b-b17a-b715a34134ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"
Oct 30 01:33:53.988532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:53.988499    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84652b6a-91b8-472b-b17a-b715a34134ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c62fa63e-9749s\" (UID: \"84652b6a-91b8-472b-b17a-b715a34134ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"
Oct 30 01:33:53.988728 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:53.988603    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2kx2\" (UniqueName: \"kubernetes.io/projected/84652b6a-91b8-472b-b17a-b715a34134ca-kube-api-access-s2kx2\") pod \"maas-api-revoke-sub-c62fa63e-9749s\" (UID: \"84652b6a-91b8-472b-b17a-b715a34134ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"
Oct 30 01:33:53.988728 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:53.988637    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84652b6a-91b8-472b-b17a-b715a34134ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c62fa63e-9749s\" (UID: \"84652b6a-91b8-472b-b17a-b715a34134ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"
Oct 30 01:33:53.989900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:53.989488    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84652b6a-91b8-472b-b17a-b715a34134ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c62fa63e-9749s\" (UID: \"84652b6a-91b8-472b-b17a-b715a34134ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"
Oct 30 01:33:54.011407 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.011370    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2kx2\" (UniqueName: \"kubernetes.io/projected/84652b6a-91b8-472b-b17a-b715a34134ca-kube-api-access-s2kx2\") pod \"maas-api-revoke-sub-c62fa63e-9749s\" (UID: \"84652b6a-91b8-472b-b17a-b715a34134ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"
Oct 30 01:33:54.011564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.011537    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84652b6a-91b8-472b-b17a-b715a34134ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c62fa63e-9749s\" (UID: \"84652b6a-91b8-472b-b17a-b715a34134ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"
Oct 30 01:33:54.046490 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.046454    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"]
Oct 30 01:33:54.051824 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.051805    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"
Oct 30 01:33:54.071855 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.071822    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"]
Oct 30 01:33:54.076583 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.076562    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"
Oct 30 01:33:54.199049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.199009    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ccc1766-07a9-4c61-8528-83f7666a7f41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92369f04-tpbk2\" (UID: \"1ccc1766-07a9-4c61-8528-83f7666a7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"
Oct 30 01:33:54.199331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.199308    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhfmj\" (UniqueName: \"kubernetes.io/projected/1ccc1766-07a9-4c61-8528-83f7666a7f41-kube-api-access-nhfmj\") pod \"maas-api-revoke-sub-92369f04-tpbk2\" (UID: \"1ccc1766-07a9-4c61-8528-83f7666a7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"
Oct 30 01:33:54.199879 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.199425    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ccc1766-07a9-4c61-8528-83f7666a7f41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92369f04-tpbk2\" (UID: \"1ccc1766-07a9-4c61-8528-83f7666a7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"
Oct 30 01:33:54.310356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.305392    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ccc1766-07a9-4c61-8528-83f7666a7f41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92369f04-tpbk2\" (UID: \"1ccc1766-07a9-4c61-8528-83f7666a7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"
Oct 30 01:33:54.310356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.306562    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhfmj\" (UniqueName: \"kubernetes.io/projected/1ccc1766-07a9-4c61-8528-83f7666a7f41-kube-api-access-nhfmj\") pod \"maas-api-revoke-sub-92369f04-tpbk2\" (UID: \"1ccc1766-07a9-4c61-8528-83f7666a7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"
Oct 30 01:33:54.310356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.306683    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ccc1766-07a9-4c61-8528-83f7666a7f41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92369f04-tpbk2\" (UID: \"1ccc1766-07a9-4c61-8528-83f7666a7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"
Oct 30 01:33:54.310356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.307546    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ccc1766-07a9-4c61-8528-83f7666a7f41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92369f04-tpbk2\" (UID: \"1ccc1766-07a9-4c61-8528-83f7666a7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"
Oct 30 01:33:54.311293 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.311112    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ccc1766-07a9-4c61-8528-83f7666a7f41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92369f04-tpbk2\" (UID: \"1ccc1766-07a9-4c61-8528-83f7666a7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"
Oct 30 01:33:54.333139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.333090    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhfmj\" (UniqueName: \"kubernetes.io/projected/1ccc1766-07a9-4c61-8528-83f7666a7f41-kube-api-access-nhfmj\") pod \"maas-api-revoke-sub-92369f04-tpbk2\" (UID: \"1ccc1766-07a9-4c61-8528-83f7666a7f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"
Oct 30 01:33:54.376810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.376391    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"
Oct 30 01:33:54.501507 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.501469    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"]
Oct 30 01:33:54.507995 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:33:54.507725    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84652b6a_91b8_472b_b17a_b715a34134ca.slice/crio-e3a0809f34916e6f4ebae7476f48eb82ab141caa1c47396df6420cb530090365 WatchSource:0}: Error finding container e3a0809f34916e6f4ebae7476f48eb82ab141caa1c47396df6420cb530090365: Status 404 returned error can't find the container with id e3a0809f34916e6f4ebae7476f48eb82ab141caa1c47396df6420cb530090365
Oct 30 01:33:54.751425 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.751378    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"]
Oct 30 01:33:54.755896 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:33:54.755867    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ccc1766_07a9_4c61_8528_83f7666a7f41.slice/crio-f5e46e24511f478ea46fb24b22ac61d3d0a1916023189c6bedb8a3f4c00f4f6b WatchSource:0}: Error finding container f5e46e24511f478ea46fb24b22ac61d3d0a1916023189c6bedb8a3f4c00f4f6b: Status 404 returned error can't find the container with id f5e46e24511f478ea46fb24b22ac61d3d0a1916023189c6bedb8a3f4c00f4f6b
Oct 30 01:33:54.812665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.812625    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2" event={"ID":"1ccc1766-07a9-4c61-8528-83f7666a7f41","Type":"ContainerStarted","Data":"f5e46e24511f478ea46fb24b22ac61d3d0a1916023189c6bedb8a3f4c00f4f6b"}
Oct 30 01:33:54.814714 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:54.814641    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s" event={"ID":"84652b6a-91b8-472b-b17a-b715a34134ca","Type":"ContainerStarted","Data":"e3a0809f34916e6f4ebae7476f48eb82ab141caa1c47396df6420cb530090365"}
Oct 30 01:33:55.824058 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:55.824024    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2" event={"ID":"1ccc1766-07a9-4c61-8528-83f7666a7f41","Type":"ContainerStarted","Data":"529b6b92334facecbf93038ab5cf740ad3998881408682fac2d6182b45362f1c"}
Oct 30 01:33:55.825776 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:55.825749    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s" event={"ID":"84652b6a-91b8-472b-b17a-b715a34134ca","Type":"ContainerStarted","Data":"69a588b38f885f195b177ad7498b17d9abf38cf3a03d85caf874d468af81bacb"}
Oct 30 01:33:55.875875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:55.875797    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2" podStartSLOduration=1.8757785249999999 podStartE2EDuration="1.875778525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:55.862994917 +0000 UTC m=+3144.612650238" watchObservedRunningTime="2026-10-30 01:33:55.875778525 +0000 UTC m=+3144.625433835"
Oct 30 01:33:55.903271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:55.903229    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s" podStartSLOduration=2.903217501 podStartE2EDuration="2.903217501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:55.89807638 +0000 UTC m=+3144.647731689" watchObservedRunningTime="2026-10-30 01:33:55.903217501 +0000 UTC m=+3144.652872840"
Oct 30 01:33:56.835556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:56.835522    4425 generic.go:356] "Generic (PLEG): container finished" podID="84652b6a-91b8-472b-b17a-b715a34134ca" containerID="69a588b38f885f195b177ad7498b17d9abf38cf3a03d85caf874d468af81bacb" exitCode=0
Oct 30 01:33:56.836108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:56.835608    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s" event={"ID":"84652b6a-91b8-472b-b17a-b715a34134ca","Type":"ContainerDied","Data":"69a588b38f885f195b177ad7498b17d9abf38cf3a03d85caf874d468af81bacb"}
Oct 30 01:33:56.837469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:56.837444    4425 generic.go:356] "Generic (PLEG): container finished" podID="1ccc1766-07a9-4c61-8528-83f7666a7f41" containerID="529b6b92334facecbf93038ab5cf740ad3998881408682fac2d6182b45362f1c" exitCode=0
Oct 30 01:33:56.837595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:56.837491    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2" event={"ID":"1ccc1766-07a9-4c61-8528-83f7666a7f41","Type":"ContainerDied","Data":"529b6b92334facecbf93038ab5cf740ad3998881408682fac2d6182b45362f1c"}
Oct 30 01:33:58.032240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.032208    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"
Oct 30 01:33:58.069611 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.069589    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"
Oct 30 01:33:58.078371 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.078343    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ccc1766-07a9-4c61-8528-83f7666a7f41-kube-api-access-nhfmj\" (UniqueName: \"kubernetes.io/projected/1ccc1766-07a9-4c61-8528-83f7666a7f41-kube-api-access-nhfmj\") pod \"1ccc1766-07a9-4c61-8528-83f7666a7f41\" (UID: \"1ccc1766-07a9-4c61-8528-83f7666a7f41\") "
Oct 30 01:33:58.078484 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.078434    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ccc1766-07a9-4c61-8528-83f7666a7f41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ccc1766-07a9-4c61-8528-83f7666a7f41-maas-api-xks-ca\") pod \"1ccc1766-07a9-4c61-8528-83f7666a7f41\" (UID: \"1ccc1766-07a9-4c61-8528-83f7666a7f41\") "
Oct 30 01:33:58.078570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.078548    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ccc1766-07a9-4c61-8528-83f7666a7f41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ccc1766-07a9-4c61-8528-83f7666a7f41-maas-api-openshift-ca\") pod \"1ccc1766-07a9-4c61-8528-83f7666a7f41\" (UID: \"1ccc1766-07a9-4c61-8528-83f7666a7f41\") "
Oct 30 01:33:58.078945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.078916    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ccc1766-07a9-4c61-8528-83f7666a7f41-maas-api-openshift-ca" pod "1ccc1766-07a9-4c61-8528-83f7666a7f41" (UID: "1ccc1766-07a9-4c61-8528-83f7666a7f41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:33:58.079330 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.079307    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ccc1766-07a9-4c61-8528-83f7666a7f41-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:33:58.080638 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.080610    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ccc1766-07a9-4c61-8528-83f7666a7f41-maas-api-xks-ca" pod "1ccc1766-07a9-4c61-8528-83f7666a7f41" (UID: "1ccc1766-07a9-4c61-8528-83f7666a7f41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:33:58.080860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.080837    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ccc1766-07a9-4c61-8528-83f7666a7f41-kube-api-access-nhfmj" pod "1ccc1766-07a9-4c61-8528-83f7666a7f41" (UID: "1ccc1766-07a9-4c61-8528-83f7666a7f41"). InnerVolumeSpecName "kube-api-access-nhfmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:33:58.180032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.179945    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84652b6a-91b8-472b-b17a-b715a34134ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84652b6a-91b8-472b-b17a-b715a34134ca-maas-api-openshift-ca\") pod \"84652b6a-91b8-472b-b17a-b715a34134ca\" (UID: \"84652b6a-91b8-472b-b17a-b715a34134ca\") "
Oct 30 01:33:58.180231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.180087    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84652b6a-91b8-472b-b17a-b715a34134ca-kube-api-access-s2kx2\" (UniqueName: \"kubernetes.io/projected/84652b6a-91b8-472b-b17a-b715a34134ca-kube-api-access-s2kx2\") pod \"84652b6a-91b8-472b-b17a-b715a34134ca\" (UID: \"84652b6a-91b8-472b-b17a-b715a34134ca\") "
Oct 30 01:33:58.180299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.180241    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84652b6a-91b8-472b-b17a-b715a34134ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84652b6a-91b8-472b-b17a-b715a34134ca-maas-api-xks-ca\") pod \"84652b6a-91b8-472b-b17a-b715a34134ca\" (UID: \"84652b6a-91b8-472b-b17a-b715a34134ca\") "
Oct 30 01:33:58.180433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.180410    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84652b6a-91b8-472b-b17a-b715a34134ca-maas-api-openshift-ca" pod "84652b6a-91b8-472b-b17a-b715a34134ca" (UID: "84652b6a-91b8-472b-b17a-b715a34134ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:33:58.180827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.180801    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhfmj\" (UniqueName: \"kubernetes.io/projected/1ccc1766-07a9-4c61-8528-83f7666a7f41-kube-api-access-nhfmj\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:33:58.180827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.180829    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84652b6a-91b8-472b-b17a-b715a34134ca-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:33:58.181028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.180843    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ccc1766-07a9-4c61-8528-83f7666a7f41-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:33:58.182429 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.182405    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84652b6a-91b8-472b-b17a-b715a34134ca-maas-api-xks-ca" pod "84652b6a-91b8-472b-b17a-b715a34134ca" (UID: "84652b6a-91b8-472b-b17a-b715a34134ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:33:58.182901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.182883    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84652b6a-91b8-472b-b17a-b715a34134ca-kube-api-access-s2kx2" pod "84652b6a-91b8-472b-b17a-b715a34134ca" (UID: "84652b6a-91b8-472b-b17a-b715a34134ca"). InnerVolumeSpecName "kube-api-access-s2kx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:33:58.281958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.281921    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2kx2\" (UniqueName: \"kubernetes.io/projected/84652b6a-91b8-472b-b17a-b715a34134ca-kube-api-access-s2kx2\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:33:58.281958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.281952    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84652b6a-91b8-472b-b17a-b715a34134ca-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:33:58.855537 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.855512    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"
Oct 30 01:33:58.855731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.855546    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2" event={"ID":"1ccc1766-07a9-4c61-8528-83f7666a7f41","Type":"ContainerDied","Data":"f5e46e24511f478ea46fb24b22ac61d3d0a1916023189c6bedb8a3f4c00f4f6b"}
Oct 30 01:33:58.855731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.855601    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5e46e24511f478ea46fb24b22ac61d3d0a1916023189c6bedb8a3f4c00f4f6b"
Oct 30 01:33:58.857054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.857030    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"
Oct 30 01:33:58.857237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.857066    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s" event={"ID":"84652b6a-91b8-472b-b17a-b715a34134ca","Type":"ContainerDied","Data":"e3a0809f34916e6f4ebae7476f48eb82ab141caa1c47396df6420cb530090365"}
Oct 30 01:33:58.857237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:33:58.857115    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3a0809f34916e6f4ebae7476f48eb82ab141caa1c47396df6420cb530090365"
Oct 30 01:34:04.032704 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.032668    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"]
Oct 30 01:34:04.041577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.041548    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49f5225c-6p52f"]
Oct 30 01:34:04.793358 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.793325    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"]
Oct 30 01:34:04.794184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.794146    4425 cpu_manager.go:414] "containerMap: removing container" podUID="84652b6a-91b8-472b-b17a-b715a34134ca" containerName="revoke-keys"
Oct 30 01:34:04.794184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.794177    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84652b6a-91b8-472b-b17a-b715a34134ca" containerName="revoke-keys"
Oct 30 01:34:04.794373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.794235    4425 cpu_manager.go:414] "containerMap: removing container" podUID="1ccc1766-07a9-4c61-8528-83f7666a7f41" containerName="revoke-keys"
Oct 30 01:34:04.794373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.794241    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ccc1766-07a9-4c61-8528-83f7666a7f41" containerName="revoke-keys"
Oct 30 01:34:04.794373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.794347    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="84652b6a-91b8-472b-b17a-b715a34134ca" containerName="revoke-keys"
Oct 30 01:34:04.794373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.794363    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ccc1766-07a9-4c61-8528-83f7666a7f41" containerName="revoke-keys"
Oct 30 01:34:04.798079 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.798063    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"
Oct 30 01:34:04.822807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.822785    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:34:04.822925 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.822825    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:34:04.827863 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.827833    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"]
Oct 30 01:34:04.869902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.869875    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bf54df-0367-4579-a29d-2ac44a5c45eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f53a04cb-qk2ss\" (UID: \"37bf54df-0367-4579-a29d-2ac44a5c45eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"
Oct 30 01:34:04.870054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.870035    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjz82\" (UniqueName: \"kubernetes.io/projected/37bf54df-0367-4579-a29d-2ac44a5c45eb-kube-api-access-fjz82\") pod \"maas-api-revoke-sub-f53a04cb-qk2ss\" (UID: \"37bf54df-0367-4579-a29d-2ac44a5c45eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"
Oct 30 01:34:04.870183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.870140    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bf54df-0367-4579-a29d-2ac44a5c45eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f53a04cb-qk2ss\" (UID: \"37bf54df-0367-4579-a29d-2ac44a5c45eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"
Oct 30 01:34:04.914875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.914844    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"]
Oct 30 01:34:04.919570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.919551    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"
Oct 30 01:34:04.938756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.938700    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"]
Oct 30 01:34:04.971806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.971780    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bf54df-0367-4579-a29d-2ac44a5c45eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f53a04cb-qk2ss\" (UID: \"37bf54df-0367-4579-a29d-2ac44a5c45eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"
Oct 30 01:34:04.971935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.971913    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjz82\" (UniqueName: \"kubernetes.io/projected/37bf54df-0367-4579-a29d-2ac44a5c45eb-kube-api-access-fjz82\") pod \"maas-api-revoke-sub-f53a04cb-qk2ss\" (UID: \"37bf54df-0367-4579-a29d-2ac44a5c45eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"
Oct 30 01:34:04.972003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.971989    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58aba58-d556-4af6-8e73-e4868ee15f47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7a1076f-jtmvt\" (UID: \"f58aba58-d556-4af6-8e73-e4868ee15f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"
Oct 30 01:34:04.972059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.972020    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzr94\" (UniqueName: \"kubernetes.io/projected/f58aba58-d556-4af6-8e73-e4868ee15f47-kube-api-access-wzr94\") pod \"maas-api-revoke-sub-c7a1076f-jtmvt\" (UID: \"f58aba58-d556-4af6-8e73-e4868ee15f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"
Oct 30 01:34:04.972125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.972070    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bf54df-0367-4579-a29d-2ac44a5c45eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f53a04cb-qk2ss\" (UID: \"37bf54df-0367-4579-a29d-2ac44a5c45eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"
Oct 30 01:34:04.972295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.972269    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58aba58-d556-4af6-8e73-e4868ee15f47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7a1076f-jtmvt\" (UID: \"f58aba58-d556-4af6-8e73-e4868ee15f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"
Oct 30 01:34:04.972584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.972565    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bf54df-0367-4579-a29d-2ac44a5c45eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f53a04cb-qk2ss\" (UID: \"37bf54df-0367-4579-a29d-2ac44a5c45eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"
Oct 30 01:34:04.974829 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.974807    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bf54df-0367-4579-a29d-2ac44a5c45eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f53a04cb-qk2ss\" (UID: \"37bf54df-0367-4579-a29d-2ac44a5c45eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"
Oct 30 01:34:04.992958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:04.992936    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjz82\" (UniqueName: \"kubernetes.io/projected/37bf54df-0367-4579-a29d-2ac44a5c45eb-kube-api-access-fjz82\") pod \"maas-api-revoke-sub-f53a04cb-qk2ss\" (UID: \"37bf54df-0367-4579-a29d-2ac44a5c45eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"
Oct 30 01:34:05.073871 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.073786    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58aba58-d556-4af6-8e73-e4868ee15f47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7a1076f-jtmvt\" (UID: \"f58aba58-d556-4af6-8e73-e4868ee15f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"
Oct 30 01:34:05.073871 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.073828    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzr94\" (UniqueName: \"kubernetes.io/projected/f58aba58-d556-4af6-8e73-e4868ee15f47-kube-api-access-wzr94\") pod \"maas-api-revoke-sub-c7a1076f-jtmvt\" (UID: \"f58aba58-d556-4af6-8e73-e4868ee15f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"
Oct 30 01:34:05.074353 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.073885    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58aba58-d556-4af6-8e73-e4868ee15f47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7a1076f-jtmvt\" (UID: \"f58aba58-d556-4af6-8e73-e4868ee15f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"
Oct 30 01:34:05.074625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.074603    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58aba58-d556-4af6-8e73-e4868ee15f47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7a1076f-jtmvt\" (UID: \"f58aba58-d556-4af6-8e73-e4868ee15f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"
Oct 30 01:34:05.076280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.076257    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58aba58-d556-4af6-8e73-e4868ee15f47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7a1076f-jtmvt\" (UID: \"f58aba58-d556-4af6-8e73-e4868ee15f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"
Oct 30 01:34:05.094652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.094627    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzr94\" (UniqueName: \"kubernetes.io/projected/f58aba58-d556-4af6-8e73-e4868ee15f47-kube-api-access-wzr94\") pod \"maas-api-revoke-sub-c7a1076f-jtmvt\" (UID: \"f58aba58-d556-4af6-8e73-e4868ee15f47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"
Oct 30 01:34:05.112469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.112447    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"
Oct 30 01:34:05.234055 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.234022    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"
Oct 30 01:34:05.360889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.359562    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"]
Oct 30 01:34:05.371358 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.371328    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"
Oct 30 01:34:05.385448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.383678    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"]
Oct 30 01:34:05.385448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.383843    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-190407\""
Oct 30 01:34:05.385448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.383947    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-190407-dockercfg-drbx7\""
Oct 30 01:34:05.434576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.434550    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf" path="/var/lib/kubelet/pods/f97d4fc2-afb0-4a9b-a05a-b4b6737e0caf/volumes"
Oct 30 01:34:05.434963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.434949    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"]
Oct 30 01:34:05.440807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.440789    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"]
Oct 30 01:34:05.440921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.440872    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"
Oct 30 01:34:05.444323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.444166    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"]
Oct 30 01:34:05.486000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.485966    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260bf897-bd50-4c1e-8e09-ebf7971e0a7e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p\" (UID: \"260bf897-bd50-4c1e-8e09-ebf7971e0a7e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"
Oct 30 01:34:05.486126 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.486103    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp9dw\" (UniqueName: \"kubernetes.io/projected/260bf897-bd50-4c1e-8e09-ebf7971e0a7e-kube-api-access-qp9dw\") pod \"payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p\" (UID: \"260bf897-bd50-4c1e-8e09-ebf7971e0a7e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"
Oct 30 01:34:05.551939 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.551888    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"]
Oct 30 01:34:05.564095 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:34:05.560649    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58aba58_d556_4af6_8e73_e4868ee15f47.slice/crio-604edd2fd5db107ac272b438e5442ff622c6a678c2436133e8a3188f4655ab80 WatchSource:0}: Error finding container 604edd2fd5db107ac272b438e5442ff622c6a678c2436133e8a3188f4655ab80: Status 404 returned error can't find the container with id 604edd2fd5db107ac272b438e5442ff622c6a678c2436133e8a3188f4655ab80
Oct 30 01:34:05.587590 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.587559    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9444\" (UniqueName: \"kubernetes.io/projected/6c0efd4d-3b76-4b1a-a856-fde91ea1fb80-kube-api-access-r9444\") pod \"payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn\" (UID: \"6c0efd4d-3b76-4b1a-a856-fde91ea1fb80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"
Oct 30 01:34:05.587737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.587684    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260bf897-bd50-4c1e-8e09-ebf7971e0a7e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p\" (UID: \"260bf897-bd50-4c1e-8e09-ebf7971e0a7e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"
Oct 30 01:34:05.587812 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.587777    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c0efd4d-3b76-4b1a-a856-fde91ea1fb80-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn\" (UID: \"6c0efd4d-3b76-4b1a-a856-fde91ea1fb80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"
Oct 30 01:34:05.587928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.587905    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp9dw\" (UniqueName: \"kubernetes.io/projected/260bf897-bd50-4c1e-8e09-ebf7971e0a7e-kube-api-access-qp9dw\") pod \"payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p\" (UID: \"260bf897-bd50-4c1e-8e09-ebf7971e0a7e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"
Oct 30 01:34:05.588797 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.588777    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260bf897-bd50-4c1e-8e09-ebf7971e0a7e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p\" (UID: \"260bf897-bd50-4c1e-8e09-ebf7971e0a7e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"
Oct 30 01:34:05.608911 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.608889    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp9dw\" (UniqueName: \"kubernetes.io/projected/260bf897-bd50-4c1e-8e09-ebf7971e0a7e-kube-api-access-qp9dw\") pod \"payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p\" (UID: \"260bf897-bd50-4c1e-8e09-ebf7971e0a7e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"
Oct 30 01:34:05.689626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.689555    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9444\" (UniqueName: \"kubernetes.io/projected/6c0efd4d-3b76-4b1a-a856-fde91ea1fb80-kube-api-access-r9444\") pod \"payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn\" (UID: \"6c0efd4d-3b76-4b1a-a856-fde91ea1fb80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"
Oct 30 01:34:05.689783 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.689644    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c0efd4d-3b76-4b1a-a856-fde91ea1fb80-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn\" (UID: \"6c0efd4d-3b76-4b1a-a856-fde91ea1fb80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"
Oct 30 01:34:05.690412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.690388    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c0efd4d-3b76-4b1a-a856-fde91ea1fb80-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn\" (UID: \"6c0efd4d-3b76-4b1a-a856-fde91ea1fb80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"
Oct 30 01:34:05.693605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.693589    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"
Oct 30 01:34:05.712485 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.712466    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9444\" (UniqueName: \"kubernetes.io/projected/6c0efd4d-3b76-4b1a-a856-fde91ea1fb80-kube-api-access-r9444\") pod \"payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn\" (UID: \"6c0efd4d-3b76-4b1a-a856-fde91ea1fb80\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"
Oct 30 01:34:05.770112 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.769423    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"
Oct 30 01:34:05.917492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.917451    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt" event={"ID":"f58aba58-d556-4af6-8e73-e4868ee15f47","Type":"ContainerStarted","Data":"604edd2fd5db107ac272b438e5442ff622c6a678c2436133e8a3188f4655ab80"}
Oct 30 01:34:05.918863 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:05.918836    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss" event={"ID":"37bf54df-0367-4579-a29d-2ac44a5c45eb","Type":"ContainerStarted","Data":"1e3a459fdca5af317d9e829422c0ded0313c78a3698d092429f941b3ad0ddc4c"}
Oct 30 01:34:06.002294 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:34:06.002272    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260bf897_bd50_4c1e_8e09_ebf7971e0a7e.slice/crio-fdce4e360bcf7b88c2bee93069c12ae302d4abcaea1fcff25c1ff86dbcbe70c6 WatchSource:0}: Error finding container fdce4e360bcf7b88c2bee93069c12ae302d4abcaea1fcff25c1ff86dbcbe70c6: Status 404 returned error can't find the container with id fdce4e360bcf7b88c2bee93069c12ae302d4abcaea1fcff25c1ff86dbcbe70c6
Oct 30 01:34:06.003266 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:06.003236    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"]
Oct 30 01:34:06.087104 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:34:06.087056    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0efd4d_3b76_4b1a_a856_fde91ea1fb80.slice/crio-6c92258fa5b7e6df77e625c73782be68268636a7e8de024eec17f29609558a0c WatchSource:0}: Error finding container 6c92258fa5b7e6df77e625c73782be68268636a7e8de024eec17f29609558a0c: Status 404 returned error can't find the container with id 6c92258fa5b7e6df77e625c73782be68268636a7e8de024eec17f29609558a0c
Oct 30 01:34:06.087603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:06.087543    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"]
Oct 30 01:34:06.927769 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:06.927682    4425 generic.go:356] "Generic (PLEG): container finished" podID="f58aba58-d556-4af6-8e73-e4868ee15f47" containerID="eb81a5ca322a0dbee221bdcef6ea798e96ec7df9795f84260a0e91795b890bab" exitCode=0
Oct 30 01:34:06.927922 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:06.927768    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt" event={"ID":"f58aba58-d556-4af6-8e73-e4868ee15f47","Type":"ContainerDied","Data":"eb81a5ca322a0dbee221bdcef6ea798e96ec7df9795f84260a0e91795b890bab"}
Oct 30 01:34:06.929325 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:06.929296    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p" event={"ID":"260bf897-bd50-4c1e-8e09-ebf7971e0a7e","Type":"ContainerStarted","Data":"edf9779c299ad90508c1f49c9576831a82bdd0356f3af1d2667d158f090c9df0"}
Oct 30 01:34:06.929486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:06.929339    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p" event={"ID":"260bf897-bd50-4c1e-8e09-ebf7971e0a7e","Type":"ContainerStarted","Data":"fdce4e360bcf7b88c2bee93069c12ae302d4abcaea1fcff25c1ff86dbcbe70c6"}
Oct 30 01:34:06.929486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:06.929359    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"
Oct 30 01:34:06.930738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:06.930716    4425 generic.go:356] "Generic (PLEG): container finished" podID="37bf54df-0367-4579-a29d-2ac44a5c45eb" containerID="6364be1f6261250f741a7fc8e20bcf8db5efb21e09a3de99c024479ba09138ff" exitCode=0
Oct 30 01:34:06.930859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:06.930790    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss" event={"ID":"37bf54df-0367-4579-a29d-2ac44a5c45eb","Type":"ContainerDied","Data":"6364be1f6261250f741a7fc8e20bcf8db5efb21e09a3de99c024479ba09138ff"}
Oct 30 01:34:06.936571 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:06.932889    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn" event={"ID":"6c0efd4d-3b76-4b1a-a856-fde91ea1fb80","Type":"ContainerStarted","Data":"3824912c441812c8d2076e735ff9764f80838dbef70f43bc0f0983c59274de2b"}
Oct 30 01:34:06.936571 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:06.932933    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn" event={"ID":"6c0efd4d-3b76-4b1a-a856-fde91ea1fb80","Type":"ContainerStarted","Data":"6c92258fa5b7e6df77e625c73782be68268636a7e8de024eec17f29609558a0c"}
Oct 30 01:34:06.936571 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:06.933611    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"
Oct 30 01:34:07.035186 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:07.035123    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p" podStartSLOduration=2.035108192 podStartE2EDuration="2.035108192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:07.032879094 +0000 UTC m=+3155.782534402" watchObservedRunningTime="2026-10-30 01:34:07.035108192 +0000 UTC m=+3155.784763529"
Oct 30 01:34:07.070121 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:07.070072    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn" podStartSLOduration=2.070058038 podStartE2EDuration="2.070058038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:07.067726913 +0000 UTC m=+3155.817382221" watchObservedRunningTime="2026-10-30 01:34:07.070058038 +0000 UTC m=+3155.819713350"
Oct 30 01:34:07.556933 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:07.556898    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:34:08.037772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.037738    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"]
Oct 30 01:34:08.126932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.126591    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"
Oct 30 01:34:08.165591 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.165566    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"
Oct 30 01:34:08.235460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.235377    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58aba58-d556-4af6-8e73-e4868ee15f47-kube-api-access-wzr94\" (UniqueName: \"kubernetes.io/projected/f58aba58-d556-4af6-8e73-e4868ee15f47-kube-api-access-wzr94\") pod \"f58aba58-d556-4af6-8e73-e4868ee15f47\" (UID: \"f58aba58-d556-4af6-8e73-e4868ee15f47\") "
Oct 30 01:34:08.235622 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.235555    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58aba58-d556-4af6-8e73-e4868ee15f47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58aba58-d556-4af6-8e73-e4868ee15f47-maas-api-xks-ca\") pod \"f58aba58-d556-4af6-8e73-e4868ee15f47\" (UID: \"f58aba58-d556-4af6-8e73-e4868ee15f47\") "
Oct 30 01:34:08.235684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.235656    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58aba58-d556-4af6-8e73-e4868ee15f47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58aba58-d556-4af6-8e73-e4868ee15f47-maas-api-openshift-ca\") pod \"f58aba58-d556-4af6-8e73-e4868ee15f47\" (UID: \"f58aba58-d556-4af6-8e73-e4868ee15f47\") "
Oct 30 01:34:08.236009 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.235981    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58aba58-d556-4af6-8e73-e4868ee15f47-maas-api-openshift-ca" pod "f58aba58-d556-4af6-8e73-e4868ee15f47" (UID: "f58aba58-d556-4af6-8e73-e4868ee15f47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:08.236562 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.236536    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58aba58-d556-4af6-8e73-e4868ee15f47-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:08.237680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.237656    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58aba58-d556-4af6-8e73-e4868ee15f47-maas-api-xks-ca" pod "f58aba58-d556-4af6-8e73-e4868ee15f47" (UID: "f58aba58-d556-4af6-8e73-e4868ee15f47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:34:08.238102 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.238083    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58aba58-d556-4af6-8e73-e4868ee15f47-kube-api-access-wzr94" pod "f58aba58-d556-4af6-8e73-e4868ee15f47" (UID: "f58aba58-d556-4af6-8e73-e4868ee15f47"). InnerVolumeSpecName "kube-api-access-wzr94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:08.337799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.337770    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37bf54df-0367-4579-a29d-2ac44a5c45eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bf54df-0367-4579-a29d-2ac44a5c45eb-maas-api-xks-ca\") pod \"37bf54df-0367-4579-a29d-2ac44a5c45eb\" (UID: \"37bf54df-0367-4579-a29d-2ac44a5c45eb\") "
Oct 30 01:34:08.337927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.337821    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37bf54df-0367-4579-a29d-2ac44a5c45eb-kube-api-access-fjz82\" (UniqueName: \"kubernetes.io/projected/37bf54df-0367-4579-a29d-2ac44a5c45eb-kube-api-access-fjz82\") pod \"37bf54df-0367-4579-a29d-2ac44a5c45eb\" (UID: \"37bf54df-0367-4579-a29d-2ac44a5c45eb\") "
Oct 30 01:34:08.337927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.337859    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37bf54df-0367-4579-a29d-2ac44a5c45eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bf54df-0367-4579-a29d-2ac44a5c45eb-maas-api-openshift-ca\") pod \"37bf54df-0367-4579-a29d-2ac44a5c45eb\" (UID: \"37bf54df-0367-4579-a29d-2ac44a5c45eb\") "
Oct 30 01:34:08.338298 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.338271    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37bf54df-0367-4579-a29d-2ac44a5c45eb-maas-api-openshift-ca" pod "37bf54df-0367-4579-a29d-2ac44a5c45eb" (UID: "37bf54df-0367-4579-a29d-2ac44a5c45eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:08.338489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.338474    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzr94\" (UniqueName: \"kubernetes.io/projected/f58aba58-d556-4af6-8e73-e4868ee15f47-kube-api-access-wzr94\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:08.338533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.338492    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58aba58-d556-4af6-8e73-e4868ee15f47-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:08.338533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.338503    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37bf54df-0367-4579-a29d-2ac44a5c45eb-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:08.339776 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.339755    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37bf54df-0367-4579-a29d-2ac44a5c45eb-maas-api-xks-ca" pod "37bf54df-0367-4579-a29d-2ac44a5c45eb" (UID: "37bf54df-0367-4579-a29d-2ac44a5c45eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:34:08.340202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.340150    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37bf54df-0367-4579-a29d-2ac44a5c45eb-kube-api-access-fjz82" pod "37bf54df-0367-4579-a29d-2ac44a5c45eb" (UID: "37bf54df-0367-4579-a29d-2ac44a5c45eb"). InnerVolumeSpecName "kube-api-access-fjz82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:08.439850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.439825    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjz82\" (UniqueName: \"kubernetes.io/projected/37bf54df-0367-4579-a29d-2ac44a5c45eb-kube-api-access-fjz82\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:08.439850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.439848    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37bf54df-0367-4579-a29d-2ac44a5c45eb-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:08.951051 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.951022    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"
Oct 30 01:34:08.951541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.951059    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt" event={"ID":"f58aba58-d556-4af6-8e73-e4868ee15f47","Type":"ContainerDied","Data":"604edd2fd5db107ac272b438e5442ff622c6a678c2436133e8a3188f4655ab80"}
Oct 30 01:34:08.951541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.951105    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="604edd2fd5db107ac272b438e5442ff622c6a678c2436133e8a3188f4655ab80"
Oct 30 01:34:08.952617 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.952595    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"
Oct 30 01:34:08.952770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.952607    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss" event={"ID":"37bf54df-0367-4579-a29d-2ac44a5c45eb","Type":"ContainerDied","Data":"1e3a459fdca5af317d9e829422c0ded0313c78a3698d092429f941b3ad0ddc4c"}
Oct 30 01:34:08.952770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:08.952659    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e3a459fdca5af317d9e829422c0ded0313c78a3698d092429f941b3ad0ddc4c"
Oct 30 01:34:09.015432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:09.015397    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"]
Oct 30 01:34:13.038414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.038370    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s" podUID="9d7877a3-3659-4043-bc54-ffdd6e9f2369" containerName="payload-processing" containerID="cri-o://0f5fec64245f1ef503b78f1ff7376385f596ecd6af1ecbb2906642ae4ff37005" gracePeriod=25
Oct 30 01:34:13.333233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.333202    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:34:13.401737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.401696    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7877a3-3659-4043-bc54-ffdd6e9f2369-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d7877a3-3659-4043-bc54-ffdd6e9f2369-routing-overlay-volume\") pod \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\" (UID: \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\") "
Oct 30 01:34:13.401954 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.401818    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7877a3-3659-4043-bc54-ffdd6e9f2369-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d7877a3-3659-4043-bc54-ffdd6e9f2369-provider-credentials\") pod \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\" (UID: \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\") "
Oct 30 01:34:13.401954 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.401846    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7877a3-3659-4043-bc54-ffdd6e9f2369-kube-api-access-hqbsc\" (UniqueName: \"kubernetes.io/projected/9d7877a3-3659-4043-bc54-ffdd6e9f2369-kube-api-access-hqbsc\") pod \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\" (UID: \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\") "
Oct 30 01:34:13.402098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.402065    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7877a3-3659-4043-bc54-ffdd6e9f2369-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7877a3-3659-4043-bc54-ffdd6e9f2369-plugins-config-volume\") pod \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\" (UID: \"9d7877a3-3659-4043-bc54-ffdd6e9f2369\") "
Oct 30 01:34:13.402173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.402113    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7877a3-3659-4043-bc54-ffdd6e9f2369-routing-overlay-volume" pod "9d7877a3-3659-4043-bc54-ffdd6e9f2369" (UID: "9d7877a3-3659-4043-bc54-ffdd6e9f2369"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:13.402568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.402538    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7877a3-3659-4043-bc54-ffdd6e9f2369-plugins-config-volume" pod "9d7877a3-3659-4043-bc54-ffdd6e9f2369" (UID: "9d7877a3-3659-4043-bc54-ffdd6e9f2369"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:13.402730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.402712    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d7877a3-3659-4043-bc54-ffdd6e9f2369-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:13.402807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.402735    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d7877a3-3659-4043-bc54-ffdd6e9f2369-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:13.404311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.404270    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7877a3-3659-4043-bc54-ffdd6e9f2369-kube-api-access-hqbsc" pod "9d7877a3-3659-4043-bc54-ffdd6e9f2369" (UID: "9d7877a3-3659-4043-bc54-ffdd6e9f2369"). InnerVolumeSpecName "kube-api-access-hqbsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:13.404538 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.404513    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7877a3-3659-4043-bc54-ffdd6e9f2369-provider-credentials" pod "9d7877a3-3659-4043-bc54-ffdd6e9f2369" (UID: "9d7877a3-3659-4043-bc54-ffdd6e9f2369"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:13.504280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.504211    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d7877a3-3659-4043-bc54-ffdd6e9f2369-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:13.504280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.504247    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqbsc\" (UniqueName: \"kubernetes.io/projected/9d7877a3-3659-4043-bc54-ffdd6e9f2369-kube-api-access-hqbsc\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:13.994141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.994106    4425 generic.go:356] "Generic (PLEG): container finished" podID="9d7877a3-3659-4043-bc54-ffdd6e9f2369" containerID="0f5fec64245f1ef503b78f1ff7376385f596ecd6af1ecbb2906642ae4ff37005" exitCode=0
Oct 30 01:34:13.994347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.994173    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"
Oct 30 01:34:13.994347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.994197    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s" event={"ID":"9d7877a3-3659-4043-bc54-ffdd6e9f2369","Type":"ContainerDied","Data":"0f5fec64245f1ef503b78f1ff7376385f596ecd6af1ecbb2906642ae4ff37005"}
Oct 30 01:34:13.994347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.994255    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s" event={"ID":"9d7877a3-3659-4043-bc54-ffdd6e9f2369","Type":"ContainerDied","Data":"a974ef2b53524c0821006daef03d958a7e18554eaaff795bd1b8fe5d465efeae"}
Oct 30 01:34:13.994347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:13.994277    4425 scope.go:122] "RemoveContainer" containerID="0f5fec64245f1ef503b78f1ff7376385f596ecd6af1ecbb2906642ae4ff37005"
Oct 30 01:34:14.005279 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.005224    4425 scope.go:122] "RemoveContainer" containerID="0f5fec64245f1ef503b78f1ff7376385f596ecd6af1ecbb2906642ae4ff37005"
Oct 30 01:34:14.005658 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:14.005632    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f5fec64245f1ef503b78f1ff7376385f596ecd6af1ecbb2906642ae4ff37005\": container with ID starting with 0f5fec64245f1ef503b78f1ff7376385f596ecd6af1ecbb2906642ae4ff37005 not found: ID does not exist" containerID="0f5fec64245f1ef503b78f1ff7376385f596ecd6af1ecbb2906642ae4ff37005"
Oct 30 01:34:14.005769 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.005665    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5fec64245f1ef503b78f1ff7376385f596ecd6af1ecbb2906642ae4ff37005"} err="failed to get container status \"0f5fec64245f1ef503b78f1ff7376385f596ecd6af1ecbb2906642ae4ff37005\": rpc error: code = NotFound desc = could not find container \"0f5fec64245f1ef503b78f1ff7376385f596ecd6af1ecbb2906642ae4ff37005\": container with ID starting with 0f5fec64245f1ef503b78f1ff7376385f596ecd6af1ecbb2906642ae4ff37005 not found: ID does not exist"
Oct 30 01:34:14.016473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.016438    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n" podUID="75600e6d-1dcf-4c52-b072-31666c67f51e" containerName="payload-processing" containerID="cri-o://229180e69a8030fdef34548540e475cf2649477f01e84c96cb639828bdf9bfd7" gracePeriod=25
Oct 30 01:34:14.041334 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.041314    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"]
Oct 30 01:34:14.050917 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.050896    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdclld7s"]
Oct 30 01:34:14.298975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.298951    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:34:14.314195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.314138    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75600e6d-1dcf-4c52-b072-31666c67f51e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75600e6d-1dcf-4c52-b072-31666c67f51e-routing-overlay-volume\") pod \"75600e6d-1dcf-4c52-b072-31666c67f51e\" (UID: \"75600e6d-1dcf-4c52-b072-31666c67f51e\") "
Oct 30 01:34:14.314317 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.314249    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75600e6d-1dcf-4c52-b072-31666c67f51e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/75600e6d-1dcf-4c52-b072-31666c67f51e-provider-credentials\") pod \"75600e6d-1dcf-4c52-b072-31666c67f51e\" (UID: \"75600e6d-1dcf-4c52-b072-31666c67f51e\") "
Oct 30 01:34:14.314488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.314466    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75600e6d-1dcf-4c52-b072-31666c67f51e-routing-overlay-volume" pod "75600e6d-1dcf-4c52-b072-31666c67f51e" (UID: "75600e6d-1dcf-4c52-b072-31666c67f51e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:14.314577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.314559    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75600e6d-1dcf-4c52-b072-31666c67f51e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75600e6d-1dcf-4c52-b072-31666c67f51e-plugins-config-volume\") pod \"75600e6d-1dcf-4c52-b072-31666c67f51e\" (UID: \"75600e6d-1dcf-4c52-b072-31666c67f51e\") "
Oct 30 01:34:14.314632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.314594    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75600e6d-1dcf-4c52-b072-31666c67f51e-kube-api-access-6bdf5\" (UniqueName: \"kubernetes.io/projected/75600e6d-1dcf-4c52-b072-31666c67f51e-kube-api-access-6bdf5\") pod \"75600e6d-1dcf-4c52-b072-31666c67f51e\" (UID: \"75600e6d-1dcf-4c52-b072-31666c67f51e\") "
Oct 30 01:34:14.314954 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.314932    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75600e6d-1dcf-4c52-b072-31666c67f51e-plugins-config-volume" pod "75600e6d-1dcf-4c52-b072-31666c67f51e" (UID: "75600e6d-1dcf-4c52-b072-31666c67f51e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:14.315262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.315239    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75600e6d-1dcf-4c52-b072-31666c67f51e-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:14.315262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.315257    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75600e6d-1dcf-4c52-b072-31666c67f51e-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:14.316358 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.316329    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75600e6d-1dcf-4c52-b072-31666c67f51e-provider-credentials" pod "75600e6d-1dcf-4c52-b072-31666c67f51e" (UID: "75600e6d-1dcf-4c52-b072-31666c67f51e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:14.316635 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.316616    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75600e6d-1dcf-4c52-b072-31666c67f51e-kube-api-access-6bdf5" pod "75600e6d-1dcf-4c52-b072-31666c67f51e" (UID: "75600e6d-1dcf-4c52-b072-31666c67f51e"). InnerVolumeSpecName "kube-api-access-6bdf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:14.416722 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.416689    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75600e6d-1dcf-4c52-b072-31666c67f51e-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:14.416722 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:14.416716    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bdf5\" (UniqueName: \"kubernetes.io/projected/75600e6d-1dcf-4c52-b072-31666c67f51e-kube-api-access-6bdf5\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:15.015959 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:15.015925    4425 generic.go:356] "Generic (PLEG): container finished" podID="75600e6d-1dcf-4c52-b072-31666c67f51e" containerID="229180e69a8030fdef34548540e475cf2649477f01e84c96cb639828bdf9bfd7" exitCode=0
Oct 30 01:34:15.016192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:15.015980    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"
Oct 30 01:34:15.016192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:15.016014    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n" event={"ID":"75600e6d-1dcf-4c52-b072-31666c67f51e","Type":"ContainerDied","Data":"229180e69a8030fdef34548540e475cf2649477f01e84c96cb639828bdf9bfd7"}
Oct 30 01:34:15.016192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:15.016098    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n" event={"ID":"75600e6d-1dcf-4c52-b072-31666c67f51e","Type":"ContainerDied","Data":"b54ea15dc80d44147fd41eff5f6f8012775376079539b18edafa447c9a191cf5"}
Oct 30 01:34:15.016192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:15.016125    4425 scope.go:122] "RemoveContainer" containerID="229180e69a8030fdef34548540e475cf2649477f01e84c96cb639828bdf9bfd7"
Oct 30 01:34:15.028316 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:15.028297    4425 scope.go:122] "RemoveContainer" containerID="229180e69a8030fdef34548540e475cf2649477f01e84c96cb639828bdf9bfd7"
Oct 30 01:34:15.028599 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:15.028569    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"229180e69a8030fdef34548540e475cf2649477f01e84c96cb639828bdf9bfd7\": container with ID starting with 229180e69a8030fdef34548540e475cf2649477f01e84c96cb639828bdf9bfd7 not found: ID does not exist" containerID="229180e69a8030fdef34548540e475cf2649477f01e84c96cb639828bdf9bfd7"
Oct 30 01:34:15.028687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:15.028598    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229180e69a8030fdef34548540e475cf2649477f01e84c96cb639828bdf9bfd7"} err="failed to get container status \"229180e69a8030fdef34548540e475cf2649477f01e84c96cb639828bdf9bfd7\": rpc error: code = NotFound desc = could not find container \"229180e69a8030fdef34548540e475cf2649477f01e84c96cb639828bdf9bfd7\": container with ID starting with 229180e69a8030fdef34548540e475cf2649477f01e84c96cb639828bdf9bfd7 not found: ID does not exist"
Oct 30 01:34:15.062003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:15.061975    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"]
Oct 30 01:34:15.071835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:15.071810    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a47b9a/payload-processing-external-model-e2e-extmod-a-a47b9a-5b9djx52n"]
Oct 30 01:34:15.338518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:15.338440    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:34:15.425656 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:15.425625    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75600e6d-1dcf-4c52-b072-31666c67f51e" path="/var/lib/kubelet/pods/75600e6d-1dcf-4c52-b072-31666c67f51e/volumes"
Oct 30 01:34:15.426056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:15.426042    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d7877a3-3659-4043-bc54-ffdd6e9f2369" path="/var/lib/kubelet/pods/9d7877a3-3659-4043-bc54-ffdd6e9f2369/volumes"
Oct 30 01:34:16.615328 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:16.615289    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"]
Oct 30 01:34:17.274981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:17.274942    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"]
Oct 30 01:34:17.286859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:17.284529    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx" podUID="ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c" containerName="sim" containerID="cri-o://9c76dcedd3e972584b7bd4f7a2ea81ab8aa4a19dab0132860b0ff7d8f1708bda" gracePeriod=30
Oct 30 01:34:17.566127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:17.566097    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"]
Oct 30 01:34:17.579349 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:17.579274    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"
Oct 30 01:34:17.676819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:17.676374    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c-kube-api-access-dk29q\" (UniqueName: \"kubernetes.io/projected/ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c-kube-api-access-dk29q\") pod \"ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c\" (UID: \"ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c\") "
Oct 30 01:34:17.676819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:17.676527    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c-tls\" (UniqueName: \"kubernetes.io/secret/ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c-tls\") pod \"ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c\" (UID: \"ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c\") "
Oct 30 01:34:17.679797 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:17.679766    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c-kube-api-access-dk29q" pod "ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c" (UID: "ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c"). InnerVolumeSpecName "kube-api-access-dk29q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:17.690820 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:17.690786    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c-tls" pod "ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c" (UID: "ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:34:17.778276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:17.778243    4425 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c-tls\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:17.778276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:17.778277    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk29q\" (UniqueName: \"kubernetes.io/projected/ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c-kube-api-access-dk29q\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:17.945329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:17.945254    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"
Oct 30 01:34:18.051975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.051941    4425 generic.go:356] "Generic (PLEG): container finished" podID="ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c" containerID="9c76dcedd3e972584b7bd4f7a2ea81ab8aa4a19dab0132860b0ff7d8f1708bda" exitCode=0
Oct 30 01:34:18.052203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.051998    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"
Oct 30 01:34:18.052203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.052023    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx" event={"ID":"ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c","Type":"ContainerDied","Data":"9c76dcedd3e972584b7bd4f7a2ea81ab8aa4a19dab0132860b0ff7d8f1708bda"}
Oct 30 01:34:18.052203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.052068    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx" event={"ID":"ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c","Type":"ContainerDied","Data":"591906ae6f4795965504b5955eb7773158ce874405db9c14801dd1533e0feda4"}
Oct 30 01:34:18.052203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.052090    4425 scope.go:122] "RemoveContainer" containerID="9c76dcedd3e972584b7bd4f7a2ea81ab8aa4a19dab0132860b0ff7d8f1708bda"
Oct 30 01:34:18.065598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.065580    4425 scope.go:122] "RemoveContainer" containerID="9c76dcedd3e972584b7bd4f7a2ea81ab8aa4a19dab0132860b0ff7d8f1708bda"
Oct 30 01:34:18.065858 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:18.065833    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c76dcedd3e972584b7bd4f7a2ea81ab8aa4a19dab0132860b0ff7d8f1708bda\": container with ID starting with 9c76dcedd3e972584b7bd4f7a2ea81ab8aa4a19dab0132860b0ff7d8f1708bda not found: ID does not exist" containerID="9c76dcedd3e972584b7bd4f7a2ea81ab8aa4a19dab0132860b0ff7d8f1708bda"
Oct 30 01:34:18.065934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.065870    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c76dcedd3e972584b7bd4f7a2ea81ab8aa4a19dab0132860b0ff7d8f1708bda"} err="failed to get container status \"9c76dcedd3e972584b7bd4f7a2ea81ab8aa4a19dab0132860b0ff7d8f1708bda\": rpc error: code = NotFound desc = could not find container \"9c76dcedd3e972584b7bd4f7a2ea81ab8aa4a19dab0132860b0ff7d8f1708bda\": container with ID starting with 9c76dcedd3e972584b7bd4f7a2ea81ab8aa4a19dab0132860b0ff7d8f1708bda not found: ID does not exist"
Oct 30 01:34:18.100293 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.100263    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"]
Oct 30 01:34:18.104366 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.104340    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-5e8b4d/e2e-usage-sim-055d3c-f6d9c6bc5-rvnvx"]
Oct 30 01:34:18.730110 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.730075    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"]
Oct 30 01:34:18.730895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.730878    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c" containerName="sim"
Oct 30 01:34:18.730895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.730895    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c" containerName="sim"
Oct 30 01:34:18.731053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.730907    4425 cpu_manager.go:414] "containerMap: removing container" podUID="f58aba58-d556-4af6-8e73-e4868ee15f47" containerName="revoke-keys"
Oct 30 01:34:18.731053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.730912    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58aba58-d556-4af6-8e73-e4868ee15f47" containerName="revoke-keys"
Oct 30 01:34:18.731053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.730923    4425 cpu_manager.go:414] "containerMap: removing container" podUID="75600e6d-1dcf-4c52-b072-31666c67f51e" containerName="payload-processing"
Oct 30 01:34:18.731053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.730928    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75600e6d-1dcf-4c52-b072-31666c67f51e" containerName="payload-processing"
Oct 30 01:34:18.731053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.730934    4425 cpu_manager.go:414] "containerMap: removing container" podUID="37bf54df-0367-4579-a29d-2ac44a5c45eb" containerName="revoke-keys"
Oct 30 01:34:18.731053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.730938    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37bf54df-0367-4579-a29d-2ac44a5c45eb" containerName="revoke-keys"
Oct 30 01:34:18.731053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.730986    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9d7877a3-3659-4043-bc54-ffdd6e9f2369" containerName="payload-processing"
Oct 30 01:34:18.731053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.730990    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7877a3-3659-4043-bc54-ffdd6e9f2369" containerName="payload-processing"
Oct 30 01:34:18.731503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.731093    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7877a3-3659-4043-bc54-ffdd6e9f2369" containerName="payload-processing"
Oct 30 01:34:18.731503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.731104    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c" containerName="sim"
Oct 30 01:34:18.731503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.731111    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58aba58-d556-4af6-8e73-e4868ee15f47" containerName="revoke-keys"
Oct 30 01:34:18.731503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.731118    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="75600e6d-1dcf-4c52-b072-31666c67f51e" containerName="payload-processing"
Oct 30 01:34:18.731503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.731124    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="37bf54df-0367-4579-a29d-2ac44a5c45eb" containerName="revoke-keys"
Oct 30 01:34:18.736848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.736758    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:18.761183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.754390    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"agc-cred-6fb94fa6b5f6\""
Oct 30 01:34:18.761183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.757373    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"]
Oct 30 01:34:18.761183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.760405    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"routing-overlay\""
Oct 30 01:34:18.761183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.760471    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"openshift-service-ca.crt\""
Oct 30 01:34:18.761183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.760534    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"kube-root-ca.crt\""
Oct 30 01:34:18.761183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.760635    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"payload-processing-external-model-e2e--42876bb9-dockercfg-snpkn\""
Oct 30 01:34:18.761183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.760680    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c7bba9\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c7bba9\""
Oct 30 01:34:18.796735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.796692    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf\" (UID: \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:18.796918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.796755    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf\" (UID: \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:18.796918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.796853    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7q2d\" (UniqueName: \"kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-kube-api-access-b7q2d\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf\" (UID: \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:18.797129 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.797099    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf\" (UID: \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:18.898938 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.898906    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf\" (UID: \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:18.899085 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.899008    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf\" (UID: \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:18.899085 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.899040    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf\" (UID: \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:18.899195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.899093    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7q2d\" (UniqueName: \"kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-kube-api-access-b7q2d\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf\" (UID: \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:18.899238 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:18.899222    4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c7bba9/agc-cred-6fb94fa6b5f6: secret "agc-cred-6fb94fa6b5f6" not found
Oct 30 01:34:18.899277 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:18.899242    4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf: secret "agc-cred-6fb94fa6b5f6" not found
Oct 30 01:34:18.899310 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:18.899305    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-provider-credentials podName:a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1 nodeName:}" failed. No retries permitted until 2026-10-30 01:34:19.399290669 +0000 UTC m=+3168.148945968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf" (UID: "a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1") : secret "agc-cred-6fb94fa6b5f6" not found
Oct 30 01:34:18.899778 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.899756    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf\" (UID: \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:18.899842 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.899826    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf\" (UID: \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:18.920108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.920084    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7q2d\" (UniqueName: \"kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-kube-api-access-b7q2d\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf\" (UID: \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:18.955817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:18.955773    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"
Oct 30 01:34:19.408099 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:19.408067    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf\" (UID: \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:19.408299 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:19.408234    4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c7bba9/agc-cred-6fb94fa6b5f6: secret "agc-cred-6fb94fa6b5f6" not found
Oct 30 01:34:19.408299 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:19.408254    4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf: secret "agc-cred-6fb94fa6b5f6" not found
Oct 30 01:34:19.408408 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:19.408311    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-provider-credentials podName:a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1 nodeName:}" failed. No retries permitted until 2026-10-30 01:34:20.408297808 +0000 UTC m=+3169.157953095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf" (UID: "a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1") : secret "agc-cred-6fb94fa6b5f6" not found
Oct 30 01:34:19.424677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:19.424648    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c" path="/var/lib/kubelet/pods/ba9d3959-08ef-4607-bf1e-4e29c3ec3d0c/volumes"
Oct 30 01:34:20.420741 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:20.420704    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf\" (UID: \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:20.421190 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:20.420850    4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c7bba9/agc-cred-6fb94fa6b5f6: secret "agc-cred-6fb94fa6b5f6" not found
Oct 30 01:34:20.421190 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:20.420870    4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf: secret "agc-cred-6fb94fa6b5f6" not found
Oct 30 01:34:20.421190 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:20.420932    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-provider-credentials podName:a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1 nodeName:}" failed. No retries permitted until 2026-10-30 01:34:22.42091746 +0000 UTC m=+3171.170572747 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf" (UID: "a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1") : secret "agc-cred-6fb94fa6b5f6" not found
Oct 30 01:34:21.403748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:21.403707    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"]
Oct 30 01:34:21.404027 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:21.403996    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf" podUID="a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1"
Oct 30 01:34:21.616181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:21.616112    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp" podUID="5077895f-c887-4fac-a91e-562c5311ece8" containerName="payload-processing" containerID="cri-o://3ac7b90c9f73f21fed9b632cd3dbada536c6ba08ce1fe4d5f8160ab86e062338" gracePeriod=25
Oct 30 01:34:21.907784 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:21.907761    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:34:22.041762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.041728    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5077895f-c887-4fac-a91e-562c5311ece8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5077895f-c887-4fac-a91e-562c5311ece8-routing-overlay-volume\") pod \"5077895f-c887-4fac-a91e-562c5311ece8\" (UID: \"5077895f-c887-4fac-a91e-562c5311ece8\") "
Oct 30 01:34:22.041969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.041806    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5077895f-c887-4fac-a91e-562c5311ece8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5077895f-c887-4fac-a91e-562c5311ece8-provider-credentials\") pod \"5077895f-c887-4fac-a91e-562c5311ece8\" (UID: \"5077895f-c887-4fac-a91e-562c5311ece8\") "
Oct 30 01:34:22.041969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.041871    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5077895f-c887-4fac-a91e-562c5311ece8-kube-api-access-6d6sg\" (UniqueName: \"kubernetes.io/projected/5077895f-c887-4fac-a91e-562c5311ece8-kube-api-access-6d6sg\") pod \"5077895f-c887-4fac-a91e-562c5311ece8\" (UID: \"5077895f-c887-4fac-a91e-562c5311ece8\") "
Oct 30 01:34:22.041969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.041920    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5077895f-c887-4fac-a91e-562c5311ece8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5077895f-c887-4fac-a91e-562c5311ece8-plugins-config-volume\") pod \"5077895f-c887-4fac-a91e-562c5311ece8\" (UID: \"5077895f-c887-4fac-a91e-562c5311ece8\") "
Oct 30 01:34:22.042146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.042103    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5077895f-c887-4fac-a91e-562c5311ece8-routing-overlay-volume" pod "5077895f-c887-4fac-a91e-562c5311ece8" (UID: "5077895f-c887-4fac-a91e-562c5311ece8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:22.042437 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.042407    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5077895f-c887-4fac-a91e-562c5311ece8-plugins-config-volume" pod "5077895f-c887-4fac-a91e-562c5311ece8" (UID: "5077895f-c887-4fac-a91e-562c5311ece8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:22.042730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.042710    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5077895f-c887-4fac-a91e-562c5311ece8-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:22.042822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.042735    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5077895f-c887-4fac-a91e-562c5311ece8-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:22.044086 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.044059    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5077895f-c887-4fac-a91e-562c5311ece8-provider-credentials" pod "5077895f-c887-4fac-a91e-562c5311ece8" (UID: "5077895f-c887-4fac-a91e-562c5311ece8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:22.044189 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.044121    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5077895f-c887-4fac-a91e-562c5311ece8-kube-api-access-6d6sg" pod "5077895f-c887-4fac-a91e-562c5311ece8" (UID: "5077895f-c887-4fac-a91e-562c5311ece8"). InnerVolumeSpecName "kube-api-access-6d6sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:22.088662 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.088633    4425 generic.go:356] "Generic (PLEG): container finished" podID="5077895f-c887-4fac-a91e-562c5311ece8" containerID="3ac7b90c9f73f21fed9b632cd3dbada536c6ba08ce1fe4d5f8160ab86e062338" exitCode=0
Oct 30 01:34:22.088807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.088683    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"
Oct 30 01:34:22.088807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.088717    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp" event={"ID":"5077895f-c887-4fac-a91e-562c5311ece8","Type":"ContainerDied","Data":"3ac7b90c9f73f21fed9b632cd3dbada536c6ba08ce1fe4d5f8160ab86e062338"}
Oct 30 01:34:22.088807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.088775    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp" event={"ID":"5077895f-c887-4fac-a91e-562c5311ece8","Type":"ContainerDied","Data":"d6ba7968ee3079b4c84250234b3fdca510e7a135d50653d24efdf137d66f3a75"}
Oct 30 01:34:22.088807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.088796    4425 scope.go:122] "RemoveContainer" containerID="3ac7b90c9f73f21fed9b632cd3dbada536c6ba08ce1fe4d5f8160ab86e062338"
Oct 30 01:34:22.089006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.088965    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:22.101465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.101442    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:22.101825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.101806    4425 scope.go:122] "RemoveContainer" containerID="3ac7b90c9f73f21fed9b632cd3dbada536c6ba08ce1fe4d5f8160ab86e062338"
Oct 30 01:34:22.102149 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:22.102127    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac7b90c9f73f21fed9b632cd3dbada536c6ba08ce1fe4d5f8160ab86e062338\": container with ID starting with 3ac7b90c9f73f21fed9b632cd3dbada536c6ba08ce1fe4d5f8160ab86e062338 not found: ID does not exist" containerID="3ac7b90c9f73f21fed9b632cd3dbada536c6ba08ce1fe4d5f8160ab86e062338"
Oct 30 01:34:22.102249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.102171    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac7b90c9f73f21fed9b632cd3dbada536c6ba08ce1fe4d5f8160ab86e062338"} err="failed to get container status \"3ac7b90c9f73f21fed9b632cd3dbada536c6ba08ce1fe4d5f8160ab86e062338\": rpc error: code = NotFound desc = could not find container \"3ac7b90c9f73f21fed9b632cd3dbada536c6ba08ce1fe4d5f8160ab86e062338\": container with ID starting with 3ac7b90c9f73f21fed9b632cd3dbada536c6ba08ce1fe4d5f8160ab86e062338 not found: ID does not exist"
Oct 30 01:34:22.135495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.135472    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"]
Oct 30 01:34:22.139687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.139654    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-499482/payload-processing-external-model-e2e-extmod-a-499482-57c726bdp"]
Oct 30 01:34:22.144647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.144625    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5077895f-c887-4fac-a91e-562c5311ece8-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:22.144647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.144648    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d6sg\" (UniqueName: \"kubernetes.io/projected/5077895f-c887-4fac-a91e-562c5311ece8-kube-api-access-6d6sg\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:22.245435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.245411    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-kube-api-access-b7q2d\" (UniqueName: \"kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-kube-api-access-b7q2d\") pod \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\" (UID: \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\") "
Oct 30 01:34:22.245588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.245475    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-routing-overlay-volume\") pod \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\" (UID: \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\") "
Oct 30 01:34:22.245588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.245500    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-plugins-config-volume\") pod \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\" (UID: \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\") "
Oct 30 01:34:22.245895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.245869    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-routing-overlay-volume" pod "a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1" (UID: "a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:22.245964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.245905    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-plugins-config-volume" pod "a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1" (UID: "a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:22.246336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.246314    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:22.246402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.246343    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:22.247821 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.247796    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-kube-api-access-b7q2d" pod "a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1" (UID: "a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1"). InnerVolumeSpecName "kube-api-access-b7q2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:22.357288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.348054    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7q2d\" (UniqueName: \"kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-kube-api-access-b7q2d\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:22.448726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.448693    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf\" (UID: \"a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1\") " pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:22.448887 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:22.448832    4425 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c7bba9/agc-cred-6fb94fa6b5f6: secret "agc-cred-6fb94fa6b5f6" not found
Oct 30 01:34:22.448887 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:22.448845    4425 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf: secret "agc-cred-6fb94fa6b5f6" not found
Oct 30 01:34:22.448963 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:22.448894    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-provider-credentials podName:a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1 nodeName:}" failed. No retries permitted until 2026-10-30 01:34:26.448879641 +0000 UTC m=+3175.198534942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf" (UID: "a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1") : secret "agc-cred-6fb94fa6b5f6" not found
Oct 30 01:34:22.566923 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.566883    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54" podUID="ad95246c-631e-459f-aef2-83d8303f6593" containerName="payload-processing" containerID="cri-o://9fa6308b4394b01643353a3d4b7414107c599e8cf409f570b0ac720ea14e493e" gracePeriod=25
Oct 30 01:34:22.837873 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.837846    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:34:22.954859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.954766    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad95246c-631e-459f-aef2-83d8303f6593-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad95246c-631e-459f-aef2-83d8303f6593-provider-credentials\") pod \"ad95246c-631e-459f-aef2-83d8303f6593\" (UID: \"ad95246c-631e-459f-aef2-83d8303f6593\") "
Oct 30 01:34:22.954859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.954824    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad95246c-631e-459f-aef2-83d8303f6593-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad95246c-631e-459f-aef2-83d8303f6593-plugins-config-volume\") pod \"ad95246c-631e-459f-aef2-83d8303f6593\" (UID: \"ad95246c-631e-459f-aef2-83d8303f6593\") "
Oct 30 01:34:22.955085 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.954901    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad95246c-631e-459f-aef2-83d8303f6593-kube-api-access-h9q9c\" (UniqueName: \"kubernetes.io/projected/ad95246c-631e-459f-aef2-83d8303f6593-kube-api-access-h9q9c\") pod \"ad95246c-631e-459f-aef2-83d8303f6593\" (UID: \"ad95246c-631e-459f-aef2-83d8303f6593\") "
Oct 30 01:34:22.955085 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.954931    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad95246c-631e-459f-aef2-83d8303f6593-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad95246c-631e-459f-aef2-83d8303f6593-routing-overlay-volume\") pod \"ad95246c-631e-459f-aef2-83d8303f6593\" (UID: \"ad95246c-631e-459f-aef2-83d8303f6593\") "
Oct 30 01:34:22.955330 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.955308    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad95246c-631e-459f-aef2-83d8303f6593-plugins-config-volume" pod "ad95246c-631e-459f-aef2-83d8303f6593" (UID: "ad95246c-631e-459f-aef2-83d8303f6593"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:22.955384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.955368    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad95246c-631e-459f-aef2-83d8303f6593-routing-overlay-volume" pod "ad95246c-631e-459f-aef2-83d8303f6593" (UID: "ad95246c-631e-459f-aef2-83d8303f6593"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:22.955602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.955581    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad95246c-631e-459f-aef2-83d8303f6593-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:22.955602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.955602    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad95246c-631e-459f-aef2-83d8303f6593-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:22.956976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.956957    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad95246c-631e-459f-aef2-83d8303f6593-provider-credentials" pod "ad95246c-631e-459f-aef2-83d8303f6593" (UID: "ad95246c-631e-459f-aef2-83d8303f6593"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:22.957188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:22.957146    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad95246c-631e-459f-aef2-83d8303f6593-kube-api-access-h9q9c" pod "ad95246c-631e-459f-aef2-83d8303f6593" (UID: "ad95246c-631e-459f-aef2-83d8303f6593"). InnerVolumeSpecName "kube-api-access-h9q9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:23.031251 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.031216    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"]
Oct 30 01:34:23.035388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.035361    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d61281a-grqfz"]
Oct 30 01:34:23.056486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.056458    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad95246c-631e-459f-aef2-83d8303f6593-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:23.056486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.056484    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9q9c\" (UniqueName: \"kubernetes.io/projected/ad95246c-631e-459f-aef2-83d8303f6593-kube-api-access-h9q9c\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:23.098614 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.098584    4425 generic.go:356] "Generic (PLEG): container finished" podID="ad95246c-631e-459f-aef2-83d8303f6593" containerID="9fa6308b4394b01643353a3d4b7414107c599e8cf409f570b0ac720ea14e493e" exitCode=0
Oct 30 01:34:23.098799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.098628    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"
Oct 30 01:34:23.098799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.098666    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54" event={"ID":"ad95246c-631e-459f-aef2-83d8303f6593","Type":"ContainerDied","Data":"9fa6308b4394b01643353a3d4b7414107c599e8cf409f570b0ac720ea14e493e"}
Oct 30 01:34:23.098799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.098714    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54" event={"ID":"ad95246c-631e-459f-aef2-83d8303f6593","Type":"ContainerDied","Data":"2bd9a2d2d0810a4f034c8aa7e8bf772e55e71242aac040314aa76db9ca2072b5"}
Oct 30 01:34:23.098799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.098737    4425 scope.go:122] "RemoveContainer" containerID="9fa6308b4394b01643353a3d4b7414107c599e8cf409f570b0ac720ea14e493e"
Oct 30 01:34:23.100241 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.100220    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"
Oct 30 01:34:23.111598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.111581    4425 scope.go:122] "RemoveContainer" containerID="9fa6308b4394b01643353a3d4b7414107c599e8cf409f570b0ac720ea14e493e"
Oct 30 01:34:23.111880 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:23.111863    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fa6308b4394b01643353a3d4b7414107c599e8cf409f570b0ac720ea14e493e\": container with ID starting with 9fa6308b4394b01643353a3d4b7414107c599e8cf409f570b0ac720ea14e493e not found: ID does not exist" containerID="9fa6308b4394b01643353a3d4b7414107c599e8cf409f570b0ac720ea14e493e"
Oct 30 01:34:23.111951 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.111886    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fa6308b4394b01643353a3d4b7414107c599e8cf409f570b0ac720ea14e493e"} err="failed to get container status \"9fa6308b4394b01643353a3d4b7414107c599e8cf409f570b0ac720ea14e493e\": rpc error: code = NotFound desc = could not find container \"9fa6308b4394b01643353a3d4b7414107c599e8cf409f570b0ac720ea14e493e\": container with ID starting with 9fa6308b4394b01643353a3d4b7414107c599e8cf409f570b0ac720ea14e493e not found: ID does not exist"
Oct 30 01:34:23.170059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.170027    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"]
Oct 30 01:34:23.176493 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.176464    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c7bba9/payload-processing-external-model-e2e-extmod-a-c7bba9-7b76q8pjf"]
Oct 30 01:34:23.202197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.202171    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"]
Oct 30 01:34:23.206219 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.206143    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-dd80cb/payload-processing-external-model-e2e-extmod-b-dd80cb-6fb5nwv54"]
Oct 30 01:34:23.259514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.259490    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:34:23.425694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.425665    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e0e257-3b3c-414f-a8ac-d4fef17ef3be" path="/var/lib/kubelet/pods/04e0e257-3b3c-414f-a8ac-d4fef17ef3be/volumes"
Oct 30 01:34:23.426102 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.426085    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5077895f-c887-4fac-a91e-562c5311ece8" path="/var/lib/kubelet/pods/5077895f-c887-4fac-a91e-562c5311ece8/volumes"
Oct 30 01:34:23.426501 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.426485    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1" path="/var/lib/kubelet/pods/a6c9daf2-84ab-4bdf-911f-c0ca4844c9a1/volumes"
Oct 30 01:34:23.426763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:23.426749    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad95246c-631e-459f-aef2-83d8303f6593" path="/var/lib/kubelet/pods/ad95246c-631e-459f-aef2-83d8303f6593/volumes"
Oct 30 01:34:31.005892 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.005817    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"]
Oct 30 01:34:31.006671 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.006654    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ad95246c-631e-459f-aef2-83d8303f6593" containerName="payload-processing"
Oct 30 01:34:31.006732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.006673    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad95246c-631e-459f-aef2-83d8303f6593" containerName="payload-processing"
Oct 30 01:34:31.006732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.006694    4425 cpu_manager.go:414] "containerMap: removing container" podUID="5077895f-c887-4fac-a91e-562c5311ece8" containerName="payload-processing"
Oct 30 01:34:31.006732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.006698    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5077895f-c887-4fac-a91e-562c5311ece8" containerName="payload-processing"
Oct 30 01:34:31.006841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.006823    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad95246c-631e-459f-aef2-83d8303f6593" containerName="payload-processing"
Oct 30 01:34:31.006841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.006836    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5077895f-c887-4fac-a91e-562c5311ece8" containerName="payload-processing"
Oct 30 01:34:31.012530 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.012510    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"
Oct 30 01:34:31.032192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.032150    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"]
Oct 30 01:34:31.036823 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.036798    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-db4dd2\""
Oct 30 01:34:31.058840 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.058810    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8qmh\" (UniqueName: \"kubernetes.io/projected/ab5a52e6-500d-4e2f-8e40-44b5f595388f-kube-api-access-k8qmh\") pod \"payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n\" (UID: \"ab5a52e6-500d-4e2f-8e40-44b5f595388f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"
Oct 30 01:34:31.059127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.059101    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5a52e6-500d-4e2f-8e40-44b5f595388f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n\" (UID: \"ab5a52e6-500d-4e2f-8e40-44b5f595388f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"
Oct 30 01:34:31.077736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.077706    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"]
Oct 30 01:34:31.083207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.083183    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"
Oct 30 01:34:31.096735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.096716    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-db4dd2-dockercfg-rgrbr\""
Oct 30 01:34:31.103306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.103285    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"]
Oct 30 01:34:31.160144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.160109    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8qmh\" (UniqueName: \"kubernetes.io/projected/ab5a52e6-500d-4e2f-8e40-44b5f595388f-kube-api-access-k8qmh\") pod \"payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n\" (UID: \"ab5a52e6-500d-4e2f-8e40-44b5f595388f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"
Oct 30 01:34:31.160349 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.160319    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5vvx\" (UniqueName: \"kubernetes.io/projected/e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5-kube-api-access-c5vvx\") pod \"payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp\" (UID: \"e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"
Oct 30 01:34:31.160528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.160485    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5a52e6-500d-4e2f-8e40-44b5f595388f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n\" (UID: \"ab5a52e6-500d-4e2f-8e40-44b5f595388f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"
Oct 30 01:34:31.160688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.160668    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp\" (UID: \"e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"
Oct 30 01:34:31.161466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.161392    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5a52e6-500d-4e2f-8e40-44b5f595388f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n\" (UID: \"ab5a52e6-500d-4e2f-8e40-44b5f595388f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"
Oct 30 01:34:31.186228 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.182860    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8qmh\" (UniqueName: \"kubernetes.io/projected/ab5a52e6-500d-4e2f-8e40-44b5f595388f-kube-api-access-k8qmh\") pod \"payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n\" (UID: \"ab5a52e6-500d-4e2f-8e40-44b5f595388f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"
Oct 30 01:34:31.262104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.262077    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp\" (UID: \"e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"
Oct 30 01:34:31.262307 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.262212    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5vvx\" (UniqueName: \"kubernetes.io/projected/e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5-kube-api-access-c5vvx\") pod \"payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp\" (UID: \"e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"
Oct 30 01:34:31.262898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.262872    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp\" (UID: \"e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"
Oct 30 01:34:31.285356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.285322    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5vvx\" (UniqueName: \"kubernetes.io/projected/e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5-kube-api-access-c5vvx\") pod \"payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp\" (UID: \"e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"
Oct 30 01:34:31.327449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.327421    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"
Oct 30 01:34:31.399540 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.398729    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"
Oct 30 01:34:31.648560 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:34:31.648528    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5a52e6_500d_4e2f_8e40_44b5f595388f.slice/crio-39b1985fe269a336b8c9acb6d0ef3e00b19995c03ab77ae0fd75ec89e8439508 WatchSource:0}: Error finding container 39b1985fe269a336b8c9acb6d0ef3e00b19995c03ab77ae0fd75ec89e8439508: Status 404 returned error can't find the container with id 39b1985fe269a336b8c9acb6d0ef3e00b19995c03ab77ae0fd75ec89e8439508
Oct 30 01:34:31.648732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.648705    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"]
Oct 30 01:34:31.761408 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:31.761382    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"]
Oct 30 01:34:31.761948 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:34:31.761922    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3adcbf8_6ae0_48a2_8a4e_d914a4c681b5.slice/crio-7e8b4ade888ef032ff2b2936f1042adc19ff9a4d97f3f66c9a9c66e1419e779c WatchSource:0}: Error finding container 7e8b4ade888ef032ff2b2936f1042adc19ff9a4d97f3f66c9a9c66e1419e779c: Status 404 returned error can't find the container with id 7e8b4ade888ef032ff2b2936f1042adc19ff9a4d97f3f66c9a9c66e1419e779c
Oct 30 01:34:32.181283 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:32.181191    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n" event={"ID":"ab5a52e6-500d-4e2f-8e40-44b5f595388f","Type":"ContainerStarted","Data":"d45b1ed6bd0100d4c4775eb39289a8b7ebb876a8f195670e4b3f158835bb19d9"}
Oct 30 01:34:32.181283 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:32.181263    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n" event={"ID":"ab5a52e6-500d-4e2f-8e40-44b5f595388f","Type":"ContainerStarted","Data":"39b1985fe269a336b8c9acb6d0ef3e00b19995c03ab77ae0fd75ec89e8439508"}
Oct 30 01:34:32.181750 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:32.181399    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"
Oct 30 01:34:32.183314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:32.183288    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp" event={"ID":"e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5","Type":"ContainerStarted","Data":"a5e282886e0696123bb6192b45abce0cfb3bc38f130f656f1c08921903d7cecb"}
Oct 30 01:34:32.183471 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:32.183335    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp" event={"ID":"e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5","Type":"ContainerStarted","Data":"7e8b4ade888ef032ff2b2936f1042adc19ff9a4d97f3f66c9a9c66e1419e779c"}
Oct 30 01:34:32.183676 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:32.183627    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"
Oct 30 01:34:32.218515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:32.218468    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n" podStartSLOduration=2.218455885 podStartE2EDuration="2.218455885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:32.216580813 +0000 UTC m=+3180.966236120" watchObservedRunningTime="2026-10-30 01:34:32.218455885 +0000 UTC m=+3180.968111206"
Oct 30 01:34:32.253523 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:32.253476    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp" podStartSLOduration=1.253460367 podStartE2EDuration="1.253460367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:32.251423774 +0000 UTC m=+3181.001079064" watchObservedRunningTime="2026-10-30 01:34:32.253460367 +0000 UTC m=+3181.003115681"
Oct 30 01:34:32.965417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:32.965379    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"]
Oct 30 01:34:32.970187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:32.970168    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"
Oct 30 01:34:32.991188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:32.991144    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"]
Oct 30 01:34:32.994464 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:32.994440    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b29d4a\""
Oct 30 01:34:32.994590 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:32.994559    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b29d4a-dockercfg-x7rvb\""
Oct 30 01:34:33.030228 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.030194    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"]
Oct 30 01:34:33.035216 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.035194    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"
Oct 30 01:34:33.055375 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.055347    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"]
Oct 30 01:34:33.135778 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.116207    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d47fca09-7546-4877-bd87-3dee036c13c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x\" (UID: \"d47fca09-7546-4877-bd87-3dee036c13c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"
Oct 30 01:34:33.138381 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.136640    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndgfj\" (UniqueName: \"kubernetes.io/projected/a2157c2a-b5cf-497e-8f76-0d4b290440ec-kube-api-access-ndgfj\") pod \"payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9\" (UID: \"a2157c2a-b5cf-497e-8f76-0d4b290440ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"
Oct 30 01:34:33.138786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.138756    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwpsp\" (UniqueName: \"kubernetes.io/projected/d47fca09-7546-4877-bd87-3dee036c13c9-kube-api-access-qwpsp\") pod \"payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x\" (UID: \"d47fca09-7546-4877-bd87-3dee036c13c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"
Oct 30 01:34:33.139054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.139029    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2157c2a-b5cf-497e-8f76-0d4b290440ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9\" (UID: \"a2157c2a-b5cf-497e-8f76-0d4b290440ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"
Oct 30 01:34:33.240196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.240085    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwpsp\" (UniqueName: \"kubernetes.io/projected/d47fca09-7546-4877-bd87-3dee036c13c9-kube-api-access-qwpsp\") pod \"payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x\" (UID: \"d47fca09-7546-4877-bd87-3dee036c13c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"
Oct 30 01:34:33.240196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.240168    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2157c2a-b5cf-497e-8f76-0d4b290440ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9\" (UID: \"a2157c2a-b5cf-497e-8f76-0d4b290440ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"
Oct 30 01:34:33.240606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.240410    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d47fca09-7546-4877-bd87-3dee036c13c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x\" (UID: \"d47fca09-7546-4877-bd87-3dee036c13c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"
Oct 30 01:34:33.240606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.240480    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndgfj\" (UniqueName: \"kubernetes.io/projected/a2157c2a-b5cf-497e-8f76-0d4b290440ec-kube-api-access-ndgfj\") pod \"payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9\" (UID: \"a2157c2a-b5cf-497e-8f76-0d4b290440ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"
Oct 30 01:34:33.240941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.240921    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2157c2a-b5cf-497e-8f76-0d4b290440ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9\" (UID: \"a2157c2a-b5cf-497e-8f76-0d4b290440ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"
Oct 30 01:34:33.241067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.241046    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d47fca09-7546-4877-bd87-3dee036c13c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x\" (UID: \"d47fca09-7546-4877-bd87-3dee036c13c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"
Oct 30 01:34:33.261083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.261059    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndgfj\" (UniqueName: \"kubernetes.io/projected/a2157c2a-b5cf-497e-8f76-0d4b290440ec-kube-api-access-ndgfj\") pod \"payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9\" (UID: \"a2157c2a-b5cf-497e-8f76-0d4b290440ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"
Oct 30 01:34:33.261252 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.261123    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwpsp\" (UniqueName: \"kubernetes.io/projected/d47fca09-7546-4877-bd87-3dee036c13c9-kube-api-access-qwpsp\") pod \"payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x\" (UID: \"d47fca09-7546-4877-bd87-3dee036c13c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"
Oct 30 01:34:33.290322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.290293    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"
Oct 30 01:34:33.350150 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.350119    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"
Oct 30 01:34:33.613078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.613051    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"]
Oct 30 01:34:33.613477 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:34:33.613456    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2157c2a_b5cf_497e_8f76_0d4b290440ec.slice/crio-a387e3642bbab125cc3cd45298af71bed806bc5cb437be4ddaab4597c3c9f169 WatchSource:0}: Error finding container a387e3642bbab125cc3cd45298af71bed806bc5cb437be4ddaab4597c3c9f169: Status 404 returned error can't find the container with id a387e3642bbab125cc3cd45298af71bed806bc5cb437be4ddaab4597c3c9f169
Oct 30 01:34:33.679059 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:34:33.679031    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47fca09_7546_4877_bd87_3dee036c13c9.slice/crio-59904834fd77f7bc7afc2ed0f0a64b493500a796f3d29b15c68b958b1b18f10a WatchSource:0}: Error finding container 59904834fd77f7bc7afc2ed0f0a64b493500a796f3d29b15c68b958b1b18f10a: Status 404 returned error can't find the container with id 59904834fd77f7bc7afc2ed0f0a64b493500a796f3d29b15c68b958b1b18f10a
Oct 30 01:34:33.679259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:33.679063    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"]
Oct 30 01:34:34.202274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:34.202223    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x" event={"ID":"d47fca09-7546-4877-bd87-3dee036c13c9","Type":"ContainerStarted","Data":"46bc0a71d895e01e29a113d8c97fafb5e609f956aa142c11394adb8030305cc4"}
Oct 30 01:34:34.202462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:34.202330    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"
Oct 30 01:34:34.202462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:34.202352    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x" event={"ID":"d47fca09-7546-4877-bd87-3dee036c13c9","Type":"ContainerStarted","Data":"59904834fd77f7bc7afc2ed0f0a64b493500a796f3d29b15c68b958b1b18f10a"}
Oct 30 01:34:34.204148 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:34.204117    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9" event={"ID":"a2157c2a-b5cf-497e-8f76-0d4b290440ec","Type":"ContainerStarted","Data":"7f3116ee376b4339eb5f69b8c675a5b042ffac72c0b493a870215694ac171335"}
Oct 30 01:34:34.204302 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:34.204182    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9" event={"ID":"a2157c2a-b5cf-497e-8f76-0d4b290440ec","Type":"ContainerStarted","Data":"a387e3642bbab125cc3cd45298af71bed806bc5cb437be4ddaab4597c3c9f169"}
Oct 30 01:34:34.204302 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:34.204211    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"
Oct 30 01:34:34.241234 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:34.241149    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x" podStartSLOduration=1.241133 podStartE2EDuration="1.241133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:34.237483792 +0000 UTC m=+3182.987139101" watchObservedRunningTime="2026-10-30 01:34:34.241133 +0000 UTC m=+3182.990788310"
Oct 30 01:34:34.275179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:34.275126    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9" podStartSLOduration=2.275110916 podStartE2EDuration="2.275110916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:34.272209314 +0000 UTC m=+3183.021864621" watchObservedRunningTime="2026-10-30 01:34:34.275110916 +0000 UTC m=+3183.024766224"
Oct 30 01:34:35.407343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:35.407313    4425 scope.go:122] "RemoveContainer" containerID="c17a5276789e039011da8532fc6d8cc065b6203a47c85c7dfa8803976accdfbd"
Oct 30 01:34:35.417749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:35.417732    4425 scope.go:122] "RemoveContainer" containerID="29bf32c6a30de007662724758e19aca303181b8fedacc9c6dc766c3c982a18b2"
Oct 30 01:34:37.555493 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:34:37.555451    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:34:41.771759 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:41.771723    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"]
Oct 30 01:34:41.778516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:41.778495    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:34:41.793776 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:41.793745    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ab0a2c-serving-cert\""
Oct 30 01:34:41.812757 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:41.812726    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"]
Oct 30 01:34:41.844385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:41.844357    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/709fe2e0-eb31-41da-bf22-daca403ffbbb-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8\" (UID: \"709fe2e0-eb31-41da-bf22-daca403ffbbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:34:41.844536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:41.844501    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/709fe2e0-eb31-41da-bf22-daca403ffbbb-metrics-certs\") pod \"maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8\" (UID: \"709fe2e0-eb31-41da-bf22-daca403ffbbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:34:41.844603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:41.844560    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/709fe2e0-eb31-41da-bf22-daca403ffbbb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8\" (UID: \"709fe2e0-eb31-41da-bf22-daca403ffbbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:34:41.844733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:41.844710    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hfw8\" (UniqueName: \"kubernetes.io/projected/709fe2e0-eb31-41da-bf22-daca403ffbbb-kube-api-access-6hfw8\") pod \"maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8\" (UID: \"709fe2e0-eb31-41da-bf22-daca403ffbbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:34:41.945888 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:41.945861    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/709fe2e0-eb31-41da-bf22-daca403ffbbb-metrics-certs\") pod \"maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8\" (UID: \"709fe2e0-eb31-41da-bf22-daca403ffbbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:34:41.946040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:41.945905    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/709fe2e0-eb31-41da-bf22-daca403ffbbb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8\" (UID: \"709fe2e0-eb31-41da-bf22-daca403ffbbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:34:41.946040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:41.945969    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hfw8\" (UniqueName: \"kubernetes.io/projected/709fe2e0-eb31-41da-bf22-daca403ffbbb-kube-api-access-6hfw8\") pod \"maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8\" (UID: \"709fe2e0-eb31-41da-bf22-daca403ffbbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:34:41.946125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:41.946044    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/709fe2e0-eb31-41da-bf22-daca403ffbbb-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8\" (UID: \"709fe2e0-eb31-41da-bf22-daca403ffbbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:34:41.946576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:41.946551    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/709fe2e0-eb31-41da-bf22-daca403ffbbb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8\" (UID: \"709fe2e0-eb31-41da-bf22-daca403ffbbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:34:41.948128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:41.948098    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/709fe2e0-eb31-41da-bf22-daca403ffbbb-metrics-certs\") pod \"maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8\" (UID: \"709fe2e0-eb31-41da-bf22-daca403ffbbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:34:41.948273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:41.948250    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/709fe2e0-eb31-41da-bf22-daca403ffbbb-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8\" (UID: \"709fe2e0-eb31-41da-bf22-daca403ffbbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:34:41.970601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:41.970574    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hfw8\" (UniqueName: \"kubernetes.io/projected/709fe2e0-eb31-41da-bf22-daca403ffbbb-kube-api-access-6hfw8\") pod \"maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8\" (UID: \"709fe2e0-eb31-41da-bf22-daca403ffbbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:34:42.094716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:42.094633    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:34:42.403222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:42.403197    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"]
Oct 30 01:34:42.403542 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:34:42.403520    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709fe2e0_eb31_41da_bf22_daca403ffbbb.slice/crio-cda95ab63ea555e532d99af5b92ec0f2d0a424969805872bc917a6e2a938e7ea WatchSource:0}: Error finding container cda95ab63ea555e532d99af5b92ec0f2d0a424969805872bc917a6e2a938e7ea: Status 404 returned error can't find the container with id cda95ab63ea555e532d99af5b92ec0f2d0a424969805872bc917a6e2a938e7ea
Oct 30 01:34:43.196396 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:43.196367    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"
Oct 30 01:34:43.196846 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:43.196738    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"
Oct 30 01:34:43.281309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:43.281266    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8" event={"ID":"709fe2e0-eb31-41da-bf22-daca403ffbbb","Type":"ContainerStarted","Data":"0e5e8d8f05b5691e24012d14a6bb3e0d2c46ed83e24a8c72524c8af26b2d78c1"}
Oct 30 01:34:43.281500 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:43.281334    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:34:43.281500 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:43.281354    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8" event={"ID":"709fe2e0-eb31-41da-bf22-daca403ffbbb","Type":"ContainerStarted","Data":"cda95ab63ea555e532d99af5b92ec0f2d0a424969805872bc917a6e2a938e7ea"}
Oct 30 01:34:43.324295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:43.324243    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8" podStartSLOduration=2.032592569 podStartE2EDuration="2.324228797s" totalImagesPullingTime="291.636228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:43.320219573 +0000 UTC m=+3192.069874881" watchObservedRunningTime="2026-10-30 01:34:43.324228797 +0000 UTC m=+3192.073884105"
Oct 30 01:34:45.216370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:45.216285    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"
Oct 30 01:34:45.216861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:45.216372    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"
Oct 30 01:34:49.295515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:49.295481    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:34:55.880696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:55.880653    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"]
Oct 30 01:34:55.886303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:55.886273    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:34:55.901641 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:55.901618    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-190407-serving-cert\""
Oct 30 01:34:55.916435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:55.916404    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"]
Oct 30 01:34:56.028505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:56.028472    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1651d619-a962-4574-8f03-92e0dda05db5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-190407-6db54969d8-nq2gg\" (UID: \"1651d619-a962-4574-8f03-92e0dda05db5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:34:56.028694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:56.028631    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj8tl\" (UniqueName: \"kubernetes.io/projected/1651d619-a962-4574-8f03-92e0dda05db5-kube-api-access-dj8tl\") pod \"maas-api-e2e-inf-b-190407-6db54969d8-nq2gg\" (UID: \"1651d619-a962-4574-8f03-92e0dda05db5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:34:56.028769 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:56.028754    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1651d619-a962-4574-8f03-92e0dda05db5-metrics-certs\") pod \"maas-api-e2e-inf-b-190407-6db54969d8-nq2gg\" (UID: \"1651d619-a962-4574-8f03-92e0dda05db5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:34:56.028819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:56.028797    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1651d619-a962-4574-8f03-92e0dda05db5-maas-api-tls\") pod \"maas-api-e2e-inf-b-190407-6db54969d8-nq2gg\" (UID: \"1651d619-a962-4574-8f03-92e0dda05db5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:34:56.130448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:56.130413    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1651d619-a962-4574-8f03-92e0dda05db5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-190407-6db54969d8-nq2gg\" (UID: \"1651d619-a962-4574-8f03-92e0dda05db5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:34:56.130661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:56.130587    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8tl\" (UniqueName: \"kubernetes.io/projected/1651d619-a962-4574-8f03-92e0dda05db5-kube-api-access-dj8tl\") pod \"maas-api-e2e-inf-b-190407-6db54969d8-nq2gg\" (UID: \"1651d619-a962-4574-8f03-92e0dda05db5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:34:56.130810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:56.130746    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1651d619-a962-4574-8f03-92e0dda05db5-metrics-certs\") pod \"maas-api-e2e-inf-b-190407-6db54969d8-nq2gg\" (UID: \"1651d619-a962-4574-8f03-92e0dda05db5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:34:56.130952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:56.130821    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1651d619-a962-4574-8f03-92e0dda05db5-maas-api-tls\") pod \"maas-api-e2e-inf-b-190407-6db54969d8-nq2gg\" (UID: \"1651d619-a962-4574-8f03-92e0dda05db5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:34:56.131250 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:56.131230    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1651d619-a962-4574-8f03-92e0dda05db5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-190407-6db54969d8-nq2gg\" (UID: \"1651d619-a962-4574-8f03-92e0dda05db5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:34:56.133272 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:56.133250    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1651d619-a962-4574-8f03-92e0dda05db5-metrics-certs\") pod \"maas-api-e2e-inf-b-190407-6db54969d8-nq2gg\" (UID: \"1651d619-a962-4574-8f03-92e0dda05db5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:34:56.133368 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:56.133307    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1651d619-a962-4574-8f03-92e0dda05db5-maas-api-tls\") pod \"maas-api-e2e-inf-b-190407-6db54969d8-nq2gg\" (UID: \"1651d619-a962-4574-8f03-92e0dda05db5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:34:56.152200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:56.152177    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj8tl\" (UniqueName: \"kubernetes.io/projected/1651d619-a962-4574-8f03-92e0dda05db5-kube-api-access-dj8tl\") pod \"maas-api-e2e-inf-b-190407-6db54969d8-nq2gg\" (UID: \"1651d619-a962-4574-8f03-92e0dda05db5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:34:56.202618 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:56.202584    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:34:56.517443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:56.517414    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"]
Oct 30 01:34:56.517587 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:34:56.517553    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1651d619_a962_4574_8f03_92e0dda05db5.slice/crio-6571dbffc13ae43b30c33ad345f8a089e13c96dc68811580f486859fb0761e13 WatchSource:0}: Error finding container 6571dbffc13ae43b30c33ad345f8a089e13c96dc68811580f486859fb0761e13: Status 404 returned error can't find the container with id 6571dbffc13ae43b30c33ad345f8a089e13c96dc68811580f486859fb0761e13
Oct 30 01:34:57.397814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:57.397775    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg" event={"ID":"1651d619-a962-4574-8f03-92e0dda05db5","Type":"ContainerStarted","Data":"8eb7f8eee21a77f3f92e096106d01a4d113eee84309f9504e53ae11d4b33b362"}
Oct 30 01:34:57.398331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:57.397870    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg" event={"ID":"1651d619-a962-4574-8f03-92e0dda05db5","Type":"ContainerStarted","Data":"6571dbffc13ae43b30c33ad345f8a089e13c96dc68811580f486859fb0761e13"}
Oct 30 01:34:57.398331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:57.397907    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:34:57.435843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:57.435786    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg" podStartSLOduration=2.078826898 podStartE2EDuration="2.435769561s" totalImagesPullingTime="356.942663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:57.43430416 +0000 UTC m=+3206.183959468" watchObservedRunningTime="2026-10-30 01:34:57.435769561 +0000 UTC m=+3206.185424869"
Oct 30 01:34:59.038552 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:59.038516    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"]
Oct 30 01:34:59.047684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:59.047660    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18b4734-wt9x8"]
Oct 30 01:34:59.425060 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:34:59.424988    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b914fb03-5880-4634-86c1-4d43a72ec34d" path="/var/lib/kubelet/pods/b914fb03-5880-4634-86c1-4d43a72ec34d/volumes"
Oct 30 01:35:03.412172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:03.412134    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:35:07.555972 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:35:07.555934    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:35:09.030954 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:09.030923    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"]
Oct 30 01:35:09.040054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:09.040027    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90aa3e58-7zgbr"]
Oct 30 01:35:09.426097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:09.426017    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ac63f3a-d1cc-4383-abe3-cbbff6e968d5" path="/var/lib/kubelet/pods/2ac63f3a-d1cc-4383-abe3-cbbff6e968d5/volumes"
Oct 30 01:35:18.537875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:18.537838    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:35:22.800265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:22.800228    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"]
Oct 30 01:35:22.806252 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:22.806228    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"
Oct 30 01:35:22.829169 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:22.829126    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:35:22.829328 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:22.829126    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:35:22.836492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:22.836467    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"]
Oct 30 01:35:22.906199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:22.906149    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96eabac0-f69e-4824-977e-1bc744137a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb5237dd-qqf2d\" (UID: \"96eabac0-f69e-4824-977e-1bc744137a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"
Oct 30 01:35:22.906396 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:22.906268    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96eabac0-f69e-4824-977e-1bc744137a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb5237dd-qqf2d\" (UID: \"96eabac0-f69e-4824-977e-1bc744137a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"
Oct 30 01:35:22.906396 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:22.906303    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h58w7\" (UniqueName: \"kubernetes.io/projected/96eabac0-f69e-4824-977e-1bc744137a8f-kube-api-access-h58w7\") pod \"maas-api-revoke-sub-cb5237dd-qqf2d\" (UID: \"96eabac0-f69e-4824-977e-1bc744137a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"
Oct 30 01:35:23.007454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:23.007411    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96eabac0-f69e-4824-977e-1bc744137a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb5237dd-qqf2d\" (UID: \"96eabac0-f69e-4824-977e-1bc744137a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"
Oct 30 01:35:23.007658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:23.007477    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96eabac0-f69e-4824-977e-1bc744137a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb5237dd-qqf2d\" (UID: \"96eabac0-f69e-4824-977e-1bc744137a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"
Oct 30 01:35:23.007658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:23.007507    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h58w7\" (UniqueName: \"kubernetes.io/projected/96eabac0-f69e-4824-977e-1bc744137a8f-kube-api-access-h58w7\") pod \"maas-api-revoke-sub-cb5237dd-qqf2d\" (UID: \"96eabac0-f69e-4824-977e-1bc744137a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"
Oct 30 01:35:23.008404 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:23.008375    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96eabac0-f69e-4824-977e-1bc744137a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb5237dd-qqf2d\" (UID: \"96eabac0-f69e-4824-977e-1bc744137a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"
Oct 30 01:35:23.009835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:23.009803    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96eabac0-f69e-4824-977e-1bc744137a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb5237dd-qqf2d\" (UID: \"96eabac0-f69e-4824-977e-1bc744137a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"
Oct 30 01:35:23.029126 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:23.029089    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h58w7\" (UniqueName: \"kubernetes.io/projected/96eabac0-f69e-4824-977e-1bc744137a8f-kube-api-access-h58w7\") pod \"maas-api-revoke-sub-cb5237dd-qqf2d\" (UID: \"96eabac0-f69e-4824-977e-1bc744137a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"
Oct 30 01:35:23.126573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:23.126475    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"
Oct 30 01:35:23.455184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:23.455128    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"]
Oct 30 01:35:23.618762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:23.618715    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d" event={"ID":"96eabac0-f69e-4824-977e-1bc744137a8f","Type":"ContainerStarted","Data":"c147f1f7d275683f194f924fcfb262b2972383f9de8b4457b0f74a081737dce6"}
Oct 30 01:35:24.633590 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:24.633550    4425 generic.go:356] "Generic (PLEG): container finished" podID="96eabac0-f69e-4824-977e-1bc744137a8f" containerID="ddbc8b866d0b7da01cefdd2c808c20ba7df1b395c74d030462a0344825513c9b" exitCode=0
Oct 30 01:35:24.634000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:24.633628    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d" event={"ID":"96eabac0-f69e-4824-977e-1bc744137a8f","Type":"ContainerDied","Data":"ddbc8b866d0b7da01cefdd2c808c20ba7df1b395c74d030462a0344825513c9b"}
Oct 30 01:35:25.821745 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:25.821713    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"
Oct 30 01:35:25.850231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:25.850196    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96eabac0-f69e-4824-977e-1bc744137a8f-kube-api-access-h58w7\" (UniqueName: \"kubernetes.io/projected/96eabac0-f69e-4824-977e-1bc744137a8f-kube-api-access-h58w7\") pod \"96eabac0-f69e-4824-977e-1bc744137a8f\" (UID: \"96eabac0-f69e-4824-977e-1bc744137a8f\") "
Oct 30 01:35:25.850449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:25.850437    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96eabac0-f69e-4824-977e-1bc744137a8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96eabac0-f69e-4824-977e-1bc744137a8f-maas-api-openshift-ca\") pod \"96eabac0-f69e-4824-977e-1bc744137a8f\" (UID: \"96eabac0-f69e-4824-977e-1bc744137a8f\") "
Oct 30 01:35:25.850495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:25.850466    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96eabac0-f69e-4824-977e-1bc744137a8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96eabac0-f69e-4824-977e-1bc744137a8f-maas-api-xks-ca\") pod \"96eabac0-f69e-4824-977e-1bc744137a8f\" (UID: \"96eabac0-f69e-4824-977e-1bc744137a8f\") "
Oct 30 01:35:25.850747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:25.850720    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96eabac0-f69e-4824-977e-1bc744137a8f-maas-api-openshift-ca" pod "96eabac0-f69e-4824-977e-1bc744137a8f" (UID: "96eabac0-f69e-4824-977e-1bc744137a8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:35:25.851016 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:25.850996    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96eabac0-f69e-4824-977e-1bc744137a8f-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:35:25.852565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:25.852533    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96eabac0-f69e-4824-977e-1bc744137a8f-maas-api-xks-ca" pod "96eabac0-f69e-4824-977e-1bc744137a8f" (UID: "96eabac0-f69e-4824-977e-1bc744137a8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:35:25.852565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:25.852543    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96eabac0-f69e-4824-977e-1bc744137a8f-kube-api-access-h58w7" pod "96eabac0-f69e-4824-977e-1bc744137a8f" (UID: "96eabac0-f69e-4824-977e-1bc744137a8f"). InnerVolumeSpecName "kube-api-access-h58w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:35:25.952491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:25.952403    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96eabac0-f69e-4824-977e-1bc744137a8f-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:35:25.952491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:25.952453    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h58w7\" (UniqueName: \"kubernetes.io/projected/96eabac0-f69e-4824-977e-1bc744137a8f-kube-api-access-h58w7\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:35:25.967934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:25.967898    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"]
Oct 30 01:35:25.968841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:25.968824    4425 cpu_manager.go:414] "containerMap: removing container" podUID="96eabac0-f69e-4824-977e-1bc744137a8f" containerName="revoke-keys"
Oct 30 01:35:25.968894 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:25.968843    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96eabac0-f69e-4824-977e-1bc744137a8f" containerName="revoke-keys"
Oct 30 01:35:25.968988 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:25.968978    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="96eabac0-f69e-4824-977e-1bc744137a8f" containerName="revoke-keys"
Oct 30 01:35:25.973279 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:25.973258    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:35:25.988491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:25.988457    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b29d4a-serving-cert\""
Oct 30 01:35:25.994068 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:25.994031    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"]
Oct 30 01:35:26.054224 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.054189    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-maas-api-tls\") pod \"maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf\" (UID: \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:35:26.054412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.054380    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mb6z\" (UniqueName: \"kubernetes.io/projected/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-kube-api-access-8mb6z\") pod \"maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf\" (UID: \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:35:26.054575 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.054559    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf\" (UID: \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:35:26.054629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.054618    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-metrics-certs\") pod \"maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf\" (UID: \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:35:26.156336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.156300    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf\" (UID: \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:35:26.156577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.156354    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-metrics-certs\") pod \"maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf\" (UID: \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:35:26.156577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.156430    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-maas-api-tls\") pod \"maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf\" (UID: \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:35:26.156577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.156538    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mb6z\" (UniqueName: \"kubernetes.io/projected/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-kube-api-access-8mb6z\") pod \"maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf\" (UID: \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:35:26.157095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.157070    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf\" (UID: \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:35:26.159022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.158999    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-maas-api-tls\") pod \"maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf\" (UID: \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:35:26.159118 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.159038    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-metrics-certs\") pod \"maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf\" (UID: \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:35:26.178081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.178049    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mb6z\" (UniqueName: \"kubernetes.io/projected/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-kube-api-access-8mb6z\") pod \"maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf\" (UID: \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:35:26.291049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.291015    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:35:26.601406 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.601379    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"]
Oct 30 01:35:26.601894 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:35:26.601873    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75fad1b_cc67_4381_baa7_3f0aa33c44b7.slice/crio-3525a9a11bcb305846feaabfd41b8b6543ce4630610eb04973fe872dfb909ec7 WatchSource:0}: Error finding container 3525a9a11bcb305846feaabfd41b8b6543ce4630610eb04973fe872dfb909ec7: Status 404 returned error can't find the container with id 3525a9a11bcb305846feaabfd41b8b6543ce4630610eb04973fe872dfb909ec7
Oct 30 01:35:26.654526 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.654487    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf" event={"ID":"a75fad1b-cc67-4381-baa7-3f0aa33c44b7","Type":"ContainerStarted","Data":"3525a9a11bcb305846feaabfd41b8b6543ce4630610eb04973fe872dfb909ec7"}
Oct 30 01:35:26.656069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.656044    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d" event={"ID":"96eabac0-f69e-4824-977e-1bc744137a8f","Type":"ContainerDied","Data":"c147f1f7d275683f194f924fcfb262b2972383f9de8b4457b0f74a081737dce6"}
Oct 30 01:35:26.656212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.656079    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"
Oct 30 01:35:26.656212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:26.656086    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c147f1f7d275683f194f924fcfb262b2972383f9de8b4457b0f74a081737dce6"
Oct 30 01:35:27.666705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:27.666668    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf" event={"ID":"a75fad1b-cc67-4381-baa7-3f0aa33c44b7","Type":"ContainerStarted","Data":"476622b7325aa88e65d94ae10456930ae4c0d0f9e7eaae3629b7f02bd42bade3"}
Oct 30 01:35:27.667193 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:27.666755    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:35:27.705587 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:27.705529    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf" podStartSLOduration=2.360341451 podStartE2EDuration="2.705513583s" totalImagesPullingTime="345.172132ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:27.704242646 +0000 UTC m=+3236.453897956" watchObservedRunningTime="2026-10-30 01:35:27.705513583 +0000 UTC m=+3236.455168892"
Oct 30 01:35:33.596673 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:33.596643    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"]
Oct 30 01:35:33.602505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:33.602484    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"
Oct 30 01:35:33.625535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:33.625509    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:35:33.626399 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:33.626379    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:35:33.633876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:33.633848    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"]
Oct 30 01:35:33.670048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:33.670023    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v7j8\" (UniqueName: \"kubernetes.io/projected/a67e991c-0211-406a-bbcb-3933ffdce18d-kube-api-access-2v7j8\") pod \"maas-api-revoke-sub-37c25e35-nmgpc\" (UID: \"a67e991c-0211-406a-bbcb-3933ffdce18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"
Oct 30 01:35:33.670180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:33.670082    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67e991c-0211-406a-bbcb-3933ffdce18d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37c25e35-nmgpc\" (UID: \"a67e991c-0211-406a-bbcb-3933ffdce18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"
Oct 30 01:35:33.670375 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:33.670348    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67e991c-0211-406a-bbcb-3933ffdce18d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37c25e35-nmgpc\" (UID: \"a67e991c-0211-406a-bbcb-3933ffdce18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"
Oct 30 01:35:33.679606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:33.679586    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:35:33.771412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:33.771381    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67e991c-0211-406a-bbcb-3933ffdce18d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37c25e35-nmgpc\" (UID: \"a67e991c-0211-406a-bbcb-3933ffdce18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"
Oct 30 01:35:33.771589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:33.771551    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67e991c-0211-406a-bbcb-3933ffdce18d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37c25e35-nmgpc\" (UID: \"a67e991c-0211-406a-bbcb-3933ffdce18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"
Oct 30 01:35:33.771670 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:33.771651    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v7j8\" (UniqueName: \"kubernetes.io/projected/a67e991c-0211-406a-bbcb-3933ffdce18d-kube-api-access-2v7j8\") pod \"maas-api-revoke-sub-37c25e35-nmgpc\" (UID: \"a67e991c-0211-406a-bbcb-3933ffdce18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"
Oct 30 01:35:33.772287 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:33.772254    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67e991c-0211-406a-bbcb-3933ffdce18d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37c25e35-nmgpc\" (UID: \"a67e991c-0211-406a-bbcb-3933ffdce18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"
Oct 30 01:35:33.774290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:33.774268    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67e991c-0211-406a-bbcb-3933ffdce18d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37c25e35-nmgpc\" (UID: \"a67e991c-0211-406a-bbcb-3933ffdce18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"
Oct 30 01:35:33.792698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:33.792676    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v7j8\" (UniqueName: \"kubernetes.io/projected/a67e991c-0211-406a-bbcb-3933ffdce18d-kube-api-access-2v7j8\") pod \"maas-api-revoke-sub-37c25e35-nmgpc\" (UID: \"a67e991c-0211-406a-bbcb-3933ffdce18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"
Oct 30 01:35:33.919110 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:33.919032    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"
Oct 30 01:35:34.227844 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:34.227811    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"]
Oct 30 01:35:34.227965 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:35:34.227885    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67e991c_0211_406a_bbcb_3933ffdce18d.slice/crio-38ea89cd811526c114239ba532eae5b4f77e24ade0dda0aec4817452540c45e7 WatchSource:0}: Error finding container 38ea89cd811526c114239ba532eae5b4f77e24ade0dda0aec4817452540c45e7: Status 404 returned error can't find the container with id 38ea89cd811526c114239ba532eae5b4f77e24ade0dda0aec4817452540c45e7
Oct 30 01:35:34.727911 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:34.727874    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc" event={"ID":"a67e991c-0211-406a-bbcb-3933ffdce18d","Type":"ContainerStarted","Data":"4e45161aea091496c0c721370392b8df7584da86b14ee7e44004e59b572caba4"}
Oct 30 01:35:34.728346 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:34.727937    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc" event={"ID":"a67e991c-0211-406a-bbcb-3933ffdce18d","Type":"ContainerStarted","Data":"38ea89cd811526c114239ba532eae5b4f77e24ade0dda0aec4817452540c45e7"}
Oct 30 01:35:34.763805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:34.763756    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc" podStartSLOduration=1.763742536 podStartE2EDuration="1.763742536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:34.762662953 +0000 UTC m=+3243.512318261" watchObservedRunningTime="2026-10-30 01:35:34.763742536 +0000 UTC m=+3243.513397845"
Oct 30 01:35:35.475423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:35.475340    4425 scope.go:122] "RemoveContainer" containerID="49beb32709139df625d2098b9215fa84c21149242137633743feb1f386ade5dc"
Oct 30 01:35:35.485822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:35.485801    4425 scope.go:122] "RemoveContainer" containerID="d8b61d977f8c04cb05dc1c611680434459a29d5107c488de6b5fdb8b629e91c4"
Oct 30 01:35:35.495662 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:35.495638    4425 scope.go:122] "RemoveContainer" containerID="c0d0230ff0b0ea27331da6dff70708ea324a34b3f9c7f4e737a86e9b3f7cd553"
Oct 30 01:35:35.738380 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:35.738267    4425 generic.go:356] "Generic (PLEG): container finished" podID="a67e991c-0211-406a-bbcb-3933ffdce18d" containerID="4e45161aea091496c0c721370392b8df7584da86b14ee7e44004e59b572caba4" exitCode=0
Oct 30 01:35:35.738380 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:35.738352    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc" event={"ID":"a67e991c-0211-406a-bbcb-3933ffdce18d","Type":"ContainerDied","Data":"4e45161aea091496c0c721370392b8df7584da86b14ee7e44004e59b572caba4"}
Oct 30 01:35:36.916797 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:36.916764    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"
Oct 30 01:35:37.013134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:37.013107    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67e991c-0211-406a-bbcb-3933ffdce18d-kube-api-access-2v7j8\" (UniqueName: \"kubernetes.io/projected/a67e991c-0211-406a-bbcb-3933ffdce18d-kube-api-access-2v7j8\") pod \"a67e991c-0211-406a-bbcb-3933ffdce18d\" (UID: \"a67e991c-0211-406a-bbcb-3933ffdce18d\") "
Oct 30 01:35:37.013365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:37.013345    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67e991c-0211-406a-bbcb-3933ffdce18d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67e991c-0211-406a-bbcb-3933ffdce18d-maas-api-openshift-ca\") pod \"a67e991c-0211-406a-bbcb-3933ffdce18d\" (UID: \"a67e991c-0211-406a-bbcb-3933ffdce18d\") "
Oct 30 01:35:37.013468 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:37.013451    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a67e991c-0211-406a-bbcb-3933ffdce18d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67e991c-0211-406a-bbcb-3933ffdce18d-maas-api-xks-ca\") pod \"a67e991c-0211-406a-bbcb-3933ffdce18d\" (UID: \"a67e991c-0211-406a-bbcb-3933ffdce18d\") "
Oct 30 01:35:37.013736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:37.013707    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67e991c-0211-406a-bbcb-3933ffdce18d-maas-api-openshift-ca" pod "a67e991c-0211-406a-bbcb-3933ffdce18d" (UID: "a67e991c-0211-406a-bbcb-3933ffdce18d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:35:37.014037 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:37.014019    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67e991c-0211-406a-bbcb-3933ffdce18d-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:35:37.015307 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:37.015284    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67e991c-0211-406a-bbcb-3933ffdce18d-kube-api-access-2v7j8" pod "a67e991c-0211-406a-bbcb-3933ffdce18d" (UID: "a67e991c-0211-406a-bbcb-3933ffdce18d"). InnerVolumeSpecName "kube-api-access-2v7j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:35:37.015502 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:37.015483    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67e991c-0211-406a-bbcb-3933ffdce18d-maas-api-xks-ca" pod "a67e991c-0211-406a-bbcb-3933ffdce18d" (UID: "a67e991c-0211-406a-bbcb-3933ffdce18d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:35:37.115221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:37.115192    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v7j8\" (UniqueName: \"kubernetes.io/projected/a67e991c-0211-406a-bbcb-3933ffdce18d-kube-api-access-2v7j8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:35:37.115221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:37.115220    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67e991c-0211-406a-bbcb-3933ffdce18d-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:35:37.556010 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:35:37.555974    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:35:37.758810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:37.758720    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc" event={"ID":"a67e991c-0211-406a-bbcb-3933ffdce18d","Type":"ContainerDied","Data":"38ea89cd811526c114239ba532eae5b4f77e24ade0dda0aec4817452540c45e7"}
Oct 30 01:35:37.758810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:37.758737    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"
Oct 30 01:35:37.758810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:37.758771    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38ea89cd811526c114239ba532eae5b4f77e24ade0dda0aec4817452540c45e7"
Oct 30 01:35:46.046403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.046364    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"]
Oct 30 01:35:46.047508 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.047483    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a67e991c-0211-406a-bbcb-3933ffdce18d" containerName="revoke-keys"
Oct 30 01:35:46.047508 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.047506    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67e991c-0211-406a-bbcb-3933ffdce18d" containerName="revoke-keys"
Oct 30 01:35:46.047753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.047738    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67e991c-0211-406a-bbcb-3933ffdce18d" containerName="revoke-keys"
Oct 30 01:35:46.052205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.052183    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"
Oct 30 01:35:46.065492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.065456    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:35:46.074906 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.074884    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:35:46.085172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.081665    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"]
Oct 30 01:35:46.241318 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.241287    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a79586-7165-4134-a157-7cf631b357da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29d2aedf-lx9xt\" (UID: \"46a79586-7165-4134-a157-7cf631b357da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"
Oct 30 01:35:46.241496 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.241357    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a79586-7165-4134-a157-7cf631b357da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29d2aedf-lx9xt\" (UID: \"46a79586-7165-4134-a157-7cf631b357da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"
Oct 30 01:35:46.241496 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.241455    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2hr8\" (UniqueName: \"kubernetes.io/projected/46a79586-7165-4134-a157-7cf631b357da-kube-api-access-f2hr8\") pod \"maas-api-revoke-sub-29d2aedf-lx9xt\" (UID: \"46a79586-7165-4134-a157-7cf631b357da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"
Oct 30 01:35:46.343184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.343065    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a79586-7165-4134-a157-7cf631b357da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29d2aedf-lx9xt\" (UID: \"46a79586-7165-4134-a157-7cf631b357da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"
Oct 30 01:35:46.343384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.343253    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a79586-7165-4134-a157-7cf631b357da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29d2aedf-lx9xt\" (UID: \"46a79586-7165-4134-a157-7cf631b357da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"
Oct 30 01:35:46.343384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.343301    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2hr8\" (UniqueName: \"kubernetes.io/projected/46a79586-7165-4134-a157-7cf631b357da-kube-api-access-f2hr8\") pod \"maas-api-revoke-sub-29d2aedf-lx9xt\" (UID: \"46a79586-7165-4134-a157-7cf631b357da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"
Oct 30 01:35:46.344075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.344048    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a79586-7165-4134-a157-7cf631b357da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29d2aedf-lx9xt\" (UID: \"46a79586-7165-4134-a157-7cf631b357da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"
Oct 30 01:35:46.345693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.345670    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a79586-7165-4134-a157-7cf631b357da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29d2aedf-lx9xt\" (UID: \"46a79586-7165-4134-a157-7cf631b357da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"
Oct 30 01:35:46.365012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.364986    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2hr8\" (UniqueName: \"kubernetes.io/projected/46a79586-7165-4134-a157-7cf631b357da-kube-api-access-f2hr8\") pod \"maas-api-revoke-sub-29d2aedf-lx9xt\" (UID: \"46a79586-7165-4134-a157-7cf631b357da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"
Oct 30 01:35:46.370690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.370669    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"
Oct 30 01:35:46.679314 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:35:46.679279    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a79586_7165_4134_a157_7cf631b357da.slice/crio-e7e8443ec8ce719c02a15c65de5759442b4d88242f3d42207768fe9d2bc39253 WatchSource:0}: Error finding container e7e8443ec8ce719c02a15c65de5759442b4d88242f3d42207768fe9d2bc39253: Status 404 returned error can't find the container with id e7e8443ec8ce719c02a15c65de5759442b4d88242f3d42207768fe9d2bc39253
Oct 30 01:35:46.679991 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.679967    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"]
Oct 30 01:35:46.730677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.730646    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"]
Oct 30 01:35:46.737017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.736998    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:35:46.751760 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.751734    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"agc-cred-75b298c27b49\""
Oct 30 01:35:46.759195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.756235    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8481a734-88f1-489a-87b0-0abee59a7093-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s\" (UID: \"8481a734-88f1-489a-87b0-0abee59a7093\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:35:46.759195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.756322    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pdfg\" (UniqueName: \"kubernetes.io/projected/8481a734-88f1-489a-87b0-0abee59a7093-kube-api-access-9pdfg\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s\" (UID: \"8481a734-88f1-489a-87b0-0abee59a7093\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:35:46.759195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.756566    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8481a734-88f1-489a-87b0-0abee59a7093-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s\" (UID: \"8481a734-88f1-489a-87b0-0abee59a7093\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:35:46.759195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.756611    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8481a734-88f1-489a-87b0-0abee59a7093-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s\" (UID: \"8481a734-88f1-489a-87b0-0abee59a7093\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:35:46.759195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.757360    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"]
Oct 30 01:35:46.761603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.761562    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"routing-overlay\""
Oct 30 01:35:46.761603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.761601    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"openshift-service-ca.crt\""
Oct 30 01:35:46.761776 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.761608    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ae1820\""
Oct 30 01:35:46.761776 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.761602    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"payload-processing-external-model-e2e--53969a13-dockercfg-24pcd\""
Oct 30 01:35:46.761776 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.761693    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ae1820\"/\"kube-root-ca.crt\""
Oct 30 01:35:46.848055 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.848018    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt" event={"ID":"46a79586-7165-4134-a157-7cf631b357da","Type":"ContainerStarted","Data":"e7e8443ec8ce719c02a15c65de5759442b4d88242f3d42207768fe9d2bc39253"}
Oct 30 01:35:46.857870 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.857841    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8481a734-88f1-489a-87b0-0abee59a7093-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s\" (UID: \"8481a734-88f1-489a-87b0-0abee59a7093\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:35:46.858027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.857885    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdfg\" (UniqueName: \"kubernetes.io/projected/8481a734-88f1-489a-87b0-0abee59a7093-kube-api-access-9pdfg\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s\" (UID: \"8481a734-88f1-489a-87b0-0abee59a7093\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:35:46.858146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.858129    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8481a734-88f1-489a-87b0-0abee59a7093-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s\" (UID: \"8481a734-88f1-489a-87b0-0abee59a7093\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:35:46.858247 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.858210    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8481a734-88f1-489a-87b0-0abee59a7093-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s\" (UID: \"8481a734-88f1-489a-87b0-0abee59a7093\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:35:46.858572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.858553    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8481a734-88f1-489a-87b0-0abee59a7093-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s\" (UID: \"8481a734-88f1-489a-87b0-0abee59a7093\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:35:46.858868 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.858847    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8481a734-88f1-489a-87b0-0abee59a7093-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s\" (UID: \"8481a734-88f1-489a-87b0-0abee59a7093\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:35:46.860709 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.860687    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8481a734-88f1-489a-87b0-0abee59a7093-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s\" (UID: \"8481a734-88f1-489a-87b0-0abee59a7093\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:35:46.879058 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:46.879033    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pdfg\" (UniqueName: \"kubernetes.io/projected/8481a734-88f1-489a-87b0-0abee59a7093-kube-api-access-9pdfg\") pod \"payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s\" (UID: \"8481a734-88f1-489a-87b0-0abee59a7093\") " pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:35:47.052345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:47.052303    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:35:47.367692 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:47.367657    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"]
Oct 30 01:35:47.368247 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:35:47.368218    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8481a734_88f1_489a_87b0_0abee59a7093.slice/crio-b5c231917e31d500adb7fa5655ae8a3da73af42e554d02f572826cda5349279f WatchSource:0}: Error finding container b5c231917e31d500adb7fa5655ae8a3da73af42e554d02f572826cda5349279f: Status 404 returned error can't find the container with id b5c231917e31d500adb7fa5655ae8a3da73af42e554d02f572826cda5349279f
Oct 30 01:35:47.857629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:47.857594    4425 generic.go:356] "Generic (PLEG): container finished" podID="46a79586-7165-4134-a157-7cf631b357da" containerID="17d3836300c616eec1354758d16c1a18e12ed8d5e5fcaf8d827cf526cb10c0a9" exitCode=0
Oct 30 01:35:47.857827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:47.857677    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt" event={"ID":"46a79586-7165-4134-a157-7cf631b357da","Type":"ContainerDied","Data":"17d3836300c616eec1354758d16c1a18e12ed8d5e5fcaf8d827cf526cb10c0a9"}
Oct 30 01:35:47.859652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:47.859630    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s" event={"ID":"8481a734-88f1-489a-87b0-0abee59a7093","Type":"ContainerStarted","Data":"bd3b265e283d497d932a71e0cde04f48017574a9f5a98934f4398a744a51da20"}
Oct 30 01:35:47.859777 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:47.859671    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s" event={"ID":"8481a734-88f1-489a-87b0-0abee59a7093","Type":"ContainerStarted","Data":"b5c231917e31d500adb7fa5655ae8a3da73af42e554d02f572826cda5349279f"}
Oct 30 01:35:47.859777 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:47.859716    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:35:47.931681 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:47.931625    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s" podStartSLOduration=1.93160941 podStartE2EDuration="1.93160941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:47.92866297 +0000 UTC m=+3256.678318279" watchObservedRunningTime="2026-10-30 01:35:47.93160941 +0000 UTC m=+3256.681264718"
Oct 30 01:35:49.034105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:49.034081    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"
Oct 30 01:35:49.090083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:49.090048    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46a79586-7165-4134-a157-7cf631b357da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a79586-7165-4134-a157-7cf631b357da-maas-api-xks-ca\") pod \"46a79586-7165-4134-a157-7cf631b357da\" (UID: \"46a79586-7165-4134-a157-7cf631b357da\") "
Oct 30 01:35:49.090328 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:49.090309    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a79586-7165-4134-a157-7cf631b357da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a79586-7165-4134-a157-7cf631b357da-maas-api-openshift-ca\") pod \"46a79586-7165-4134-a157-7cf631b357da\" (UID: \"46a79586-7165-4134-a157-7cf631b357da\") "
Oct 30 01:35:49.090413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:49.090384    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a79586-7165-4134-a157-7cf631b357da-kube-api-access-f2hr8\" (UniqueName: \"kubernetes.io/projected/46a79586-7165-4134-a157-7cf631b357da-kube-api-access-f2hr8\") pod \"46a79586-7165-4134-a157-7cf631b357da\" (UID: \"46a79586-7165-4134-a157-7cf631b357da\") "
Oct 30 01:35:49.090735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:49.090705    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a79586-7165-4134-a157-7cf631b357da-maas-api-openshift-ca" pod "46a79586-7165-4134-a157-7cf631b357da" (UID: "46a79586-7165-4134-a157-7cf631b357da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:35:49.091001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:49.090984    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46a79586-7165-4134-a157-7cf631b357da-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:35:49.092536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:49.092502    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46a79586-7165-4134-a157-7cf631b357da-maas-api-xks-ca" pod "46a79586-7165-4134-a157-7cf631b357da" (UID: "46a79586-7165-4134-a157-7cf631b357da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:35:49.093083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:49.093053    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a79586-7165-4134-a157-7cf631b357da-kube-api-access-f2hr8" pod "46a79586-7165-4134-a157-7cf631b357da" (UID: "46a79586-7165-4134-a157-7cf631b357da"). InnerVolumeSpecName "kube-api-access-f2hr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:35:49.192608 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:49.192525    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2hr8\" (UniqueName: \"kubernetes.io/projected/46a79586-7165-4134-a157-7cf631b357da-kube-api-access-f2hr8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:35:49.192608 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:49.192554    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46a79586-7165-4134-a157-7cf631b357da-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:35:49.876869 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:49.876839    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"
Oct 30 01:35:49.877069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:49.876874    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt" event={"ID":"46a79586-7165-4134-a157-7cf631b357da","Type":"ContainerDied","Data":"e7e8443ec8ce719c02a15c65de5759442b4d88242f3d42207768fe9d2bc39253"}
Oct 30 01:35:49.877069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:49.876919    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7e8443ec8ce719c02a15c65de5759442b4d88242f3d42207768fe9d2bc39253"
Oct 30 01:35:56.777732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:56.777690    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"]
Oct 30 01:35:56.778786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:56.778760    4425 cpu_manager.go:414] "containerMap: removing container" podUID="46a79586-7165-4134-a157-7cf631b357da" containerName="revoke-keys"
Oct 30 01:35:56.778786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:56.778786    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a79586-7165-4134-a157-7cf631b357da" containerName="revoke-keys"
Oct 30 01:35:56.778987 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:56.778975    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a79586-7165-4134-a157-7cf631b357da" containerName="revoke-keys"
Oct 30 01:35:56.784071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:56.784045    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"
Oct 30 01:35:56.803438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:56.803405    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"]
Oct 30 01:35:56.808072 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:56.808045    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:35:56.808243 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:56.808121    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:35:56.904735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:56.904697    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196a8ae2-eb90-45d3-8da7-3be8661f6951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7237ce6-hjwdb\" (UID: \"196a8ae2-eb90-45d3-8da7-3be8661f6951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"
Oct 30 01:35:56.904735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:56.904741    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69fl8\" (UniqueName: \"kubernetes.io/projected/196a8ae2-eb90-45d3-8da7-3be8661f6951-kube-api-access-69fl8\") pod \"maas-api-revoke-sub-d7237ce6-hjwdb\" (UID: \"196a8ae2-eb90-45d3-8da7-3be8661f6951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"
Oct 30 01:35:56.905054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:56.905029    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196a8ae2-eb90-45d3-8da7-3be8661f6951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7237ce6-hjwdb\" (UID: \"196a8ae2-eb90-45d3-8da7-3be8661f6951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"
Oct 30 01:35:57.006525 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:57.006487    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196a8ae2-eb90-45d3-8da7-3be8661f6951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7237ce6-hjwdb\" (UID: \"196a8ae2-eb90-45d3-8da7-3be8661f6951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"
Oct 30 01:35:57.006709 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:57.006558    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196a8ae2-eb90-45d3-8da7-3be8661f6951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7237ce6-hjwdb\" (UID: \"196a8ae2-eb90-45d3-8da7-3be8661f6951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"
Oct 30 01:35:57.006709 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:57.006583    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69fl8\" (UniqueName: \"kubernetes.io/projected/196a8ae2-eb90-45d3-8da7-3be8661f6951-kube-api-access-69fl8\") pod \"maas-api-revoke-sub-d7237ce6-hjwdb\" (UID: \"196a8ae2-eb90-45d3-8da7-3be8661f6951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"
Oct 30 01:35:57.007219 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:57.007195    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196a8ae2-eb90-45d3-8da7-3be8661f6951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7237ce6-hjwdb\" (UID: \"196a8ae2-eb90-45d3-8da7-3be8661f6951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"
Oct 30 01:35:57.008966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:57.008938    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196a8ae2-eb90-45d3-8da7-3be8661f6951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7237ce6-hjwdb\" (UID: \"196a8ae2-eb90-45d3-8da7-3be8661f6951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"
Oct 30 01:35:57.034630 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:57.034554    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69fl8\" (UniqueName: \"kubernetes.io/projected/196a8ae2-eb90-45d3-8da7-3be8661f6951-kube-api-access-69fl8\") pod \"maas-api-revoke-sub-d7237ce6-hjwdb\" (UID: \"196a8ae2-eb90-45d3-8da7-3be8661f6951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"
Oct 30 01:35:57.108107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:57.108071    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"
Oct 30 01:35:57.421840 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:35:57.421812    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196a8ae2_eb90_45d3_8da7_3be8661f6951.slice/crio-eb01b386973ff12e7b9e974946b4a8443cd6adcddc105f5a2827409793233cd8 WatchSource:0}: Error finding container eb01b386973ff12e7b9e974946b4a8443cd6adcddc105f5a2827409793233cd8: Status 404 returned error can't find the container with id eb01b386973ff12e7b9e974946b4a8443cd6adcddc105f5a2827409793233cd8
Oct 30 01:35:57.428955 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:57.428927    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"]
Oct 30 01:35:57.957228 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:57.957186    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb" event={"ID":"196a8ae2-eb90-45d3-8da7-3be8661f6951","Type":"ContainerStarted","Data":"eb01b386973ff12e7b9e974946b4a8443cd6adcddc105f5a2827409793233cd8"}
Oct 30 01:35:58.871279 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:58.871248    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:35:58.970710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:58.970667    4425 generic.go:356] "Generic (PLEG): container finished" podID="196a8ae2-eb90-45d3-8da7-3be8661f6951" containerID="c5ecb05b9252eaf3bff7f626816afa030b3c17157fb5e198d07d7978c3ea1dd8" exitCode=0
Oct 30 01:35:58.970710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:35:58.970710    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb" event={"ID":"196a8ae2-eb90-45d3-8da7-3be8661f6951","Type":"ContainerDied","Data":"c5ecb05b9252eaf3bff7f626816afa030b3c17157fb5e198d07d7978c3ea1dd8"}
Oct 30 01:36:00.167208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:00.167176    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"
Oct 30 01:36:00.253349 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:00.253249    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/196a8ae2-eb90-45d3-8da7-3be8661f6951-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196a8ae2-eb90-45d3-8da7-3be8661f6951-maas-api-xks-ca\") pod \"196a8ae2-eb90-45d3-8da7-3be8661f6951\" (UID: \"196a8ae2-eb90-45d3-8da7-3be8661f6951\") "
Oct 30 01:36:00.253526 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:00.253417    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/196a8ae2-eb90-45d3-8da7-3be8661f6951-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196a8ae2-eb90-45d3-8da7-3be8661f6951-maas-api-openshift-ca\") pod \"196a8ae2-eb90-45d3-8da7-3be8661f6951\" (UID: \"196a8ae2-eb90-45d3-8da7-3be8661f6951\") "
Oct 30 01:36:00.253526 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:00.253455    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/196a8ae2-eb90-45d3-8da7-3be8661f6951-kube-api-access-69fl8\" (UniqueName: \"kubernetes.io/projected/196a8ae2-eb90-45d3-8da7-3be8661f6951-kube-api-access-69fl8\") pod \"196a8ae2-eb90-45d3-8da7-3be8661f6951\" (UID: \"196a8ae2-eb90-45d3-8da7-3be8661f6951\") "
Oct 30 01:36:00.254645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:00.254612    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196a8ae2-eb90-45d3-8da7-3be8661f6951-maas-api-openshift-ca" pod "196a8ae2-eb90-45d3-8da7-3be8661f6951" (UID: "196a8ae2-eb90-45d3-8da7-3be8661f6951"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:00.255849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:00.255826    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/196a8ae2-eb90-45d3-8da7-3be8661f6951-maas-api-xks-ca" pod "196a8ae2-eb90-45d3-8da7-3be8661f6951" (UID: "196a8ae2-eb90-45d3-8da7-3be8661f6951"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:36:00.256104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:00.256087    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196a8ae2-eb90-45d3-8da7-3be8661f6951-kube-api-access-69fl8" pod "196a8ae2-eb90-45d3-8da7-3be8661f6951" (UID: "196a8ae2-eb90-45d3-8da7-3be8661f6951"). InnerVolumeSpecName "kube-api-access-69fl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:00.355917 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:00.355882    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/196a8ae2-eb90-45d3-8da7-3be8661f6951-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:36:00.355917 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:00.355914    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69fl8\" (UniqueName: \"kubernetes.io/projected/196a8ae2-eb90-45d3-8da7-3be8661f6951-kube-api-access-69fl8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:36:00.355917 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:00.355924    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/196a8ae2-eb90-45d3-8da7-3be8661f6951-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:36:01.003850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:01.003820    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"
Oct 30 01:36:01.004073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:01.003866    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb" event={"ID":"196a8ae2-eb90-45d3-8da7-3be8661f6951","Type":"ContainerDied","Data":"eb01b386973ff12e7b9e974946b4a8443cd6adcddc105f5a2827409793233cd8"}
Oct 30 01:36:01.004073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:01.003967    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb01b386973ff12e7b9e974946b4a8443cd6adcddc105f5a2827409793233cd8"
Oct 30 01:36:07.557236 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:36:07.557187    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:36:10.601424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:10.601381    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"]
Oct 30 01:36:13.122082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.122039    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"]
Oct 30 01:36:13.123444 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.123418    4425 cpu_manager.go:414] "containerMap: removing container" podUID="196a8ae2-eb90-45d3-8da7-3be8661f6951" containerName="revoke-keys"
Oct 30 01:36:13.123560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.123446    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="196a8ae2-eb90-45d3-8da7-3be8661f6951" containerName="revoke-keys"
Oct 30 01:36:13.123740 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.123721    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="196a8ae2-eb90-45d3-8da7-3be8661f6951" containerName="revoke-keys"
Oct 30 01:36:13.128778 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.128745    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.149114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.149072    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"]
Oct 30 01:36:13.151357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.151324    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bd287f\"/\"kube-root-ca.crt\""
Oct 30 01:36:13.151357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.151333    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bd287f\"/\"openshift-service-ca.crt\""
Oct 30 01:36:13.154476 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.154448    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bd287f\"/\"default-dockercfg-hgm69\""
Oct 30 01:36:13.154476 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.154448    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bd287f\"/\"test-model-bd287f-kserve-self-signed-certs\""
Oct 30 01:36:13.248626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.248581    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfmtx\" (UniqueName: \"kubernetes.io/projected/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-kube-api-access-sfmtx\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.248845 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.248637    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-tmp-dir\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.248845 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.248662    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-home\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.248845 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.248767    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-dshm\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.249010 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.248991    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-tls-certs\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.249071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.249032    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-model-cache\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.351179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.350521    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-dshm\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.351179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.350769    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-tls-certs\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.351179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.350822    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-model-cache\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.351179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.350923    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfmtx\" (UniqueName: \"kubernetes.io/projected/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-kube-api-access-sfmtx\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.351179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.350977    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-tmp-dir\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.351179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.351008    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-home\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.351677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.351496    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-home\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.351813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.351766    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-model-cache\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.351943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.351821    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-tmp-dir\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.353878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.353851    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-dshm\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.357321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.357292    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-tls-certs\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.373233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.373136    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfmtx\" (UniqueName: \"kubernetes.io/projected/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-kube-api-access-sfmtx\") pod \"test-model-bd287f-kserve-7ffc54564f-wzgrq\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") " pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.455365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.455321    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:13.805063 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:13.805026    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"]
Oct 30 01:36:13.805708 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:36:13.805660    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c9778fb_1954_4c6a_a3d9_fc1cb633045b.slice/crio-8e6bc799050ed54557b91c0a4cac9343a0a2251b43bc1e07ff934601028acd89 WatchSource:0}: Error finding container 8e6bc799050ed54557b91c0a4cac9343a0a2251b43bc1e07ff934601028acd89: Status 404 returned error can't find the container with id 8e6bc799050ed54557b91c0a4cac9343a0a2251b43bc1e07ff934601028acd89
Oct 30 01:36:14.132527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:14.132417    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq" event={"ID":"9c9778fb-1954-4c6a-a3d9-fc1cb633045b","Type":"ContainerStarted","Data":"a38309589aa3bb6d40125d0939154db52678962e336647e0bdf129144d220910"}
Oct 30 01:36:14.132527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:14.132472    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq" event={"ID":"9c9778fb-1954-4c6a-a3d9-fc1cb633045b","Type":"ContainerStarted","Data":"8e6bc799050ed54557b91c0a4cac9343a0a2251b43bc1e07ff934601028acd89"}
Oct 30 01:36:14.173117 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:14.173059    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq" podStartSLOduration=1.173041372 podStartE2EDuration="1.173041372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:14.168197631 +0000 UTC m=+3282.917852934" watchObservedRunningTime="2026-10-30 01:36:14.173041372 +0000 UTC m=+3282.922696763"
Oct 30 01:36:14.455852 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:14.455743    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:15.602124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:15.602064    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s" podUID="8481a734-88f1-489a-87b0-0abee59a7093" containerName="payload-processing" containerID="cri-o://bd3b265e283d497d932a71e0cde04f48017574a9f5a98934f4398a744a51da20" gracePeriod=25
Oct 30 01:36:15.926289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:15.926258    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:36:15.997029 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:15.996990    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8481a734-88f1-489a-87b0-0abee59a7093-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8481a734-88f1-489a-87b0-0abee59a7093-plugins-config-volume\") pod \"8481a734-88f1-489a-87b0-0abee59a7093\" (UID: \"8481a734-88f1-489a-87b0-0abee59a7093\") "
Oct 30 01:36:15.997311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:15.997092    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8481a734-88f1-489a-87b0-0abee59a7093-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8481a734-88f1-489a-87b0-0abee59a7093-provider-credentials\") pod \"8481a734-88f1-489a-87b0-0abee59a7093\" (UID: \"8481a734-88f1-489a-87b0-0abee59a7093\") "
Oct 30 01:36:15.997311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:15.997249    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8481a734-88f1-489a-87b0-0abee59a7093-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8481a734-88f1-489a-87b0-0abee59a7093-routing-overlay-volume\") pod \"8481a734-88f1-489a-87b0-0abee59a7093\" (UID: \"8481a734-88f1-489a-87b0-0abee59a7093\") "
Oct 30 01:36:15.997457 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:15.997412    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8481a734-88f1-489a-87b0-0abee59a7093-kube-api-access-9pdfg\" (UniqueName: \"kubernetes.io/projected/8481a734-88f1-489a-87b0-0abee59a7093-kube-api-access-9pdfg\") pod \"8481a734-88f1-489a-87b0-0abee59a7093\" (UID: \"8481a734-88f1-489a-87b0-0abee59a7093\") "
Oct 30 01:36:15.997617 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:15.997589    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8481a734-88f1-489a-87b0-0abee59a7093-plugins-config-volume" pod "8481a734-88f1-489a-87b0-0abee59a7093" (UID: "8481a734-88f1-489a-87b0-0abee59a7093"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:15.997728 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:15.997695    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8481a734-88f1-489a-87b0-0abee59a7093-routing-overlay-volume" pod "8481a734-88f1-489a-87b0-0abee59a7093" (UID: "8481a734-88f1-489a-87b0-0abee59a7093"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:15.998188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:15.998141    4425 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8481a734-88f1-489a-87b0-0abee59a7093-routing-overlay-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:36:15.998188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:15.998181    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8481a734-88f1-489a-87b0-0abee59a7093-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:36:16.000076 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:16.000047    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8481a734-88f1-489a-87b0-0abee59a7093-provider-credentials" pod "8481a734-88f1-489a-87b0-0abee59a7093" (UID: "8481a734-88f1-489a-87b0-0abee59a7093"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:16.000322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:16.000291    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8481a734-88f1-489a-87b0-0abee59a7093-kube-api-access-9pdfg" pod "8481a734-88f1-489a-87b0-0abee59a7093" (UID: "8481a734-88f1-489a-87b0-0abee59a7093"). InnerVolumeSpecName "kube-api-access-9pdfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:16.099843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:16.099795    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pdfg\" (UniqueName: \"kubernetes.io/projected/8481a734-88f1-489a-87b0-0abee59a7093-kube-api-access-9pdfg\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:36:16.099843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:16.099833    4425 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8481a734-88f1-489a-87b0-0abee59a7093-provider-credentials\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:36:16.150612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:16.150527    4425 generic.go:356] "Generic (PLEG): container finished" podID="8481a734-88f1-489a-87b0-0abee59a7093" containerID="bd3b265e283d497d932a71e0cde04f48017574a9f5a98934f4398a744a51da20" exitCode=0
Oct 30 01:36:16.150612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:16.150574    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"
Oct 30 01:36:16.150915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:16.150610    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s" event={"ID":"8481a734-88f1-489a-87b0-0abee59a7093","Type":"ContainerDied","Data":"bd3b265e283d497d932a71e0cde04f48017574a9f5a98934f4398a744a51da20"}
Oct 30 01:36:16.150915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:16.150671    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s" event={"ID":"8481a734-88f1-489a-87b0-0abee59a7093","Type":"ContainerDied","Data":"b5c231917e31d500adb7fa5655ae8a3da73af42e554d02f572826cda5349279f"}
Oct 30 01:36:16.150915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:16.150696    4425 scope.go:122] "RemoveContainer" containerID="bd3b265e283d497d932a71e0cde04f48017574a9f5a98934f4398a744a51da20"
Oct 30 01:36:16.164018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:16.163999    4425 scope.go:122] "RemoveContainer" containerID="bd3b265e283d497d932a71e0cde04f48017574a9f5a98934f4398a744a51da20"
Oct 30 01:36:16.164409 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:36:16.164388    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd3b265e283d497d932a71e0cde04f48017574a9f5a98934f4398a744a51da20\": container with ID starting with bd3b265e283d497d932a71e0cde04f48017574a9f5a98934f4398a744a51da20 not found: ID does not exist" containerID="bd3b265e283d497d932a71e0cde04f48017574a9f5a98934f4398a744a51da20"
Oct 30 01:36:16.164479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:16.164421    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd3b265e283d497d932a71e0cde04f48017574a9f5a98934f4398a744a51da20"} err="failed to get container status \"bd3b265e283d497d932a71e0cde04f48017574a9f5a98934f4398a744a51da20\": rpc error: code = NotFound desc = could not find container \"bd3b265e283d497d932a71e0cde04f48017574a9f5a98934f4398a744a51da20\": container with ID starting with bd3b265e283d497d932a71e0cde04f48017574a9f5a98934f4398a744a51da20 not found: ID does not exist"
Oct 30 01:36:16.198282 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:16.198242    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"]
Oct 30 01:36:16.205979 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:16.205939    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ae1820/payload-processing-external-model-e2e-extmod-b-ae1820-5fdctkr9s"]
Oct 30 01:36:17.431204 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:17.427562    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8481a734-88f1-489a-87b0-0abee59a7093" path="/var/lib/kubelet/pods/8481a734-88f1-489a-87b0-0abee59a7093/volumes"
Oct 30 01:36:21.037104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:21.037051    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"]
Oct 30 01:36:21.045823 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:21.045793    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed55ee2-j6ddp"]
Oct 30 01:36:21.425801 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:21.425721    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01b43a26-79d4-4136-8735-bd75b5897c07" path="/var/lib/kubelet/pods/01b43a26-79d4-4136-8735-bd75b5897c07/volumes"
Oct 30 01:36:23.455938 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:23.455902    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:23.468741 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:23.468710    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:24.253163 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:24.253107    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:36:25.645367 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.645331    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"]
Oct 30 01:36:25.646459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.646432    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8481a734-88f1-489a-87b0-0abee59a7093" containerName="payload-processing"
Oct 30 01:36:25.646459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.646459    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8481a734-88f1-489a-87b0-0abee59a7093" containerName="payload-processing"
Oct 30 01:36:25.646720 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.646704    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8481a734-88f1-489a-87b0-0abee59a7093" containerName="payload-processing"
Oct 30 01:36:25.652723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.652699    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.675311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.675270    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"]
Oct 30 01:36:25.676366 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.676263    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-190407\"/\"test-model-190407-kserve-self-signed-certs\""
Oct 30 01:36:25.676505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.676420    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-190407\"/\"openshift-service-ca.crt\""
Oct 30 01:36:25.685483 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.685450    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-190407\"/\"default-dockercfg-dfbw7\""
Oct 30 01:36:25.685661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.685627    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-190407\"/\"kube-root-ca.crt\""
Oct 30 01:36:25.749242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.749201    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-home\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.749488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.749276    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-model-cache\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.749488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.749355    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-dshm\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.749488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.749393    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jsxf\" (UniqueName: \"kubernetes.io/projected/5f308cb4-16bb-4c79-89da-3ad7f2633de8-kube-api-access-9jsxf\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.749689 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.749552    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-tmp-dir\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.749803 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.749784    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f308cb4-16bb-4c79-89da-3ad7f2633de8-tls-certs\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.851752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.851703    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-home\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.851945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.851870    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-model-cache\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.851945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.851911    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-dshm\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.852052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.851956    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jsxf\" (UniqueName: \"kubernetes.io/projected/5f308cb4-16bb-4c79-89da-3ad7f2633de8-kube-api-access-9jsxf\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.852131 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.852049    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-tmp-dir\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.852289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.852268    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f308cb4-16bb-4c79-89da-3ad7f2633de8-tls-certs\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.852579 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.852521    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-tmp-dir\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.852579 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.852298    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-model-cache\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.852579 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.852268    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-home\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.854843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.854814    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-dshm\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.854976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.854951    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f308cb4-16bb-4c79-89da-3ad7f2633de8-tls-certs\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.873947 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.873915    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jsxf\" (UniqueName: \"kubernetes.io/projected/5f308cb4-16bb-4c79-89da-3ad7f2633de8-kube-api-access-9jsxf\") pod \"test-model-190407-kserve-b9766996f-9tbcl\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") " pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:25.991231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:25.990444    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:26.350324 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:26.350291    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"]
Oct 30 01:36:26.353492 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:36:26.353461    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f308cb4_16bb_4c79_89da_3ad7f2633de8.slice/crio-eb06837a9e35d6852321eed07719dca172c1586cd73c8e83a289e19514174a6d WatchSource:0}: Error finding container eb06837a9e35d6852321eed07719dca172c1586cd73c8e83a289e19514174a6d: Status 404 returned error can't find the container with id eb06837a9e35d6852321eed07719dca172c1586cd73c8e83a289e19514174a6d
Oct 30 01:36:27.272778 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:27.272738    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl" event={"ID":"5f308cb4-16bb-4c79-89da-3ad7f2633de8","Type":"ContainerStarted","Data":"4842c9916e6250168fdfd81bd269328e55cee8f869c82a2647a42184dfadc93d"}
Oct 30 01:36:27.272778 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:27.272789    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl" event={"ID":"5f308cb4-16bb-4c79-89da-3ad7f2633de8","Type":"ContainerStarted","Data":"eb06837a9e35d6852321eed07719dca172c1586cd73c8e83a289e19514174a6d"}
Oct 30 01:36:27.314034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:27.313981    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl" podStartSLOduration=2.313965752 podStartE2EDuration="2.313965752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:27.309202655 +0000 UTC m=+3296.058857958" watchObservedRunningTime="2026-10-30 01:36:27.313965752 +0000 UTC m=+3296.063621061"
Oct 30 01:36:27.992191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:27.991147    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:32.030255 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:32.030203    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"]
Oct 30 01:36:32.039417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:32.039385    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-959aec9b-flzql"]
Oct 30 01:36:33.427017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:33.426981    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98dbf13a-0e47-45ab-869f-8d6122e5a7e7" path="/var/lib/kubelet/pods/98dbf13a-0e47-45ab-869f-8d6122e5a7e7/volumes"
Oct 30 01:36:35.533599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:35.533565    4425 scope.go:122] "RemoveContainer" containerID="88f0fc6c65a5a2c2b4de4d8a979f7f47e0a85d0e42a8fd3d30ad1ee4177304cf"
Oct 30 01:36:35.547975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:35.547951    4425 scope.go:122] "RemoveContainer" containerID="31598938d29994dfdcc48011a9d878ed46a1a8a53bd028150ff272fd6f27f712"
Oct 30 01:36:35.877859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:35.877749    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"]
Oct 30 01:36:35.884276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:35.884247    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"
Oct 30 01:36:35.904840 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:35.904812    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"]
Oct 30 01:36:35.909404 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:35.909376    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5457f2\""
Oct 30 01:36:35.909569 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:35.909488    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5457f2-dockercfg-vnwcc\""
Oct 30 01:36:35.990951 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:35.990912    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:36.009869 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:36.009836    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:36.043881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:36.040266    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq97j\" (UniqueName: \"kubernetes.io/projected/2efb0b19-a05d-46bd-bf1d-622e6410cafe-kube-api-access-zq97j\") pod \"payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9\" (UID: \"2efb0b19-a05d-46bd-bf1d-622e6410cafe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"
Oct 30 01:36:36.044513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:36.044479    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2efb0b19-a05d-46bd-bf1d-622e6410cafe-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9\" (UID: \"2efb0b19-a05d-46bd-bf1d-622e6410cafe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"
Oct 30 01:36:36.146004 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:36.145908    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq97j\" (UniqueName: \"kubernetes.io/projected/2efb0b19-a05d-46bd-bf1d-622e6410cafe-kube-api-access-zq97j\") pod \"payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9\" (UID: \"2efb0b19-a05d-46bd-bf1d-622e6410cafe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"
Oct 30 01:36:36.146248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:36.146084    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2efb0b19-a05d-46bd-bf1d-622e6410cafe-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9\" (UID: \"2efb0b19-a05d-46bd-bf1d-622e6410cafe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"
Oct 30 01:36:36.146864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:36.146842    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2efb0b19-a05d-46bd-bf1d-622e6410cafe-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9\" (UID: \"2efb0b19-a05d-46bd-bf1d-622e6410cafe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"
Oct 30 01:36:36.167488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:36.167459    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq97j\" (UniqueName: \"kubernetes.io/projected/2efb0b19-a05d-46bd-bf1d-622e6410cafe-kube-api-access-zq97j\") pod \"payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9\" (UID: \"2efb0b19-a05d-46bd-bf1d-622e6410cafe\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"
Oct 30 01:36:36.200387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:36.200361    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"
Oct 30 01:36:36.377599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:36.377560    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:36:36.514085 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:36.514055    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"]
Oct 30 01:36:36.514746 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:36:36.514722    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2efb0b19_a05d_46bd_bf1d_622e6410cafe.slice/crio-eb41ca5afb5a6f643f84fa0b9478f897b2305c19696fc87ebe682d508a55abe8 WatchSource:0}: Error finding container eb41ca5afb5a6f643f84fa0b9478f897b2305c19696fc87ebe682d508a55abe8: Status 404 returned error can't find the container with id eb41ca5afb5a6f643f84fa0b9478f897b2305c19696fc87ebe682d508a55abe8
Oct 30 01:36:37.379631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:37.379592    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9" event={"ID":"2efb0b19-a05d-46bd-bf1d-622e6410cafe","Type":"ContainerStarted","Data":"95e685605b3853240b34d92e8629923d777c4ca5f1f344e92b022d2217e79309"}
Oct 30 01:36:37.380087 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:37.379660    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9" event={"ID":"2efb0b19-a05d-46bd-bf1d-622e6410cafe","Type":"ContainerStarted","Data":"eb41ca5afb5a6f643f84fa0b9478f897b2305c19696fc87ebe682d508a55abe8"}
Oct 30 01:36:37.380087 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:37.379708    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"
Oct 30 01:36:37.418063 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:37.418010    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9" podStartSLOduration=2.417995946 podStartE2EDuration="2.417995946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:37.415150445 +0000 UTC m=+3306.164805753" watchObservedRunningTime="2026-10-30 01:36:37.417995946 +0000 UTC m=+3306.167651254"
Oct 30 01:36:37.562017 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:36:37.561975    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:36:46.827675 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:46.827635    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:36:48.391953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:48.391922    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"
Oct 30 01:36:57.030608 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:57.030553    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"]
Oct 30 01:36:57.040003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:57.039972    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4b26e33-q2zfr"]
Oct 30 01:36:57.425628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:36:57.425550    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e278fe-f9a9-4a71-b02d-4ef30ed8c049" path="/var/lib/kubelet/pods/88e278fe-f9a9-4a71-b02d-4ef30ed8c049/volumes"
Oct 30 01:37:07.558002 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:37:07.557960    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:37:18.289689 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.289649    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"]
Oct 30 01:37:18.295011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.294983    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:37:18.316149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.316117    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6aa9aa-serving-cert\""
Oct 30 01:37:18.324114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.324075    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"]
Oct 30 01:37:18.383819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.383782    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24025a00-4226-4ea7-9c91-7d5805cec35c-maas-api-tls\") pod \"maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9\" (UID: \"24025a00-4226-4ea7-9c91-7d5805cec35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:37:18.384013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.383827    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24025a00-4226-4ea7-9c91-7d5805cec35c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9\" (UID: \"24025a00-4226-4ea7-9c91-7d5805cec35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:37:18.384013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.383917    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24025a00-4226-4ea7-9c91-7d5805cec35c-metrics-certs\") pod \"maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9\" (UID: \"24025a00-4226-4ea7-9c91-7d5805cec35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:37:18.384013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.383954    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdlwd\" (UniqueName: \"kubernetes.io/projected/24025a00-4226-4ea7-9c91-7d5805cec35c-kube-api-access-xdlwd\") pod \"maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9\" (UID: \"24025a00-4226-4ea7-9c91-7d5805cec35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:37:18.485800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.485759    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24025a00-4226-4ea7-9c91-7d5805cec35c-maas-api-tls\") pod \"maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9\" (UID: \"24025a00-4226-4ea7-9c91-7d5805cec35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:37:18.486031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.485809    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24025a00-4226-4ea7-9c91-7d5805cec35c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9\" (UID: \"24025a00-4226-4ea7-9c91-7d5805cec35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:37:18.486031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.485861    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24025a00-4226-4ea7-9c91-7d5805cec35c-metrics-certs\") pod \"maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9\" (UID: \"24025a00-4226-4ea7-9c91-7d5805cec35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:37:18.486031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.485899    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdlwd\" (UniqueName: \"kubernetes.io/projected/24025a00-4226-4ea7-9c91-7d5805cec35c-kube-api-access-xdlwd\") pod \"maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9\" (UID: \"24025a00-4226-4ea7-9c91-7d5805cec35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:37:18.486733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.486700    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24025a00-4226-4ea7-9c91-7d5805cec35c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9\" (UID: \"24025a00-4226-4ea7-9c91-7d5805cec35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:37:18.488665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.488638    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24025a00-4226-4ea7-9c91-7d5805cec35c-maas-api-tls\") pod \"maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9\" (UID: \"24025a00-4226-4ea7-9c91-7d5805cec35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:37:18.488772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.488645    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24025a00-4226-4ea7-9c91-7d5805cec35c-metrics-certs\") pod \"maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9\" (UID: \"24025a00-4226-4ea7-9c91-7d5805cec35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:37:18.507596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.507564    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdlwd\" (UniqueName: \"kubernetes.io/projected/24025a00-4226-4ea7-9c91-7d5805cec35c-kube-api-access-xdlwd\") pod \"maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9\" (UID: \"24025a00-4226-4ea7-9c91-7d5805cec35c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:37:18.611242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.611056    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:37:18.923666 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:37:18.923634    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24025a00_4226_4ea7_9c91_7d5805cec35c.slice/crio-05f8b28c36209273225e804c2443e65f42f06b7c437800b23f0e8f4d97010b36 WatchSource:0}: Error finding container 05f8b28c36209273225e804c2443e65f42f06b7c437800b23f0e8f4d97010b36: Status 404 returned error can't find the container with id 05f8b28c36209273225e804c2443e65f42f06b7c437800b23f0e8f4d97010b36
Oct 30 01:37:18.923999 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:18.923973    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"]
Oct 30 01:37:19.750292 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:19.750184    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9" event={"ID":"24025a00-4226-4ea7-9c91-7d5805cec35c","Type":"ContainerStarted","Data":"ce8d34b7a9eb4d45f17e8de1cbcfff8bcc9169408eb58d52b08465ffe58ff600"}
Oct 30 01:37:19.750292 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:19.750253    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9" event={"ID":"24025a00-4226-4ea7-9c91-7d5805cec35c","Type":"ContainerStarted","Data":"05f8b28c36209273225e804c2443e65f42f06b7c437800b23f0e8f4d97010b36"}
Oct 30 01:37:19.750776 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:19.750475    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:37:19.790024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:19.789968    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9" podStartSLOduration=1.385228503 podStartE2EDuration="1.789952179s" totalImagesPullingTime="404.723676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:19.78922729 +0000 UTC m=+3348.538882599" watchObservedRunningTime="2026-10-30 01:37:19.789952179 +0000 UTC m=+3348.539607488"
Oct 30 01:37:25.765209 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:25.765147    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:37:30.676688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:30.676651    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"]
Oct 30 01:37:30.682247 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:30.682223    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:37:30.699552 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:30.699527    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-73423b-serving-cert\""
Oct 30 01:37:30.715763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:30.715728    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"]
Oct 30 01:37:30.765507 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:30.765468    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v\" (UID: \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:37:30.765720 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:30.765611    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdbhl\" (UniqueName: \"kubernetes.io/projected/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-kube-api-access-kdbhl\") pod \"maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v\" (UID: \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:37:30.765720 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:30.765694    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-metrics-certs\") pod \"maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v\" (UID: \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:37:30.765848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:30.765835    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-maas-api-tls\") pod \"maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v\" (UID: \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:37:30.867715 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:30.867673    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v\" (UID: \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:37:30.867935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:30.867886    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbhl\" (UniqueName: \"kubernetes.io/projected/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-kube-api-access-kdbhl\") pod \"maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v\" (UID: \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:37:30.867935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:30.867926    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-metrics-certs\") pod \"maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v\" (UID: \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:37:30.868105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:30.868087    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-maas-api-tls\") pod \"maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v\" (UID: \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:37:30.868522 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:37:30.868493    4425 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-serving-cert: secret "maas-api-e2e-inf-b-73423b-serving-cert" not found
Oct 30 01:37:30.869200 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:37:30.868566    4425 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-maas-api-tls podName:1e2b9d25-c7ed-451c-8b66-ec294e97b3e2 nodeName:}" failed. No retries permitted until 2026-10-30 01:37:31.36854504 +0000 UTC m=+3360.118200345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-maas-api-tls") pod "maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v" (UID: "1e2b9d25-c7ed-451c-8b66-ec294e97b3e2") : secret "maas-api-e2e-inf-b-73423b-serving-cert" not found
Oct 30 01:37:30.869565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:30.869535    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v\" (UID: \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:37:30.872113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:30.872088    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-metrics-certs\") pod \"maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v\" (UID: \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:37:30.891024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:30.890982    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdbhl\" (UniqueName: \"kubernetes.io/projected/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-kube-api-access-kdbhl\") pod \"maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v\" (UID: \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:37:31.376693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:31.376660    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-maas-api-tls\") pod \"maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v\" (UID: \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:37:31.379318 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:31.379290    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-maas-api-tls\") pod \"maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v\" (UID: \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:37:31.598236 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:31.598197    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:37:31.918483 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:37:31.918443    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2b9d25_c7ed_451c_8b66_ec294e97b3e2.slice/crio-6aeec947b78cd74b3fbfa3460eed816bd3cac114734e4470c8207b23886baa7a WatchSource:0}: Error finding container 6aeec947b78cd74b3fbfa3460eed816bd3cac114734e4470c8207b23886baa7a: Status 404 returned error can't find the container with id 6aeec947b78cd74b3fbfa3460eed816bd3cac114734e4470c8207b23886baa7a
Oct 30 01:37:31.918820 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:31.918746    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"]
Oct 30 01:37:32.863250 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:32.863197    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v" event={"ID":"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2","Type":"ContainerStarted","Data":"46ae6bc5a5c19f2d8e3a86f84a8a32c1b7af4b20522bc29b51937c4a3e47431c"}
Oct 30 01:37:32.863447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:32.863294    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v" event={"ID":"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2","Type":"ContainerStarted","Data":"6aeec947b78cd74b3fbfa3460eed816bd3cac114734e4470c8207b23886baa7a"}
Oct 30 01:37:32.863447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:32.863322    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:37:32.905921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:32.905860    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v" podStartSLOduration=2.52102647 podStartE2EDuration="2.905845381s" totalImagesPullingTime="384.818911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:32.90176308 +0000 UTC m=+3361.651418392" watchObservedRunningTime="2026-10-30 01:37:32.905845381 +0000 UTC m=+3361.655500690"
Oct 30 01:37:34.260043 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.259996    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"]
Oct 30 01:37:34.265567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.265540    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:37:34.280593 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.280563    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5457f2-serving-cert\""
Oct 30 01:37:34.298672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.298627    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"]
Oct 30 01:37:34.424790 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.424750    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45aca910-a167-41a3-8792-43502dd22283-maas-api-tls\") pod \"maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b\" (UID: \"45aca910-a167-41a3-8792-43502dd22283\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:37:34.425013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.424889    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxl66\" (UniqueName: \"kubernetes.io/projected/45aca910-a167-41a3-8792-43502dd22283-kube-api-access-gxl66\") pod \"maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b\" (UID: \"45aca910-a167-41a3-8792-43502dd22283\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:37:34.425083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.425037    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45aca910-a167-41a3-8792-43502dd22283-metrics-certs\") pod \"maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b\" (UID: \"45aca910-a167-41a3-8792-43502dd22283\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:37:34.425138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.425112    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45aca910-a167-41a3-8792-43502dd22283-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b\" (UID: \"45aca910-a167-41a3-8792-43502dd22283\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:37:34.527132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.527026    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45aca910-a167-41a3-8792-43502dd22283-metrics-certs\") pod \"maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b\" (UID: \"45aca910-a167-41a3-8792-43502dd22283\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:37:34.527132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.527116    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45aca910-a167-41a3-8792-43502dd22283-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b\" (UID: \"45aca910-a167-41a3-8792-43502dd22283\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:37:34.527373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.527284    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45aca910-a167-41a3-8792-43502dd22283-maas-api-tls\") pod \"maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b\" (UID: \"45aca910-a167-41a3-8792-43502dd22283\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:37:34.527373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.527347    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxl66\" (UniqueName: \"kubernetes.io/projected/45aca910-a167-41a3-8792-43502dd22283-kube-api-access-gxl66\") pod \"maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b\" (UID: \"45aca910-a167-41a3-8792-43502dd22283\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:37:34.527857 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.527820    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45aca910-a167-41a3-8792-43502dd22283-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b\" (UID: \"45aca910-a167-41a3-8792-43502dd22283\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:37:34.530098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.530064    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45aca910-a167-41a3-8792-43502dd22283-metrics-certs\") pod \"maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b\" (UID: \"45aca910-a167-41a3-8792-43502dd22283\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:37:34.530262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.530102    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45aca910-a167-41a3-8792-43502dd22283-maas-api-tls\") pod \"maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b\" (UID: \"45aca910-a167-41a3-8792-43502dd22283\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:37:34.549150 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.549109    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxl66\" (UniqueName: \"kubernetes.io/projected/45aca910-a167-41a3-8792-43502dd22283-kube-api-access-gxl66\") pod \"maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b\" (UID: \"45aca910-a167-41a3-8792-43502dd22283\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:37:34.583010 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.582973    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:37:34.896689 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:34.896657    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"]
Oct 30 01:37:34.897001 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:37:34.896974    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45aca910_a167_41a3_8792_43502dd22283.slice/crio-28be4bf89009b942056aa6926f13ba904985ced0d6c7ccde3af0029555cf24d0 WatchSource:0}: Error finding container 28be4bf89009b942056aa6926f13ba904985ced0d6c7ccde3af0029555cf24d0: Status 404 returned error can't find the container with id 28be4bf89009b942056aa6926f13ba904985ced0d6c7ccde3af0029555cf24d0
Oct 30 01:37:35.595974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:35.595892    4425 scope.go:122] "RemoveContainer" containerID="be5b37d2a6cf55bc4cedcf7159153bebb1b3ceec1290dc6dcd2895fb678749aa"
Oct 30 01:37:35.607258 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:35.607233    4425 scope.go:122] "RemoveContainer" containerID="c8f57c3fc79f9837bccfdb9f2f7cb42e0438cd0a31a17f05c5fccd3c989e8e29"
Oct 30 01:37:35.893362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:35.893254    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b" event={"ID":"45aca910-a167-41a3-8792-43502dd22283","Type":"ContainerStarted","Data":"0fcf5d10b55f07f02f684969a3430fd02602345202ba1f6110ac959f32cf9ce1"}
Oct 30 01:37:35.893362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:35.893327    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b" event={"ID":"45aca910-a167-41a3-8792-43502dd22283","Type":"ContainerStarted","Data":"28be4bf89009b942056aa6926f13ba904985ced0d6c7ccde3af0029555cf24d0"}
Oct 30 01:37:35.893362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:35.893363    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:37:35.934267 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:35.934192    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b" podStartSLOduration=1.549599336 podStartE2EDuration="1.934174418s" totalImagesPullingTime="384.575082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:35.930642565 +0000 UTC m=+3364.680297874" watchObservedRunningTime="2026-10-30 01:37:35.934174418 +0000 UTC m=+3364.683829728"
Oct 30 01:37:37.557711 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:37:37.557666    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:37:38.878854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:38.878816    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:37:41.909464 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:41.909436    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:37:57.263811 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:37:57.263705    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:38:07.557580 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:38:07.557540    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:38:35.646388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:35.646353    4425 scope.go:122] "RemoveContainer" containerID="8105e7d297222d6e5f2a95713f43709acecc5d0fe1bf201655f7e4c18ea2ca86"
Oct 30 01:38:37.557553 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:38:37.557506    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:38:41.662405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.662357    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"]
Oct 30 01:38:41.683028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.682983    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"]
Oct 30 01:38:41.683270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.683176    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.708332 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.708301    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-db4dd2\"/\"default-dockercfg-njg9v\""
Oct 30 01:38:41.708772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.708749    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-db4dd2\"/\"test-model-db4dd2-kserve-self-signed-certs\""
Oct 30 01:38:41.714941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.714905    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-db4dd2\"/\"openshift-service-ca.crt\""
Oct 30 01:38:41.714941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.714927    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-db4dd2\"/\"kube-root-ca.crt\""
Oct 30 01:38:41.786164 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.786117    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-home\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.786357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.786266    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-tmp-dir\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.786357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.786345    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-dshm\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.786442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.786387    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bac33d7a-c77f-4bfd-9881-790ba5559455-tls-certs\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.786664 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.786633    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-model-cache\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.786704 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.786689    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqzfw\" (UniqueName: \"kubernetes.io/projected/bac33d7a-c77f-4bfd-9881-790ba5559455-kube-api-access-bqzfw\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.888603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.888562    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-model-cache\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.888816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.888613    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqzfw\" (UniqueName: \"kubernetes.io/projected/bac33d7a-c77f-4bfd-9881-790ba5559455-kube-api-access-bqzfw\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.888816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.888706    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-home\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.888816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.888791    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-tmp-dir\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.889006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.888895    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-dshm\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.889071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.889046    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bac33d7a-c77f-4bfd-9881-790ba5559455-tls-certs\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.889071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.889060    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-model-cache\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.889194 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.889125    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-home\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.889616 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.889591    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-tmp-dir\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.891539 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.891510    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-dshm\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.891760 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.891739    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bac33d7a-c77f-4bfd-9881-790ba5559455-tls-certs\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:41.917610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:41.917530    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqzfw\" (UniqueName: \"kubernetes.io/projected/bac33d7a-c77f-4bfd-9881-790ba5559455-kube-api-access-bqzfw\") pod \"test-model-db4dd2-kserve-5b7d4677ff-p7jvk\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") " pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:42.014246 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:42.014206    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:42.388648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:42.388617    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"]
Oct 30 01:38:42.388804 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:38:42.388699    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbac33d7a_c77f_4bfd_9881_790ba5559455.slice/crio-6ef97a64bf17e04392c7df97af33e644149c44d3d13d99fa569ea3a1c7646cba WatchSource:0}: Error finding container 6ef97a64bf17e04392c7df97af33e644149c44d3d13d99fa569ea3a1c7646cba: Status 404 returned error can't find the container with id 6ef97a64bf17e04392c7df97af33e644149c44d3d13d99fa569ea3a1c7646cba
Oct 30 01:38:42.476295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:42.476258    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk" event={"ID":"bac33d7a-c77f-4bfd-9881-790ba5559455","Type":"ContainerStarted","Data":"6ef97a64bf17e04392c7df97af33e644149c44d3d13d99fa569ea3a1c7646cba"}
Oct 30 01:38:43.487961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:43.487919    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk" event={"ID":"bac33d7a-c77f-4bfd-9881-790ba5559455","Type":"ContainerStarted","Data":"08592d6661b9dfe04349e25d4be58be24e8208b6c4388893ac99981b7e7f2e45"}
Oct 30 01:38:43.528289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:43.528216    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk" podStartSLOduration=2.528197165 podStartE2EDuration="2.528197165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:43.525597449 +0000 UTC m=+3432.275252758" watchObservedRunningTime="2026-10-30 01:38:43.528197165 +0000 UTC m=+3432.277852474"
Oct 30 01:38:44.015085 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:44.015031    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:45.431086 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:45.431049    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:38:52.015019 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:52.014980    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:52.028460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:52.028434    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:52.582423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:52.582390    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:38:58.035656 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:58.035613    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"]
Oct 30 01:38:58.044521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:58.044483    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92369f04-tpbk2"]
Oct 30 01:38:59.035058 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:59.035021    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"]
Oct 30 01:38:59.044429 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:59.044387    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c62fa63e-9749s"]
Oct 30 01:38:59.426622 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:59.426543    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ccc1766-07a9-4c61-8528-83f7666a7f41" path="/var/lib/kubelet/pods/1ccc1766-07a9-4c61-8528-83f7666a7f41/volumes"
Oct 30 01:38:59.427109 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:38:59.427094    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84652b6a-91b8-472b-b17a-b715a34134ca" path="/var/lib/kubelet/pods/84652b6a-91b8-472b-b17a-b715a34134ca/volumes"
Oct 30 01:39:07.561700 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:39:07.561607    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:39:08.472780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:08.472736    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"]
Oct 30 01:39:08.472986 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:08.472965    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl" podUID="c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" containerName="istio-proxy" containerID="cri-o://2fa833472a79a8027ac56e751f311b2b8917468c23fee3912693671eee8656c9" gracePeriod=30
Oct 30 01:39:09.038725 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.038687    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"]
Oct 30 01:39:09.049083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.049043    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"]
Oct 30 01:39:09.055271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.055237    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7a1076f-jtmvt"]
Oct 30 01:39:09.059245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.059205    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"]
Oct 30 01:39:09.066040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.066012    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f53a04cb-qk2ss"]
Oct 30 01:39:09.066204 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.066128    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.080507 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.079513    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-73423b\"/\"openshift-service-ca.crt\""
Oct 30 01:39:09.080507 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.079610    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-73423b\"/\"kube-root-ca.crt\""
Oct 30 01:39:09.081012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.080821    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-73423b\"/\"test-model-73423b-kserve-self-signed-certs\""
Oct 30 01:39:09.081012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.080967    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-73423b\"/\"default-dockercfg-r9cfh\""
Oct 30 01:39:09.093205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.093168    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"]
Oct 30 01:39:09.141019 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.140976    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-dshm\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.141281 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.141046    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-tls-certs\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.141281 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.141140    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd5np\" (UniqueName: \"kubernetes.io/projected/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-kube-api-access-pd5np\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.141419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.141337    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-model-cache\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.141419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.141370    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-tmp-dir\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.141419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.141409    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-home\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.243389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.243342    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-dshm\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.243617 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.243410    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-tls-certs\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.243617 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.243519    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd5np\" (UniqueName: \"kubernetes.io/projected/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-kube-api-access-pd5np\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.243747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.243720    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-model-cache\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.243812 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.243771    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-tmp-dir\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.243871 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.243812    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-home\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.244346 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.244310    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-home\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.244501 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.244401    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-model-cache\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.244615 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.244592    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-tmp-dir\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.246190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.246138    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-dshm\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.251822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.251785    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-tls-certs\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.265078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.264975    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd5np\" (UniqueName: \"kubernetes.io/projected/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-kube-api-access-pd5np\") pod \"test-model-73423b-kserve-799894db4f-scxz4\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") " pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.397067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.395848    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:09.434409 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.434361    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37bf54df-0367-4579-a29d-2ac44a5c45eb" path="/var/lib/kubelet/pods/37bf54df-0367-4579-a29d-2ac44a5c45eb/volumes"
Oct 30 01:39:09.435445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.435417    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58aba58-d556-4af6-8e73-e4868ee15f47" path="/var/lib/kubelet/pods/f58aba58-d556-4af6-8e73-e4868ee15f47/volumes"
Oct 30 01:39:09.739249 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:39:09.733402    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b835d5c_e76c_4c40_a74c_ac9b6f5125af.slice/crio-667cebab3fe788f71c51ee877542db2c0608fb11b085dc2d900d42dee0b044af WatchSource:0}: Error finding container 667cebab3fe788f71c51ee877542db2c0608fb11b085dc2d900d42dee0b044af: Status 404 returned error can't find the container with id 667cebab3fe788f71c51ee877542db2c0608fb11b085dc2d900d42dee0b044af
Oct 30 01:39:09.739249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:09.734662    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"]
Oct 30 01:39:10.752560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:10.752524    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4" event={"ID":"5b835d5c-e76c-4c40-a74c-ac9b6f5125af","Type":"ContainerStarted","Data":"fc5899373694925328833cc7797196f3afd8c05e9b1ed520e8c326b9e3856288"}
Oct 30 01:39:10.752560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:10.752569    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4" event={"ID":"5b835d5c-e76c-4c40-a74c-ac9b6f5125af","Type":"ContainerStarted","Data":"667cebab3fe788f71c51ee877542db2c0608fb11b085dc2d900d42dee0b044af"}
Oct 30 01:39:10.791809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:10.791745    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4" podStartSLOduration=1.791729416 podStartE2EDuration="1.791729416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:10.790224557 +0000 UTC m=+3459.539879862" watchObservedRunningTime="2026-10-30 01:39:10.791729416 +0000 UTC m=+3459.541384741"
Oct 30 01:39:11.399271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:11.399239    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:13.542534 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.542499    4425 patch_prober.go:29] interesting pod/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 01:39:13.542886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.542558    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl" podUID="c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 01:39:13.764622 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.764591    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:39:13.783871 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.783831    4425 generic.go:356] "Generic (PLEG): container finished" podID="c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" containerID="2fa833472a79a8027ac56e751f311b2b8917468c23fee3912693671eee8656c9" exitCode=0
Oct 30 01:39:13.784092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.784055    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl" event={"ID":"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c","Type":"ContainerDied","Data":"2fa833472a79a8027ac56e751f311b2b8917468c23fee3912693671eee8656c9"}
Oct 30 01:39:13.784249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.784130    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl" event={"ID":"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c","Type":"ContainerDied","Data":"e8a9423962edd0ab0ac1aa8e55f357732c3f76b0a9a56ad39e6d0cc204e17d1f"}
Oct 30 01:39:13.784249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.784196    4425 scope.go:122] "RemoveContainer" containerID="2fa833472a79a8027ac56e751f311b2b8917468c23fee3912693671eee8656c9"
Oct 30 01:39:13.784556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.784372    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"
Oct 30 01:39:13.802055 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.802027    4425 scope.go:122] "RemoveContainer" containerID="2fa833472a79a8027ac56e751f311b2b8917468c23fee3912693671eee8656c9"
Oct 30 01:39:13.802462 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:39:13.802438    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa833472a79a8027ac56e751f311b2b8917468c23fee3912693671eee8656c9\": container with ID starting with 2fa833472a79a8027ac56e751f311b2b8917468c23fee3912693671eee8656c9 not found: ID does not exist" containerID="2fa833472a79a8027ac56e751f311b2b8917468c23fee3912693671eee8656c9"
Oct 30 01:39:13.802570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.802477    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa833472a79a8027ac56e751f311b2b8917468c23fee3912693671eee8656c9"} err="failed to get container status \"2fa833472a79a8027ac56e751f311b2b8917468c23fee3912693671eee8656c9\": rpc error: code = NotFound desc = could not find container \"2fa833472a79a8027ac56e751f311b2b8917468c23fee3912693671eee8656c9\": container with ID starting with 2fa833472a79a8027ac56e751f311b2b8917468c23fee3912693671eee8656c9 not found: ID does not exist"
Oct 30 01:39:13.919412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.919363    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-credential-socket\") pod \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") "
Oct 30 01:39:13.919622 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.919479    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-podinfo\") pod \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") "
Oct 30 01:39:13.919622 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.919513    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-workload-socket\") pod \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") "
Oct 30 01:39:13.919622 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.919552    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-workload-certs\") pod \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") "
Oct 30 01:39:13.919814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.919674    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-token\" (UniqueName: \"kubernetes.io/projected/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-token\") pod \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") "
Oct 30 01:39:13.919814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.919693    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-credential-socket" pod "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" (UID: "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:39:13.919814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.919732    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-data\") pod \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") "
Oct 30 01:39:13.919814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.919763    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-envoy\") pod \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") "
Oct 30 01:39:13.919814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.919798    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-kube-api-access-9pt54\" (UniqueName: \"kubernetes.io/projected/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-kube-api-access-9pt54\") pod \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") "
Oct 30 01:39:13.920094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.919841    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istiod-ca-cert\") pod \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\" (UID: \"c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c\") "
Oct 30 01:39:13.920094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.919907    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-workload-certs" pod "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" (UID: "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:39:13.920094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.920029    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-data" pod "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" (UID: "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:39:13.920285 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.920097    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-workload-socket" pod "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" (UID: "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:39:13.920519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.920457    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istiod-ca-cert" pod "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" (UID: "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:39:13.920727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.920679    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:39:13.920727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.920705    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:39:13.920727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.920720    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:39:13.920884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.920735    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:39:13.920884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.920748    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:39:13.922609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.922584    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-envoy" pod "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" (UID: "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:39:13.922736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.922638    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-token" pod "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" (UID: "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:39:13.922736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.922720    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-podinfo" pod "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" (UID: "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:39:13.922922 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:13.922899    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-kube-api-access-9pt54" pod "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" (UID: "c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c"). InnerVolumeSpecName "kube-api-access-9pt54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:39:14.022859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:14.022823    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:39:14.022859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:14.022858    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:39:14.023065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:14.022868    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:39:14.023065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:14.022878    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pt54\" (UniqueName: \"kubernetes.io/projected/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c-kube-api-access-9pt54\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:39:14.150931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:14.150896    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"]
Oct 30 01:39:14.157779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:14.157738    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1cb829-openshift-default-7959b897cc-fvtnl"]
Oct 30 01:39:15.427940 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:15.427899    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" path="/var/lib/kubelet/pods/c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c/volumes"
Oct 30 01:39:19.398734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:19.398697    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:19.412631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:19.412597    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:19.845811 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:19.845780    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:39:37.557417 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:39:37.557375    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:39:55.375462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:39:55.375408    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:40:03.322173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:03.322130    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"]
Oct 30 01:40:03.322831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:03.322352    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d" podUID="db496a54-58b2-4800-9cd5-0610da69a340" containerName="istio-proxy" containerID="cri-o://398df3c35b00bbeb936eecedfbb3ff55de67bef2db8bb9fc54ae0dcb4be0c694" gracePeriod=30
Oct 30 01:40:07.401761 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:07.401718    4425 patch_prober.go:29] interesting pod/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:40:07.402231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:07.401787    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d" podUID="db496a54-58b2-4800-9cd5-0610da69a340" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:40:07.556879 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:40:07.556843    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:40:08.615749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.615714    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 01:40:08.707091 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.707057    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-istio-data\") pod \"db496a54-58b2-4800-9cd5-0610da69a340\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") "
Oct 30 01:40:08.707455 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.707427    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-workload-certs\") pod \"db496a54-58b2-4800-9cd5-0610da69a340\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") "
Oct 30 01:40:08.707778 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.707760    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-workload-socket\") pod \"db496a54-58b2-4800-9cd5-0610da69a340\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") "
Oct 30 01:40:08.707973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.707947    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db496a54-58b2-4800-9cd5-0610da69a340-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db496a54-58b2-4800-9cd5-0610da69a340-istio-podinfo\") pod \"db496a54-58b2-4800-9cd5-0610da69a340\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") "
Oct 30 01:40:08.708119 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.707999    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db496a54-58b2-4800-9cd5-0610da69a340-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db496a54-58b2-4800-9cd5-0610da69a340-istiod-ca-cert\") pod \"db496a54-58b2-4800-9cd5-0610da69a340\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") "
Oct 30 01:40:08.708322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.708302    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-credential-socket\") pod \"db496a54-58b2-4800-9cd5-0610da69a340\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") "
Oct 30 01:40:08.708414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.708343    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-istio-envoy\") pod \"db496a54-58b2-4800-9cd5-0610da69a340\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") "
Oct 30 01:40:08.708414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.708390    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db496a54-58b2-4800-9cd5-0610da69a340-kube-api-access-z5q2z\" (UniqueName: \"kubernetes.io/projected/db496a54-58b2-4800-9cd5-0610da69a340-kube-api-access-z5q2z\") pod \"db496a54-58b2-4800-9cd5-0610da69a340\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") "
Oct 30 01:40:08.708545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.708432    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db496a54-58b2-4800-9cd5-0610da69a340-istio-token\" (UniqueName: \"kubernetes.io/projected/db496a54-58b2-4800-9cd5-0610da69a340-istio-token\") pod \"db496a54-58b2-4800-9cd5-0610da69a340\" (UID: \"db496a54-58b2-4800-9cd5-0610da69a340\") "
Oct 30 01:40:08.709403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.709147    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-credential-socket" pod "db496a54-58b2-4800-9cd5-0610da69a340" (UID: "db496a54-58b2-4800-9cd5-0610da69a340"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:40:08.709403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.709333    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:40:08.709972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.709695    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db496a54-58b2-4800-9cd5-0610da69a340-istiod-ca-cert" pod "db496a54-58b2-4800-9cd5-0610da69a340" (UID: "db496a54-58b2-4800-9cd5-0610da69a340"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:40:08.710898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.710806    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-istio-data" pod "db496a54-58b2-4800-9cd5-0610da69a340" (UID: "db496a54-58b2-4800-9cd5-0610da69a340"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:40:08.710898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.710873    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-workload-socket" pod "db496a54-58b2-4800-9cd5-0610da69a340" (UID: "db496a54-58b2-4800-9cd5-0610da69a340"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:40:08.711321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.711298    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-workload-certs" pod "db496a54-58b2-4800-9cd5-0610da69a340" (UID: "db496a54-58b2-4800-9cd5-0610da69a340"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:40:08.712414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.712386    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db496a54-58b2-4800-9cd5-0610da69a340-istio-token" pod "db496a54-58b2-4800-9cd5-0610da69a340" (UID: "db496a54-58b2-4800-9cd5-0610da69a340"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:40:08.713912 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.713877    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db496a54-58b2-4800-9cd5-0610da69a340-istio-podinfo" pod "db496a54-58b2-4800-9cd5-0610da69a340" (UID: "db496a54-58b2-4800-9cd5-0610da69a340"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:40:08.714116 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.714078    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-istio-envoy" pod "db496a54-58b2-4800-9cd5-0610da69a340" (UID: "db496a54-58b2-4800-9cd5-0610da69a340"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:40:08.714876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.714843    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db496a54-58b2-4800-9cd5-0610da69a340-kube-api-access-z5q2z" pod "db496a54-58b2-4800-9cd5-0610da69a340" (UID: "db496a54-58b2-4800-9cd5-0610da69a340"). InnerVolumeSpecName "kube-api-access-z5q2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:40:08.811225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.811189    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:40:08.811225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.811227    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:40:08.811518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.811241    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:40:08.811518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.811253    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db496a54-58b2-4800-9cd5-0610da69a340-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:40:08.811518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.811263    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db496a54-58b2-4800-9cd5-0610da69a340-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:40:08.811518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.811271    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db496a54-58b2-4800-9cd5-0610da69a340-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:40:08.811518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.811280    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5q2z\" (UniqueName: \"kubernetes.io/projected/db496a54-58b2-4800-9cd5-0610da69a340-kube-api-access-z5q2z\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:40:08.811518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:08.811289    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db496a54-58b2-4800-9cd5-0610da69a340-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:40:09.281498 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:09.281458    4425 generic.go:356] "Generic (PLEG): container finished" podID="db496a54-58b2-4800-9cd5-0610da69a340" containerID="398df3c35b00bbeb936eecedfbb3ff55de67bef2db8bb9fc54ae0dcb4be0c694" exitCode=0
Oct 30 01:40:09.281666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:09.281510    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"
Oct 30 01:40:09.281666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:09.281540    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d" event={"ID":"db496a54-58b2-4800-9cd5-0610da69a340","Type":"ContainerDied","Data":"398df3c35b00bbeb936eecedfbb3ff55de67bef2db8bb9fc54ae0dcb4be0c694"}
Oct 30 01:40:09.281666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:09.281598    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d" event={"ID":"db496a54-58b2-4800-9cd5-0610da69a340","Type":"ContainerDied","Data":"029cdcc7e544a79cd11701d2ec392a3db06f18281cdadecb50dd24cd50201d57"}
Oct 30 01:40:09.281666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:09.281620    4425 scope.go:122] "RemoveContainer" containerID="398df3c35b00bbeb936eecedfbb3ff55de67bef2db8bb9fc54ae0dcb4be0c694"
Oct 30 01:40:09.295349 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:09.295327    4425 scope.go:122] "RemoveContainer" containerID="398df3c35b00bbeb936eecedfbb3ff55de67bef2db8bb9fc54ae0dcb4be0c694"
Oct 30 01:40:09.295658 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:40:09.295635    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398df3c35b00bbeb936eecedfbb3ff55de67bef2db8bb9fc54ae0dcb4be0c694\": container with ID starting with 398df3c35b00bbeb936eecedfbb3ff55de67bef2db8bb9fc54ae0dcb4be0c694 not found: ID does not exist" containerID="398df3c35b00bbeb936eecedfbb3ff55de67bef2db8bb9fc54ae0dcb4be0c694"
Oct 30 01:40:09.295770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:09.295664    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398df3c35b00bbeb936eecedfbb3ff55de67bef2db8bb9fc54ae0dcb4be0c694"} err="failed to get container status \"398df3c35b00bbeb936eecedfbb3ff55de67bef2db8bb9fc54ae0dcb4be0c694\": rpc error: code = NotFound desc = could not find container \"398df3c35b00bbeb936eecedfbb3ff55de67bef2db8bb9fc54ae0dcb4be0c694\": container with ID starting with 398df3c35b00bbeb936eecedfbb3ff55de67bef2db8bb9fc54ae0dcb4be0c694 not found: ID does not exist"
Oct 30 01:40:09.330653 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:09.330616    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"]
Oct 30 01:40:09.344448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:09.344399    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-664c2b-openshift-default-8449b76bd5-spg8d"]
Oct 30 01:40:09.426069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:09.426035    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db496a54-58b2-4800-9cd5-0610da69a340" path="/var/lib/kubelet/pods/db496a54-58b2-4800-9cd5-0610da69a340/volumes"
Oct 30 01:40:26.031357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:26.031315    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"]
Oct 30 01:40:26.040465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:26.040429    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb5237dd-qqf2d"]
Oct 30 01:40:27.426053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:27.426005    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96eabac0-f69e-4824-977e-1bc744137a8f" path="/var/lib/kubelet/pods/96eabac0-f69e-4824-977e-1bc744137a8f/volumes"
Oct 30 01:40:35.705515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:35.705483    4425 scope.go:122] "RemoveContainer" containerID="eb81a5ca322a0dbee221bdcef6ea798e96ec7df9795f84260a0e91795b890bab"
Oct 30 01:40:35.717150 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:35.717129    4425 scope.go:122] "RemoveContainer" containerID="6364be1f6261250f741a7fc8e20bcf8db5efb21e09a3de99c024479ba09138ff"
Oct 30 01:40:35.727748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:35.727730    4425 scope.go:122] "RemoveContainer" containerID="529b6b92334facecbf93038ab5cf740ad3998881408682fac2d6182b45362f1c"
Oct 30 01:40:35.737489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:35.737468    4425 scope.go:122] "RemoveContainer" containerID="69a588b38f885f195b177ad7498b17d9abf38cf3a03d85caf874d468af81bacb"
Oct 30 01:40:37.031806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:37.031770    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"]
Oct 30 01:40:37.040089 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:37.040061    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37c25e35-nmgpc"]
Oct 30 01:40:37.425895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:37.425816    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67e991c-0211-406a-bbcb-3933ffdce18d" path="/var/lib/kubelet/pods/a67e991c-0211-406a-bbcb-3933ffdce18d/volumes"
Oct 30 01:40:37.559292 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:40:37.559251    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:40:44.926632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:44.926533    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"]
Oct 30 01:40:44.928058 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:44.928032    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" containerName="istio-proxy"
Oct 30 01:40:44.928171 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:44.928061    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" containerName="istio-proxy"
Oct 30 01:40:44.928171 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:44.928115    4425 cpu_manager.go:414] "containerMap: removing container" podUID="db496a54-58b2-4800-9cd5-0610da69a340" containerName="istio-proxy"
Oct 30 01:40:44.928171 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:44.928123    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db496a54-58b2-4800-9cd5-0610da69a340" containerName="istio-proxy"
Oct 30 01:40:44.928424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:44.928409    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="db496a54-58b2-4800-9cd5-0610da69a340" containerName="istio-proxy"
Oct 30 01:40:44.928458 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:44.928444    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7cfcf3c-62bf-4b6d-9b78-abf5b6ae290c" containerName="istio-proxy"
Oct 30 01:40:44.933722 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:44.933691    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:44.946953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:44.946926    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6aa9aa\"/\"openshift-service-ca.crt\""
Oct 30 01:40:44.956877 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:44.956845    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6aa9aa\"/\"default-dockercfg-sxvcs\""
Oct 30 01:40:44.957368 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:44.957336    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6aa9aa\"/\"test-model-6aa9aa-kserve-self-signed-certs\""
Oct 30 01:40:44.958333 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:44.957918    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6aa9aa\"/\"kube-root-ca.crt\""
Oct 30 01:40:44.962243 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:44.962218    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"]
Oct 30 01:40:45.063434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.063396    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-dshm\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.063667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.063578    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc7d715c-4856-4653-bc23-e7479bb24f9a-tls-certs\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.063667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.063619    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-model-cache\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.063777 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.063729    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-home\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.063824 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.063792    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-tmp-dir\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.063889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.063872    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wv4s\" (UniqueName: \"kubernetes.io/projected/fc7d715c-4856-4653-bc23-e7479bb24f9a-kube-api-access-7wv4s\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.165874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.165825    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-dshm\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.166094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.165993    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc7d715c-4856-4653-bc23-e7479bb24f9a-tls-certs\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.166094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.166023    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-model-cache\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.166094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.166068    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-home\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.166315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.166120    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-tmp-dir\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.166315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.166208    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wv4s\" (UniqueName: \"kubernetes.io/projected/fc7d715c-4856-4653-bc23-e7479bb24f9a-kube-api-access-7wv4s\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.166716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.166684    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-home\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.166848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.166818    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-model-cache\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.167024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.167000    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-tmp-dir\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.169217 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.169188    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-dshm\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.185014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.184907    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc7d715c-4856-4653-bc23-e7479bb24f9a-tls-certs\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.198591 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.196777    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wv4s\" (UniqueName: \"kubernetes.io/projected/fc7d715c-4856-4653-bc23-e7479bb24f9a-kube-api-access-7wv4s\") pod \"test-model-6aa9aa-kserve-5b7c4485d9-q4g2v\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") " pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.255461 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.255428    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:45.611808 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:45.611778    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"]
Oct 30 01:40:46.619364 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:46.619319    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v" event={"ID":"fc7d715c-4856-4653-bc23-e7479bb24f9a","Type":"ContainerStarted","Data":"800d78103f5c9d8d09c245d509f5078948edb4632caaa9896007f8f0b6036f01"}
Oct 30 01:40:46.619808 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:46.619374    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v" event={"ID":"fc7d715c-4856-4653-bc23-e7479bb24f9a","Type":"ContainerStarted","Data":"d0c3821547bce83165a4723dcb5e642b76d87e55ed161fd4f7e45aeef3db8c7d"}
Oct 30 01:40:46.659051 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:46.658994    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v" podStartSLOduration=2.658977864 podStartE2EDuration="2.658977864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:46.655145424 +0000 UTC m=+3555.404800734" watchObservedRunningTime="2026-10-30 01:40:46.658977864 +0000 UTC m=+3555.408633172"
Oct 30 01:40:47.256351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:47.256311    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:50.031426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:50.031378    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"]
Oct 30 01:40:50.040489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:50.040454    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29d2aedf-lx9xt"]
Oct 30 01:40:51.427512 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:51.427475    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46a79586-7165-4134-a157-7cf631b357da" path="/var/lib/kubelet/pods/46a79586-7165-4134-a157-7cf631b357da/volumes"
Oct 30 01:40:55.256328 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:55.256290    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:55.269878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:55.269843    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:40:55.720652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:40:55.720611    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:41:01.031440 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:01.031406    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"]
Oct 30 01:41:01.040403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:01.040373    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7237ce6-hjwdb"]
Oct 30 01:41:01.426436 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:01.426353    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="196a8ae2-eb90-45d3-8da7-3be8661f6951" path="/var/lib/kubelet/pods/196a8ae2-eb90-45d3-8da7-3be8661f6951/volumes"
Oct 30 01:41:07.557513 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:41:07.557475    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:41:11.515393 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:11.515357    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"]
Oct 30 01:41:11.520245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:11.520225    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"
Oct 30 01:41:11.541586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:11.541556    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"]
Oct 30 01:41:11.545863 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:11.545838    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:41:11.545985 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:11.545891    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:41:11.679350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:11.679315    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6bd158-e648-47bb-867d-4435ba723c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4611660-fh9hc\" (UID: \"3f6bd158-e648-47bb-867d-4435ba723c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"
Oct 30 01:41:11.679533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:11.679378    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6bd158-e648-47bb-867d-4435ba723c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4611660-fh9hc\" (UID: \"3f6bd158-e648-47bb-867d-4435ba723c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"
Oct 30 01:41:11.679533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:11.679460    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hcsd\" (UniqueName: \"kubernetes.io/projected/3f6bd158-e648-47bb-867d-4435ba723c72-kube-api-access-7hcsd\") pod \"maas-api-revoke-sub-a4611660-fh9hc\" (UID: \"3f6bd158-e648-47bb-867d-4435ba723c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"
Oct 30 01:41:11.781056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:11.780957    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6bd158-e648-47bb-867d-4435ba723c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4611660-fh9hc\" (UID: \"3f6bd158-e648-47bb-867d-4435ba723c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"
Oct 30 01:41:11.781056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:11.781026    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6bd158-e648-47bb-867d-4435ba723c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4611660-fh9hc\" (UID: \"3f6bd158-e648-47bb-867d-4435ba723c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"
Oct 30 01:41:11.781440 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:11.781072    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hcsd\" (UniqueName: \"kubernetes.io/projected/3f6bd158-e648-47bb-867d-4435ba723c72-kube-api-access-7hcsd\") pod \"maas-api-revoke-sub-a4611660-fh9hc\" (UID: \"3f6bd158-e648-47bb-867d-4435ba723c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"
Oct 30 01:41:11.781848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:11.781821    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6bd158-e648-47bb-867d-4435ba723c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4611660-fh9hc\" (UID: \"3f6bd158-e648-47bb-867d-4435ba723c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"
Oct 30 01:41:11.783564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:11.783539    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6bd158-e648-47bb-867d-4435ba723c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4611660-fh9hc\" (UID: \"3f6bd158-e648-47bb-867d-4435ba723c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"
Oct 30 01:41:11.802092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:11.802062    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hcsd\" (UniqueName: \"kubernetes.io/projected/3f6bd158-e648-47bb-867d-4435ba723c72-kube-api-access-7hcsd\") pod \"maas-api-revoke-sub-a4611660-fh9hc\" (UID: \"3f6bd158-e648-47bb-867d-4435ba723c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"
Oct 30 01:41:11.835426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:11.835390    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"
Oct 30 01:41:12.147608 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:12.147568    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"]
Oct 30 01:41:12.147749 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:41:12.147662    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6bd158_e648_47bb_867d_4435ba723c72.slice/crio-93a78b74acd59b478e47c6012948e8d2bb66df8ea9a9bf4561d3245f5ec8e108 WatchSource:0}: Error finding container 93a78b74acd59b478e47c6012948e8d2bb66df8ea9a9bf4561d3245f5ec8e108: Status 404 returned error can't find the container with id 93a78b74acd59b478e47c6012948e8d2bb66df8ea9a9bf4561d3245f5ec8e108
Oct 30 01:41:12.858846 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:12.858807    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc" event={"ID":"3f6bd158-e648-47bb-867d-4435ba723c72","Type":"ContainerStarted","Data":"d48669e53de67035a8e22fcc5556907934be8f0508cf5d551c06f6aba04ca6fc"}
Oct 30 01:41:12.859361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:12.858875    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc" event={"ID":"3f6bd158-e648-47bb-867d-4435ba723c72","Type":"ContainerStarted","Data":"93a78b74acd59b478e47c6012948e8d2bb66df8ea9a9bf4561d3245f5ec8e108"}
Oct 30 01:41:12.896087 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:12.896028    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc" podStartSLOduration=1.896013771 podStartE2EDuration="1.896013771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:12.894419603 +0000 UTC m=+3581.644074912" watchObservedRunningTime="2026-10-30 01:41:12.896013771 +0000 UTC m=+3581.645669080"
Oct 30 01:41:13.396448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.396409    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"]
Oct 30 01:41:13.403268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.403241    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.427463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.424841    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1a4b17\"/\"kube-root-ca.crt\""
Oct 30 01:41:13.427463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.424943    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1a4b17\"/\"openshift-service-ca.crt\""
Oct 30 01:41:13.427463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.425090    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1a4b17\"/\"default-dockercfg-26q2j\""
Oct 30 01:41:13.431784 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.431757    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1a4b17\"/\"test-model-1a4b17-kserve-self-signed-certs\""
Oct 30 01:41:13.440494 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.440462    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"]
Oct 30 01:41:13.508846 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.508798    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-tmp-dir\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.509006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.508950    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-model-cache\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.509209 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.509130    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e560b5a5-44db-4f11-ba85-30220486eb66-tls-certs\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.509347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.509322    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-dshm\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.509391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.509363    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7gcx\" (UniqueName: \"kubernetes.io/projected/e560b5a5-44db-4f11-ba85-30220486eb66-kube-api-access-g7gcx\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.509469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.509448    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-home\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.611557 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.611513    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-tmp-dir\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.611753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.611630    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-model-cache\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.611817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.611765    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e560b5a5-44db-4f11-ba85-30220486eb66-tls-certs\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.611899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.611876    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-dshm\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.611986 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.611910    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7gcx\" (UniqueName: \"kubernetes.io/projected/e560b5a5-44db-4f11-ba85-30220486eb66-kube-api-access-g7gcx\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.611986 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.611947    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-home\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.612109 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.612029    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-tmp-dir\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.612296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.612098    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-model-cache\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.612431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.612401    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-home\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.614858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.614830    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e560b5a5-44db-4f11-ba85-30220486eb66-tls-certs\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.615131 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.615109    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-dshm\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.633615 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.633583    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7gcx\" (UniqueName: \"kubernetes.io/projected/e560b5a5-44db-4f11-ba85-30220486eb66-kube-api-access-g7gcx\") pod \"test-model-1a4b17-kserve-54d78d64c-2jx7c\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") " pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.737100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.736992    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:13.872086 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.872051    4425 generic.go:356] "Generic (PLEG): container finished" podID="3f6bd158-e648-47bb-867d-4435ba723c72" containerID="d48669e53de67035a8e22fcc5556907934be8f0508cf5d551c06f6aba04ca6fc" exitCode=0
Oct 30 01:41:13.872644 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:13.872118    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc" event={"ID":"3f6bd158-e648-47bb-867d-4435ba723c72","Type":"ContainerDied","Data":"d48669e53de67035a8e22fcc5556907934be8f0508cf5d551c06f6aba04ca6fc"}
Oct 30 01:41:14.071859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:14.071828    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"]
Oct 30 01:41:14.073937 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:41:14.073635    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode560b5a5_44db_4f11_ba85_30220486eb66.slice/crio-39c486ec34b2a24ed6e2a5fbb6bf35274111fb5bf2ae639bf713986a53799ce1 WatchSource:0}: Error finding container 39c486ec34b2a24ed6e2a5fbb6bf35274111fb5bf2ae639bf713986a53799ce1: Status 404 returned error can't find the container with id 39c486ec34b2a24ed6e2a5fbb6bf35274111fb5bf2ae639bf713986a53799ce1
Oct 30 01:41:14.881605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:14.881561    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c" event={"ID":"e560b5a5-44db-4f11-ba85-30220486eb66","Type":"ContainerStarted","Data":"56da17c33394d79c73935b6e543c53e2ca4b69623be2b0b7c6b9963379866845"}
Oct 30 01:41:14.882110 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:14.881617    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c" event={"ID":"e560b5a5-44db-4f11-ba85-30220486eb66","Type":"ContainerStarted","Data":"39c486ec34b2a24ed6e2a5fbb6bf35274111fb5bf2ae639bf713986a53799ce1"}
Oct 30 01:41:14.922686 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:14.922630    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c" podStartSLOduration=1.92261226 podStartE2EDuration="1.92261226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:14.917414286 +0000 UTC m=+3583.667069595" watchObservedRunningTime="2026-10-30 01:41:14.92261226 +0000 UTC m=+3583.672267629"
Oct 30 01:41:15.111748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:15.111717    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"
Oct 30 01:41:15.137379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:15.137277    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f6bd158-e648-47bb-867d-4435ba723c72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6bd158-e648-47bb-867d-4435ba723c72-maas-api-xks-ca\") pod \"3f6bd158-e648-47bb-867d-4435ba723c72\" (UID: \"3f6bd158-e648-47bb-867d-4435ba723c72\") "
Oct 30 01:41:15.137379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:15.137355    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f6bd158-e648-47bb-867d-4435ba723c72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6bd158-e648-47bb-867d-4435ba723c72-maas-api-openshift-ca\") pod \"3f6bd158-e648-47bb-867d-4435ba723c72\" (UID: \"3f6bd158-e648-47bb-867d-4435ba723c72\") "
Oct 30 01:41:15.137643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:15.137612    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f6bd158-e648-47bb-867d-4435ba723c72-kube-api-access-7hcsd\" (UniqueName: \"kubernetes.io/projected/3f6bd158-e648-47bb-867d-4435ba723c72-kube-api-access-7hcsd\") pod \"3f6bd158-e648-47bb-867d-4435ba723c72\" (UID: \"3f6bd158-e648-47bb-867d-4435ba723c72\") "
Oct 30 01:41:15.137779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:15.137755    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f6bd158-e648-47bb-867d-4435ba723c72-maas-api-openshift-ca" pod "3f6bd158-e648-47bb-867d-4435ba723c72" (UID: "3f6bd158-e648-47bb-867d-4435ba723c72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:41:15.138187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:15.138135    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f6bd158-e648-47bb-867d-4435ba723c72-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:15.139497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:15.139472    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f6bd158-e648-47bb-867d-4435ba723c72-maas-api-xks-ca" pod "3f6bd158-e648-47bb-867d-4435ba723c72" (UID: "3f6bd158-e648-47bb-867d-4435ba723c72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:41:15.139687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:15.139667    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6bd158-e648-47bb-867d-4435ba723c72-kube-api-access-7hcsd" pod "3f6bd158-e648-47bb-867d-4435ba723c72" (UID: "3f6bd158-e648-47bb-867d-4435ba723c72"). InnerVolumeSpecName "kube-api-access-7hcsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:41:15.239212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:15.239174    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hcsd\" (UniqueName: \"kubernetes.io/projected/3f6bd158-e648-47bb-867d-4435ba723c72-kube-api-access-7hcsd\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:15.239212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:15.239206    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f6bd158-e648-47bb-867d-4435ba723c72-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:15.737200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:15.737103    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:15.756610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:15.756579    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:41:15.891794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:15.891764    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"
Oct 30 01:41:15.892223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:15.891804    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc" event={"ID":"3f6bd158-e648-47bb-867d-4435ba723c72","Type":"ContainerDied","Data":"93a78b74acd59b478e47c6012948e8d2bb66df8ea9a9bf4561d3245f5ec8e108"}
Oct 30 01:41:15.892223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:15.891855    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93a78b74acd59b478e47c6012948e8d2bb66df8ea9a9bf4561d3245f5ec8e108"
Oct 30 01:41:23.095297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.095258    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"]
Oct 30 01:41:23.657309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.657270    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"]
Oct 30 01:41:23.658344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.658317    4425 cpu_manager.go:414] "containerMap: removing container" podUID="3f6bd158-e648-47bb-867d-4435ba723c72" containerName="revoke-keys"
Oct 30 01:41:23.658344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.658342    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f6bd158-e648-47bb-867d-4435ba723c72" containerName="revoke-keys"
Oct 30 01:41:23.658558 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.658547    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f6bd158-e648-47bb-867d-4435ba723c72" containerName="revoke-keys"
Oct 30 01:41:23.666626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.666577    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"
Oct 30 01:41:23.682210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.682178    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:41:23.682379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.682251    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:41:23.687536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.687501    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"]
Oct 30 01:41:23.737369 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.737335    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:23.752098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.752069    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:23.785412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.785375    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs49n\" (UniqueName: \"kubernetes.io/projected/2113693f-bc34-42c1-8fea-1dc510220472-kube-api-access-bs49n\") pod \"maas-api-revoke-sub-44621952-xgh4t\" (UID: \"2113693f-bc34-42c1-8fea-1dc510220472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"
Oct 30 01:41:23.785633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.785443    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2113693f-bc34-42c1-8fea-1dc510220472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44621952-xgh4t\" (UID: \"2113693f-bc34-42c1-8fea-1dc510220472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"
Oct 30 01:41:23.785633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.785519    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2113693f-bc34-42c1-8fea-1dc510220472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44621952-xgh4t\" (UID: \"2113693f-bc34-42c1-8fea-1dc510220472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"
Oct 30 01:41:23.886860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.886825    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs49n\" (UniqueName: \"kubernetes.io/projected/2113693f-bc34-42c1-8fea-1dc510220472-kube-api-access-bs49n\") pod \"maas-api-revoke-sub-44621952-xgh4t\" (UID: \"2113693f-bc34-42c1-8fea-1dc510220472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"
Oct 30 01:41:23.887052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.886888    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2113693f-bc34-42c1-8fea-1dc510220472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44621952-xgh4t\" (UID: \"2113693f-bc34-42c1-8fea-1dc510220472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"
Oct 30 01:41:23.887052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.886910    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2113693f-bc34-42c1-8fea-1dc510220472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44621952-xgh4t\" (UID: \"2113693f-bc34-42c1-8fea-1dc510220472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"
Oct 30 01:41:23.887671 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.887639    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2113693f-bc34-42c1-8fea-1dc510220472-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44621952-xgh4t\" (UID: \"2113693f-bc34-42c1-8fea-1dc510220472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"
Oct 30 01:41:23.889546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.889516    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2113693f-bc34-42c1-8fea-1dc510220472-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44621952-xgh4t\" (UID: \"2113693f-bc34-42c1-8fea-1dc510220472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"
Oct 30 01:41:23.908418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.908348    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs49n\" (UniqueName: \"kubernetes.io/projected/2113693f-bc34-42c1-8fea-1dc510220472-kube-api-access-bs49n\") pod \"maas-api-revoke-sub-44621952-xgh4t\" (UID: \"2113693f-bc34-42c1-8fea-1dc510220472\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"
Oct 30 01:41:23.975173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.975135    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:41:23.997732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:23.997700    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"
Oct 30 01:41:24.349260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:24.346648    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"]
Oct 30 01:41:24.973460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:24.973356    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t" event={"ID":"2113693f-bc34-42c1-8fea-1dc510220472","Type":"ContainerStarted","Data":"39c9f888988d58b77e6db19e0049508562993649c29e8b64e3fee720e11dbd24"}
Oct 30 01:41:24.973460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:24.973423    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t" event={"ID":"2113693f-bc34-42c1-8fea-1dc510220472","Type":"ContainerStarted","Data":"d88ebb2e9db30aaf31c179e2428efb31d5d95ed2019dd60a0cf0e668af4dd8c1"}
Oct 30 01:41:25.032295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:25.032240    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t" podStartSLOduration=2.032227469 podStartE2EDuration="2.032227469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:25.030480969 +0000 UTC m=+3593.780136278" watchObservedRunningTime="2026-10-30 01:41:25.032227469 +0000 UTC m=+3593.781882799"
Oct 30 01:41:25.983131 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:25.983093    4425 generic.go:356] "Generic (PLEG): container finished" podID="2113693f-bc34-42c1-8fea-1dc510220472" containerID="39c9f888988d58b77e6db19e0049508562993649c29e8b64e3fee720e11dbd24" exitCode=0
Oct 30 01:41:25.983131 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:25.983126    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t" event={"ID":"2113693f-bc34-42c1-8fea-1dc510220472","Type":"ContainerDied","Data":"39c9f888988d58b77e6db19e0049508562993649c29e8b64e3fee720e11dbd24"}
Oct 30 01:41:27.192749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:27.192718    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"
Oct 30 01:41:27.342117 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:27.342079    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2113693f-bc34-42c1-8fea-1dc510220472-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2113693f-bc34-42c1-8fea-1dc510220472-maas-api-xks-ca\") pod \"2113693f-bc34-42c1-8fea-1dc510220472\" (UID: \"2113693f-bc34-42c1-8fea-1dc510220472\") "
Oct 30 01:41:27.342370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:27.342351    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2113693f-bc34-42c1-8fea-1dc510220472-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2113693f-bc34-42c1-8fea-1dc510220472-maas-api-openshift-ca\") pod \"2113693f-bc34-42c1-8fea-1dc510220472\" (UID: \"2113693f-bc34-42c1-8fea-1dc510220472\") "
Oct 30 01:41:27.342451 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:27.342441    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2113693f-bc34-42c1-8fea-1dc510220472-kube-api-access-bs49n\" (UniqueName: \"kubernetes.io/projected/2113693f-bc34-42c1-8fea-1dc510220472-kube-api-access-bs49n\") pod \"2113693f-bc34-42c1-8fea-1dc510220472\" (UID: \"2113693f-bc34-42c1-8fea-1dc510220472\") "
Oct 30 01:41:27.342740 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:27.342710    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2113693f-bc34-42c1-8fea-1dc510220472-maas-api-openshift-ca" pod "2113693f-bc34-42c1-8fea-1dc510220472" (UID: "2113693f-bc34-42c1-8fea-1dc510220472"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:41:27.343113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:27.343090    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2113693f-bc34-42c1-8fea-1dc510220472-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:27.344455 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:27.344427    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2113693f-bc34-42c1-8fea-1dc510220472-maas-api-xks-ca" pod "2113693f-bc34-42c1-8fea-1dc510220472" (UID: "2113693f-bc34-42c1-8fea-1dc510220472"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:41:27.344581 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:27.344559    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2113693f-bc34-42c1-8fea-1dc510220472-kube-api-access-bs49n" pod "2113693f-bc34-42c1-8fea-1dc510220472" (UID: "2113693f-bc34-42c1-8fea-1dc510220472"). InnerVolumeSpecName "kube-api-access-bs49n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:41:27.444521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:27.444486    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2113693f-bc34-42c1-8fea-1dc510220472-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:27.444521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:27.444519    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs49n\" (UniqueName: \"kubernetes.io/projected/2113693f-bc34-42c1-8fea-1dc510220472-kube-api-access-bs49n\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:28.004908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:28.004871    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"
Oct 30 01:41:28.004908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:28.004888    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t" event={"ID":"2113693f-bc34-42c1-8fea-1dc510220472","Type":"ContainerDied","Data":"d88ebb2e9db30aaf31c179e2428efb31d5d95ed2019dd60a0cf0e668af4dd8c1"}
Oct 30 01:41:28.005250 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:28.004940    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d88ebb2e9db30aaf31c179e2428efb31d5d95ed2019dd60a0cf0e668af4dd8c1"
Oct 30 01:41:35.790990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:35.790947    4425 scope.go:122] "RemoveContainer" containerID="4e45161aea091496c0c721370392b8df7584da86b14ee7e44004e59b572caba4"
Oct 30 01:41:35.802583 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:35.802554    4425 scope.go:122] "RemoveContainer" containerID="ddbc8b866d0b7da01cefdd2c808c20ba7df1b395c74d030462a0344825513c9b"
Oct 30 01:41:37.560006 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:41:37.559955    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:41:37.648379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.648336    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"]
Oct 30 01:41:37.649773 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.649739    4425 cpu_manager.go:414] "containerMap: removing container" podUID="2113693f-bc34-42c1-8fea-1dc510220472" containerName="revoke-keys"
Oct 30 01:41:37.649773 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.649766    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2113693f-bc34-42c1-8fea-1dc510220472" containerName="revoke-keys"
Oct 30 01:41:37.650057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.649955    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2113693f-bc34-42c1-8fea-1dc510220472" containerName="revoke-keys"
Oct 30 01:41:37.654000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.653977    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"
Oct 30 01:41:37.676373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.676337    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:41:37.676373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.676369    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:41:37.682822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.682768    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"]
Oct 30 01:41:37.804984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.804933    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfzpk\" (UniqueName: \"kubernetes.io/projected/84a361f3-187a-4583-bbb2-6e8064cc948b-kube-api-access-tfzpk\") pod \"maas-api-revoke-sub-52dd0bed-84qvv\" (UID: \"84a361f3-187a-4583-bbb2-6e8064cc948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"
Oct 30 01:41:37.805256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.805097    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a361f3-187a-4583-bbb2-6e8064cc948b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52dd0bed-84qvv\" (UID: \"84a361f3-187a-4583-bbb2-6e8064cc948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"
Oct 30 01:41:37.805325 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.805299    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a361f3-187a-4583-bbb2-6e8064cc948b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52dd0bed-84qvv\" (UID: \"84a361f3-187a-4583-bbb2-6e8064cc948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"
Oct 30 01:41:37.906908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.906815    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a361f3-187a-4583-bbb2-6e8064cc948b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52dd0bed-84qvv\" (UID: \"84a361f3-187a-4583-bbb2-6e8064cc948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"
Oct 30 01:41:37.907084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.906939    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a361f3-187a-4583-bbb2-6e8064cc948b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52dd0bed-84qvv\" (UID: \"84a361f3-187a-4583-bbb2-6e8064cc948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"
Oct 30 01:41:37.907132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.907100    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfzpk\" (UniqueName: \"kubernetes.io/projected/84a361f3-187a-4583-bbb2-6e8064cc948b-kube-api-access-tfzpk\") pod \"maas-api-revoke-sub-52dd0bed-84qvv\" (UID: \"84a361f3-187a-4583-bbb2-6e8064cc948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"
Oct 30 01:41:37.907636 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.907612    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a361f3-187a-4583-bbb2-6e8064cc948b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52dd0bed-84qvv\" (UID: \"84a361f3-187a-4583-bbb2-6e8064cc948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"
Oct 30 01:41:37.909498 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.909475    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a361f3-187a-4583-bbb2-6e8064cc948b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52dd0bed-84qvv\" (UID: \"84a361f3-187a-4583-bbb2-6e8064cc948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"
Oct 30 01:41:37.928626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.928589    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfzpk\" (UniqueName: \"kubernetes.io/projected/84a361f3-187a-4583-bbb2-6e8064cc948b-kube-api-access-tfzpk\") pod \"maas-api-revoke-sub-52dd0bed-84qvv\" (UID: \"84a361f3-187a-4583-bbb2-6e8064cc948b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"
Oct 30 01:41:37.976136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:37.976105    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"
Oct 30 01:41:38.126587 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.126541    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq" podUID="9c9778fb-1954-4c6a-a3d9-fc1cb633045b" containerName="main" containerID="cri-o://a38309589aa3bb6d40125d0939154db52678962e336647e0bdf129144d220910" gracePeriod=45
Oct 30 01:41:38.241950 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.241862    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq" podUID="9c9778fb-1954-4c6a-a3d9-fc1cb633045b" containerName="main" probeResult="failure" output="Get \"https://10.133.1.84:8000/ready\": dial tcp 10.133.1.84:8000: connect: connection refused"
Oct 30 01:41:38.290424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.290203    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"]
Oct 30 01:41:38.296000 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:41:38.295960    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a361f3_187a_4583_bbb2_6e8064cc948b.slice/crio-9220c618ed8476df0c782f1d66e7a2a44ada91f87272bbbff547aa2e5e1b2e54 WatchSource:0}: Error finding container 9220c618ed8476df0c782f1d66e7a2a44ada91f87272bbbff547aa2e5e1b2e54: Status 404 returned error can't find the container with id 9220c618ed8476df0c782f1d66e7a2a44ada91f87272bbbff547aa2e5e1b2e54
Oct 30 01:41:38.400812 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.400784    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:41:38.517667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.517623    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-kube-api-access-sfmtx\" (UniqueName: \"kubernetes.io/projected/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-kube-api-access-sfmtx\") pod \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") "
Oct 30 01:41:38.517953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.517932    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-dshm\") pod \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") "
Oct 30 01:41:38.518016 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.518009    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-model-cache\") pod \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") "
Oct 30 01:41:38.518240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.518219    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-home\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-home\") pod \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") "
Oct 30 01:41:38.518361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.518263    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-tls-certs\") pod \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") "
Oct 30 01:41:38.518429 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.518353    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-model-cache" pod "9c9778fb-1954-4c6a-a3d9-fc1cb633045b" (UID: "9c9778fb-1954-4c6a-a3d9-fc1cb633045b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:38.518429 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.518371    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-tmp-dir\") pod \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\" (UID: \"9c9778fb-1954-4c6a-a3d9-fc1cb633045b\") "
Oct 30 01:41:38.518540 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.518512    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-home" pod "9c9778fb-1954-4c6a-a3d9-fc1cb633045b" (UID: "9c9778fb-1954-4c6a-a3d9-fc1cb633045b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:38.518890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.518860    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-tmp-dir" pod "9c9778fb-1954-4c6a-a3d9-fc1cb633045b" (UID: "9c9778fb-1954-4c6a-a3d9-fc1cb633045b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:38.519901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.519878    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:38.520019 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.519972    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:38.520019 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.519992    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:38.550489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.550454    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-dshm" pod "9c9778fb-1954-4c6a-a3d9-fc1cb633045b" (UID: "9c9778fb-1954-4c6a-a3d9-fc1cb633045b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:38.550746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.550495    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-tls-certs" pod "9c9778fb-1954-4c6a-a3d9-fc1cb633045b" (UID: "9c9778fb-1954-4c6a-a3d9-fc1cb633045b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:41:38.550746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.550509    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-kube-api-access-sfmtx" pod "9c9778fb-1954-4c6a-a3d9-fc1cb633045b" (UID: "9c9778fb-1954-4c6a-a3d9-fc1cb633045b"). InnerVolumeSpecName "kube-api-access-sfmtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:41:38.621116 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.621083    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:38.621116 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.621113    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfmtx\" (UniqueName: \"kubernetes.io/projected/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-kube-api-access-sfmtx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:38.621116 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:38.621124    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c9778fb-1954-4c6a-a3d9-fc1cb633045b-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:39.104866 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:39.104827    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv" event={"ID":"84a361f3-187a-4583-bbb2-6e8064cc948b","Type":"ContainerStarted","Data":"ee8975ce0b229cd945eea694535d0a4d033a88e4f020470b3916c67b4783af22"}
Oct 30 01:41:39.105071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:39.104891    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv" event={"ID":"84a361f3-187a-4583-bbb2-6e8064cc948b","Type":"ContainerStarted","Data":"9220c618ed8476df0c782f1d66e7a2a44ada91f87272bbbff547aa2e5e1b2e54"}
Oct 30 01:41:39.106427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:39.106399    4425 generic.go:356] "Generic (PLEG): container finished" podID="9c9778fb-1954-4c6a-a3d9-fc1cb633045b" containerID="a38309589aa3bb6d40125d0939154db52678962e336647e0bdf129144d220910" exitCode=0
Oct 30 01:41:39.106553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:39.106439    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"
Oct 30 01:41:39.106553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:39.106473    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq" event={"ID":"9c9778fb-1954-4c6a-a3d9-fc1cb633045b","Type":"ContainerDied","Data":"a38309589aa3bb6d40125d0939154db52678962e336647e0bdf129144d220910"}
Oct 30 01:41:39.106553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:39.106514    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq" event={"ID":"9c9778fb-1954-4c6a-a3d9-fc1cb633045b","Type":"ContainerDied","Data":"8e6bc799050ed54557b91c0a4cac9343a0a2251b43bc1e07ff934601028acd89"}
Oct 30 01:41:39.106553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:39.106533    4425 scope.go:122] "RemoveContainer" containerID="a38309589aa3bb6d40125d0939154db52678962e336647e0bdf129144d220910"
Oct 30 01:41:39.130556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:39.129213    4425 scope.go:122] "RemoveContainer" containerID="a38309589aa3bb6d40125d0939154db52678962e336647e0bdf129144d220910"
Oct 30 01:41:39.130556 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:41:39.130453    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38309589aa3bb6d40125d0939154db52678962e336647e0bdf129144d220910\": container with ID starting with a38309589aa3bb6d40125d0939154db52678962e336647e0bdf129144d220910 not found: ID does not exist" containerID="a38309589aa3bb6d40125d0939154db52678962e336647e0bdf129144d220910"
Oct 30 01:41:39.130556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:39.130490    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38309589aa3bb6d40125d0939154db52678962e336647e0bdf129144d220910"} err="failed to get container status \"a38309589aa3bb6d40125d0939154db52678962e336647e0bdf129144d220910\": rpc error: code = NotFound desc = could not find container \"a38309589aa3bb6d40125d0939154db52678962e336647e0bdf129144d220910\": container with ID starting with a38309589aa3bb6d40125d0939154db52678962e336647e0bdf129144d220910 not found: ID does not exist"
Oct 30 01:41:39.154049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:39.153993    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv" podStartSLOduration=2.153977467 podStartE2EDuration="2.153977467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:39.152095455 +0000 UTC m=+3607.901750758" watchObservedRunningTime="2026-10-30 01:41:39.153977467 +0000 UTC m=+3607.903632776"
Oct 30 01:41:39.187656 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:39.187624    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"]
Oct 30 01:41:39.192777 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:39.192745    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bd287f/test-model-bd287f-kserve-7ffc54564f-wzgrq"]
Oct 30 01:41:39.427100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:39.426977    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c9778fb-1954-4c6a-a3d9-fc1cb633045b" path="/var/lib/kubelet/pods/9c9778fb-1954-4c6a-a3d9-fc1cb633045b/volumes"
Oct 30 01:41:40.118977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:40.118941    4425 generic.go:356] "Generic (PLEG): container finished" podID="84a361f3-187a-4583-bbb2-6e8064cc948b" containerID="ee8975ce0b229cd945eea694535d0a4d033a88e4f020470b3916c67b4783af22" exitCode=0
Oct 30 01:41:40.119535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:40.119112    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv" event={"ID":"84a361f3-187a-4583-bbb2-6e8064cc948b","Type":"ContainerDied","Data":"ee8975ce0b229cd945eea694535d0a4d033a88e4f020470b3916c67b4783af22"}
Oct 30 01:41:40.232240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:40.232176    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"]
Oct 30 01:41:41.315496 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:41.315469    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"
Oct 30 01:41:41.474645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:41.474557    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a361f3-187a-4583-bbb2-6e8064cc948b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a361f3-187a-4583-bbb2-6e8064cc948b-maas-api-openshift-ca\") pod \"84a361f3-187a-4583-bbb2-6e8064cc948b\" (UID: \"84a361f3-187a-4583-bbb2-6e8064cc948b\") "
Oct 30 01:41:41.474835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:41.474663    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a361f3-187a-4583-bbb2-6e8064cc948b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a361f3-187a-4583-bbb2-6e8064cc948b-maas-api-xks-ca\") pod \"84a361f3-187a-4583-bbb2-6e8064cc948b\" (UID: \"84a361f3-187a-4583-bbb2-6e8064cc948b\") "
Oct 30 01:41:41.474835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:41.474692    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a361f3-187a-4583-bbb2-6e8064cc948b-kube-api-access-tfzpk\" (UniqueName: \"kubernetes.io/projected/84a361f3-187a-4583-bbb2-6e8064cc948b-kube-api-access-tfzpk\") pod \"84a361f3-187a-4583-bbb2-6e8064cc948b\" (UID: \"84a361f3-187a-4583-bbb2-6e8064cc948b\") "
Oct 30 01:41:41.475043 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:41.475010    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a361f3-187a-4583-bbb2-6e8064cc948b-maas-api-openshift-ca" pod "84a361f3-187a-4583-bbb2-6e8064cc948b" (UID: "84a361f3-187a-4583-bbb2-6e8064cc948b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:41:41.475505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:41.475484    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a361f3-187a-4583-bbb2-6e8064cc948b-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:41.476904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:41.476881    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a361f3-187a-4583-bbb2-6e8064cc948b-maas-api-xks-ca" pod "84a361f3-187a-4583-bbb2-6e8064cc948b" (UID: "84a361f3-187a-4583-bbb2-6e8064cc948b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:41:41.477032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:41.477010    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a361f3-187a-4583-bbb2-6e8064cc948b-kube-api-access-tfzpk" pod "84a361f3-187a-4583-bbb2-6e8064cc948b" (UID: "84a361f3-187a-4583-bbb2-6e8064cc948b"). InnerVolumeSpecName "kube-api-access-tfzpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:41:41.576608 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:41.576569    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a361f3-187a-4583-bbb2-6e8064cc948b-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:41.576608 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:41.576601    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfzpk\" (UniqueName: \"kubernetes.io/projected/84a361f3-187a-4583-bbb2-6e8064cc948b-kube-api-access-tfzpk\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:42.137687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:42.137650    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv" event={"ID":"84a361f3-187a-4583-bbb2-6e8064cc948b","Type":"ContainerDied","Data":"9220c618ed8476df0c782f1d66e7a2a44ada91f87272bbbff547aa2e5e1b2e54"}
Oct 30 01:41:42.137687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:42.137675    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"
Oct 30 01:41:42.137922 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:42.137713    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9220c618ed8476df0c782f1d66e7a2a44ada91f87272bbbff547aa2e5e1b2e54"
Oct 30 01:41:51.015326 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:51.015286    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"]
Oct 30 01:41:51.045168 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:51.045119    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"]
Oct 30 01:41:51.886708 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:51.886663    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"]
Oct 30 01:41:51.886948 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:51.886914    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw" podUID="02874d4b-fe79-4247-b420-369003cd1d23" containerName="istio-proxy" containerID="cri-o://2bc7d679e22cd00617cc44577ee880cb0eb20214e4832d00ecf46da9811675ce" gracePeriod=30
Oct 30 01:41:54.083625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.083569    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"]
Oct 30 01:41:54.085023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.084968    4425 cpu_manager.go:414] "containerMap: removing container" podUID="84a361f3-187a-4583-bbb2-6e8064cc948b" containerName="revoke-keys"
Oct 30 01:41:54.085023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.084991    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a361f3-187a-4583-bbb2-6e8064cc948b" containerName="revoke-keys"
Oct 30 01:41:54.085330 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.085035    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9c9778fb-1954-4c6a-a3d9-fc1cb633045b" containerName="main"
Oct 30 01:41:54.085330 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.085043    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9778fb-1954-4c6a-a3d9-fc1cb633045b" containerName="main"
Oct 30 01:41:54.085330 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.085265    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a361f3-187a-4583-bbb2-6e8064cc948b" containerName="revoke-keys"
Oct 30 01:41:54.085330 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.085285    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9778fb-1954-4c6a-a3d9-fc1cb633045b" containerName="main"
Oct 30 01:41:54.091442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.091418    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"
Oct 30 01:41:54.111803 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.111775    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:41:54.131179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.127799    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:41:54.134786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.134746    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"]
Oct 30 01:41:54.175318 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.175268    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4wdv\" (UniqueName: \"kubernetes.io/projected/4ebba8f1-f202-41c9-b321-559dcfad5c72-kube-api-access-f4wdv\") pod \"maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj\" (UID: \"4ebba8f1-f202-41c9-b321-559dcfad5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"
Oct 30 01:41:54.175553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.175529    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebba8f1-f202-41c9-b321-559dcfad5c72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj\" (UID: \"4ebba8f1-f202-41c9-b321-559dcfad5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"
Oct 30 01:41:54.175749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.175729    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebba8f1-f202-41c9-b321-559dcfad5c72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj\" (UID: \"4ebba8f1-f202-41c9-b321-559dcfad5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"
Oct 30 01:41:54.276915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.276881    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebba8f1-f202-41c9-b321-559dcfad5c72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj\" (UID: \"4ebba8f1-f202-41c9-b321-559dcfad5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"
Oct 30 01:41:54.277123 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.276936    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebba8f1-f202-41c9-b321-559dcfad5c72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj\" (UID: \"4ebba8f1-f202-41c9-b321-559dcfad5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"
Oct 30 01:41:54.277265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.277123    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4wdv\" (UniqueName: \"kubernetes.io/projected/4ebba8f1-f202-41c9-b321-559dcfad5c72-kube-api-access-f4wdv\") pod \"maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj\" (UID: \"4ebba8f1-f202-41c9-b321-559dcfad5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"
Oct 30 01:41:54.277861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.277831    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebba8f1-f202-41c9-b321-559dcfad5c72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj\" (UID: \"4ebba8f1-f202-41c9-b321-559dcfad5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"
Oct 30 01:41:54.279447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.279429    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebba8f1-f202-41c9-b321-559dcfad5c72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj\" (UID: \"4ebba8f1-f202-41c9-b321-559dcfad5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"
Oct 30 01:41:54.303259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.303229    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4wdv\" (UniqueName: \"kubernetes.io/projected/4ebba8f1-f202-41c9-b321-559dcfad5c72-kube-api-access-f4wdv\") pod \"maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj\" (UID: \"4ebba8f1-f202-41c9-b321-559dcfad5c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"
Oct 30 01:41:54.410648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.410551    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"
Oct 30 01:41:54.739938 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:54.739902    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"]
Oct 30 01:41:54.740943 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:41:54.740910    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebba8f1_f202_41c9_b321_559dcfad5c72.slice/crio-e3276fa92474658fe1cca2238ae39fd85c7eb6ba2e8d19300281c70dff1b5534 WatchSource:0}: Error finding container e3276fa92474658fe1cca2238ae39fd85c7eb6ba2e8d19300281c70dff1b5534: Status 404 returned error can't find the container with id e3276fa92474658fe1cca2238ae39fd85c7eb6ba2e8d19300281c70dff1b5534
Oct 30 01:41:55.255236 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.255193    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj" event={"ID":"4ebba8f1-f202-41c9-b321-559dcfad5c72","Type":"ContainerStarted","Data":"e3276fa92474658fe1cca2238ae39fd85c7eb6ba2e8d19300281c70dff1b5534"}
Oct 30 01:41:55.258457 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.258428    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl" podUID="5f308cb4-16bb-4c79-89da-3ad7f2633de8" containerName="main" containerID="cri-o://4842c9916e6250168fdfd81bd269328e55cee8f869c82a2647a42184dfadc93d" gracePeriod=45
Oct 30 01:41:55.367003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.366964    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl" podUID="5f308cb4-16bb-4c79-89da-3ad7f2633de8" containerName="main" probeResult="failure" output="Get \"https://10.133.1.85:8000/ready\": dial tcp 10.133.1.85:8000: connect: connection refused"
Oct 30 01:41:55.550596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.550567    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:41:55.698967 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.698931    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-dshm\") pod \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") "
Oct 30 01:41:55.699147 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.699032    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f308cb4-16bb-4c79-89da-3ad7f2633de8-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f308cb4-16bb-4c79-89da-3ad7f2633de8-tls-certs\") pod \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") "
Oct 30 01:41:55.699147 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.699054    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-tmp-dir\") pod \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") "
Oct 30 01:41:55.699284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.699171    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f308cb4-16bb-4c79-89da-3ad7f2633de8-kube-api-access-9jsxf\" (UniqueName: \"kubernetes.io/projected/5f308cb4-16bb-4c79-89da-3ad7f2633de8-kube-api-access-9jsxf\") pod \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") "
Oct 30 01:41:55.699284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.699238    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-model-cache\") pod \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") "
Oct 30 01:41:55.699284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.699269    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-home\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-home\") pod \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\" (UID: \"5f308cb4-16bb-4c79-89da-3ad7f2633de8\") "
Oct 30 01:41:55.699452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.699377    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-tmp-dir" pod "5f308cb4-16bb-4c79-89da-3ad7f2633de8" (UID: "5f308cb4-16bb-4c79-89da-3ad7f2633de8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:55.699585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.699561    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-model-cache" pod "5f308cb4-16bb-4c79-89da-3ad7f2633de8" (UID: "5f308cb4-16bb-4c79-89da-3ad7f2633de8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:55.699714 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.699580    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-home" pod "5f308cb4-16bb-4c79-89da-3ad7f2633de8" (UID: "5f308cb4-16bb-4c79-89da-3ad7f2633de8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:55.700069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.700047    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:55.700118 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.700075    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:55.700118 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.700090    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:55.701286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.701262    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-dshm" pod "5f308cb4-16bb-4c79-89da-3ad7f2633de8" (UID: "5f308cb4-16bb-4c79-89da-3ad7f2633de8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:55.701437 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.701418    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f308cb4-16bb-4c79-89da-3ad7f2633de8-kube-api-access-9jsxf" pod "5f308cb4-16bb-4c79-89da-3ad7f2633de8" (UID: "5f308cb4-16bb-4c79-89da-3ad7f2633de8"). InnerVolumeSpecName "kube-api-access-9jsxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:41:55.701480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.701442    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f308cb4-16bb-4c79-89da-3ad7f2633de8-tls-certs" pod "5f308cb4-16bb-4c79-89da-3ad7f2633de8" (UID: "5f308cb4-16bb-4c79-89da-3ad7f2633de8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:41:55.801215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.801103    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jsxf\" (UniqueName: \"kubernetes.io/projected/5f308cb4-16bb-4c79-89da-3ad7f2633de8-kube-api-access-9jsxf\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:55.801215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.801137    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f308cb4-16bb-4c79-89da-3ad7f2633de8-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:55.801215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:55.801150    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f308cb4-16bb-4c79-89da-3ad7f2633de8-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:56.016226 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:56.016180    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh" podUID="29432241-060e-48e7-bb45-578a8391964b" containerName="payload-processing" containerID="cri-o://a5346c749393159b6e5526971cd495b693553885d1d391e7a79f750fcecf97d3" gracePeriod=25
Oct 30 01:41:56.046553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:56.046507    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js" podUID="863be03f-aa38-4aeb-8f71-9ebd0685c391" containerName="payload-pre-processing" containerID="cri-o://a4564e34ba0fff98d71b70936012c2e1545ea8a243a4c9f6e1f8b2d6f6423e8d" gracePeriod=25
Oct 30 01:41:56.264232 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:56.264201    4425 generic.go:356] "Generic (PLEG): container finished" podID="5f308cb4-16bb-4c79-89da-3ad7f2633de8" containerID="4842c9916e6250168fdfd81bd269328e55cee8f869c82a2647a42184dfadc93d" exitCode=0
Oct 30 01:41:56.264653 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:56.264255    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"
Oct 30 01:41:56.264653 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:56.264281    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl" event={"ID":"5f308cb4-16bb-4c79-89da-3ad7f2633de8","Type":"ContainerDied","Data":"4842c9916e6250168fdfd81bd269328e55cee8f869c82a2647a42184dfadc93d"}
Oct 30 01:41:56.264653 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:56.264322    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl" event={"ID":"5f308cb4-16bb-4c79-89da-3ad7f2633de8","Type":"ContainerDied","Data":"eb06837a9e35d6852321eed07719dca172c1586cd73c8e83a289e19514174a6d"}
Oct 30 01:41:56.264653 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:56.264343    4425 scope.go:122] "RemoveContainer" containerID="4842c9916e6250168fdfd81bd269328e55cee8f869c82a2647a42184dfadc93d"
Oct 30 01:41:56.266461 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:56.266432    4425 generic.go:356] "Generic (PLEG): container finished" podID="4ebba8f1-f202-41c9-b321-559dcfad5c72" containerID="736256b3309b6bd077804159ed73579592374a8a3b0065390e919e0300acd3c9" exitCode=0
Oct 30 01:41:56.266645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:56.266517    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj" event={"ID":"4ebba8f1-f202-41c9-b321-559dcfad5c72","Type":"ContainerDied","Data":"736256b3309b6bd077804159ed73579592374a8a3b0065390e919e0300acd3c9"}
Oct 30 01:41:56.276090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:56.276071    4425 scope.go:122] "RemoveContainer" containerID="4842c9916e6250168fdfd81bd269328e55cee8f869c82a2647a42184dfadc93d"
Oct 30 01:41:56.276415 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:41:56.276387    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4842c9916e6250168fdfd81bd269328e55cee8f869c82a2647a42184dfadc93d\": container with ID starting with 4842c9916e6250168fdfd81bd269328e55cee8f869c82a2647a42184dfadc93d not found: ID does not exist" containerID="4842c9916e6250168fdfd81bd269328e55cee8f869c82a2647a42184dfadc93d"
Oct 30 01:41:56.276470 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:56.276424    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4842c9916e6250168fdfd81bd269328e55cee8f869c82a2647a42184dfadc93d"} err="failed to get container status \"4842c9916e6250168fdfd81bd269328e55cee8f869c82a2647a42184dfadc93d\": rpc error: code = NotFound desc = could not find container \"4842c9916e6250168fdfd81bd269328e55cee8f869c82a2647a42184dfadc93d\": container with ID starting with 4842c9916e6250168fdfd81bd269328e55cee8f869c82a2647a42184dfadc93d not found: ID does not exist"
Oct 30 01:41:56.349238 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:56.349200    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"]
Oct 30 01:41:56.356100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:56.356067    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-190407/test-model-190407-kserve-b9766996f-9tbcl"]
Oct 30 01:41:57.189142 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.189110    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:41:57.279045 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.279013    4425 generic.go:356] "Generic (PLEG): container finished" podID="02874d4b-fe79-4247-b420-369003cd1d23" containerID="2bc7d679e22cd00617cc44577ee880cb0eb20214e4832d00ecf46da9811675ce" exitCode=0
Oct 30 01:41:57.279563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.279063    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"
Oct 30 01:41:57.279563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.279111    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw" event={"ID":"02874d4b-fe79-4247-b420-369003cd1d23","Type":"ContainerDied","Data":"2bc7d679e22cd00617cc44577ee880cb0eb20214e4832d00ecf46da9811675ce"}
Oct 30 01:41:57.279563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.279207    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw" event={"ID":"02874d4b-fe79-4247-b420-369003cd1d23","Type":"ContainerDied","Data":"95a5b446d73ea76228182068bab0dfe163ad76eb844dfcfc32e7571ba27bc141"}
Oct 30 01:41:57.279563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.279239    4425 scope.go:122] "RemoveContainer" containerID="2bc7d679e22cd00617cc44577ee880cb0eb20214e4832d00ecf46da9811675ce"
Oct 30 01:41:57.293460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.293435    4425 scope.go:122] "RemoveContainer" containerID="2bc7d679e22cd00617cc44577ee880cb0eb20214e4832d00ecf46da9811675ce"
Oct 30 01:41:57.293808 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:41:57.293777    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc7d679e22cd00617cc44577ee880cb0eb20214e4832d00ecf46da9811675ce\": container with ID starting with 2bc7d679e22cd00617cc44577ee880cb0eb20214e4832d00ecf46da9811675ce not found: ID does not exist" containerID="2bc7d679e22cd00617cc44577ee880cb0eb20214e4832d00ecf46da9811675ce"
Oct 30 01:41:57.293923 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.293818    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc7d679e22cd00617cc44577ee880cb0eb20214e4832d00ecf46da9811675ce"} err="failed to get container status \"2bc7d679e22cd00617cc44577ee880cb0eb20214e4832d00ecf46da9811675ce\": rpc error: code = NotFound desc = could not find container \"2bc7d679e22cd00617cc44577ee880cb0eb20214e4832d00ecf46da9811675ce\": container with ID starting with 2bc7d679e22cd00617cc44577ee880cb0eb20214e4832d00ecf46da9811675ce not found: ID does not exist"
Oct 30 01:41:57.341908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.341869    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-istio-data\") pod \"02874d4b-fe79-4247-b420-369003cd1d23\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") "
Oct 30 01:41:57.342107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.341976    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02874d4b-fe79-4247-b420-369003cd1d23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02874d4b-fe79-4247-b420-369003cd1d23-istio-podinfo\") pod \"02874d4b-fe79-4247-b420-369003cd1d23\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") "
Oct 30 01:41:57.342107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.342044    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-istio-envoy\") pod \"02874d4b-fe79-4247-b420-369003cd1d23\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") "
Oct 30 01:41:57.342107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.342097    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02874d4b-fe79-4247-b420-369003cd1d23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02874d4b-fe79-4247-b420-369003cd1d23-istiod-ca-cert\") pod \"02874d4b-fe79-4247-b420-369003cd1d23\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") "
Oct 30 01:41:57.342336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.342231    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02874d4b-fe79-4247-b420-369003cd1d23-istio-token\" (UniqueName: \"kubernetes.io/projected/02874d4b-fe79-4247-b420-369003cd1d23-istio-token\") pod \"02874d4b-fe79-4247-b420-369003cd1d23\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") "
Oct 30 01:41:57.342336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.342269    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02874d4b-fe79-4247-b420-369003cd1d23-kube-api-access-tlfjj\" (UniqueName: \"kubernetes.io/projected/02874d4b-fe79-4247-b420-369003cd1d23-kube-api-access-tlfjj\") pod \"02874d4b-fe79-4247-b420-369003cd1d23\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") "
Oct 30 01:41:57.342456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.342332    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-workload-socket\") pod \"02874d4b-fe79-4247-b420-369003cd1d23\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") "
Oct 30 01:41:57.342456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.342383    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-istio-data" pod "02874d4b-fe79-4247-b420-369003cd1d23" (UID: "02874d4b-fe79-4247-b420-369003cd1d23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:57.342456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.342424    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-credential-socket\") pod \"02874d4b-fe79-4247-b420-369003cd1d23\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") "
Oct 30 01:41:57.342626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.342530    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-workload-certs\") pod \"02874d4b-fe79-4247-b420-369003cd1d23\" (UID: \"02874d4b-fe79-4247-b420-369003cd1d23\") "
Oct 30 01:41:57.343240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.343215    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:57.343492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.343457    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-workload-certs" pod "02874d4b-fe79-4247-b420-369003cd1d23" (UID: "02874d4b-fe79-4247-b420-369003cd1d23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:57.343658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.343606    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-credential-socket" pod "02874d4b-fe79-4247-b420-369003cd1d23" (UID: "02874d4b-fe79-4247-b420-369003cd1d23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:57.343845 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.343821    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02874d4b-fe79-4247-b420-369003cd1d23-istiod-ca-cert" pod "02874d4b-fe79-4247-b420-369003cd1d23" (UID: "02874d4b-fe79-4247-b420-369003cd1d23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:41:57.344230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.344206    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-workload-socket" pod "02874d4b-fe79-4247-b420-369003cd1d23" (UID: "02874d4b-fe79-4247-b420-369003cd1d23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:57.345012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.344986    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02874d4b-fe79-4247-b420-369003cd1d23-istio-podinfo" pod "02874d4b-fe79-4247-b420-369003cd1d23" (UID: "02874d4b-fe79-4247-b420-369003cd1d23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:41:57.346587 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.346521    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02874d4b-fe79-4247-b420-369003cd1d23-kube-api-access-tlfjj" pod "02874d4b-fe79-4247-b420-369003cd1d23" (UID: "02874d4b-fe79-4247-b420-369003cd1d23"). InnerVolumeSpecName "kube-api-access-tlfjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:41:57.346829 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.346796    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-istio-envoy" pod "02874d4b-fe79-4247-b420-369003cd1d23" (UID: "02874d4b-fe79-4247-b420-369003cd1d23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:41:57.346909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.346861    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02874d4b-fe79-4247-b420-369003cd1d23-istio-token" pod "02874d4b-fe79-4247-b420-369003cd1d23" (UID: "02874d4b-fe79-4247-b420-369003cd1d23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:41:57.429522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.429422    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f308cb4-16bb-4c79-89da-3ad7f2633de8" path="/var/lib/kubelet/pods/5f308cb4-16bb-4c79-89da-3ad7f2633de8/volumes"
Oct 30 01:41:57.445046 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.445010    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:57.445046 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.445045    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02874d4b-fe79-4247-b420-369003cd1d23-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:57.445306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.445060    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:57.445306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.445073    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02874d4b-fe79-4247-b420-369003cd1d23-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:57.445306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.445086    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02874d4b-fe79-4247-b420-369003cd1d23-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:57.445306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.445110    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlfjj\" (UniqueName: \"kubernetes.io/projected/02874d4b-fe79-4247-b420-369003cd1d23-kube-api-access-tlfjj\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:57.445306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.445123    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:57.445306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.445136    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02874d4b-fe79-4247-b420-369003cd1d23-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:57.481296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.481269    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"
Oct 30 01:41:57.617856 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.617826    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"]
Oct 30 01:41:57.622351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.622324    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-rb6rw"]
Oct 30 01:41:57.647926 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.647900    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ebba8f1-f202-41c9-b321-559dcfad5c72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebba8f1-f202-41c9-b321-559dcfad5c72-maas-api-xks-ca\") pod \"4ebba8f1-f202-41c9-b321-559dcfad5c72\" (UID: \"4ebba8f1-f202-41c9-b321-559dcfad5c72\") "
Oct 30 01:41:57.648107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.648087    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ebba8f1-f202-41c9-b321-559dcfad5c72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebba8f1-f202-41c9-b321-559dcfad5c72-maas-api-openshift-ca\") pod \"4ebba8f1-f202-41c9-b321-559dcfad5c72\" (UID: \"4ebba8f1-f202-41c9-b321-559dcfad5c72\") "
Oct 30 01:41:57.648209 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.648190    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ebba8f1-f202-41c9-b321-559dcfad5c72-kube-api-access-f4wdv\" (UniqueName: \"kubernetes.io/projected/4ebba8f1-f202-41c9-b321-559dcfad5c72-kube-api-access-f4wdv\") pod \"4ebba8f1-f202-41c9-b321-559dcfad5c72\" (UID: \"4ebba8f1-f202-41c9-b321-559dcfad5c72\") "
Oct 30 01:41:57.648487 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.648463    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebba8f1-f202-41c9-b321-559dcfad5c72-maas-api-openshift-ca" pod "4ebba8f1-f202-41c9-b321-559dcfad5c72" (UID: "4ebba8f1-f202-41c9-b321-559dcfad5c72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:41:57.648676 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.648658    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebba8f1-f202-41c9-b321-559dcfad5c72-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:57.650141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.650120    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ebba8f1-f202-41c9-b321-559dcfad5c72-maas-api-xks-ca" pod "4ebba8f1-f202-41c9-b321-559dcfad5c72" (UID: "4ebba8f1-f202-41c9-b321-559dcfad5c72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:41:57.650350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.650329    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebba8f1-f202-41c9-b321-559dcfad5c72-kube-api-access-f4wdv" pod "4ebba8f1-f202-41c9-b321-559dcfad5c72" (UID: "4ebba8f1-f202-41c9-b321-559dcfad5c72"). InnerVolumeSpecName "kube-api-access-f4wdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:41:57.750022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.749926    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4wdv\" (UniqueName: \"kubernetes.io/projected/4ebba8f1-f202-41c9-b321-559dcfad5c72-kube-api-access-f4wdv\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:57.750022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:57.749958    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebba8f1-f202-41c9-b321-559dcfad5c72-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:41:58.291022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:58.290987    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"
Oct 30 01:41:58.291022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:58.290995    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj" event={"ID":"4ebba8f1-f202-41c9-b321-559dcfad5c72","Type":"ContainerDied","Data":"e3276fa92474658fe1cca2238ae39fd85c7eb6ba2e8d19300281c70dff1b5534"}
Oct 30 01:41:58.291597 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:58.291069    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3276fa92474658fe1cca2238ae39fd85c7eb6ba2e8d19300281c70dff1b5534"
Oct 30 01:41:59.427484 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:41:59.427453    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02874d4b-fe79-4247-b420-369003cd1d23" path="/var/lib/kubelet/pods/02874d4b-fe79-4247-b420-369003cd1d23/volumes"
Oct 30 01:42:00.505480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:00.505442    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"]
Oct 30 01:42:00.505973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:00.505708    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw" podUID="1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe" containerName="maas-api" containerID="cri-o://51b6e1817bedcc27b730e8380c435a21ec12edf56e225bccbfc74c9f47262a6a" gracePeriod=30
Oct 30 01:42:01.813024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:01.812960    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:42:01.906722 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:01.906687    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-maas-api-tls\") pod \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\" (UID: \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\") "
Oct 30 01:42:01.907047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:01.907027    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-metrics-certs\") pod \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\" (UID: \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\") "
Oct 30 01:42:01.907242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:01.907226    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-kube-api-access-npn9b\" (UniqueName: \"kubernetes.io/projected/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-kube-api-access-npn9b\") pod \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\" (UID: \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\") "
Oct 30 01:42:01.907401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:01.907385    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-openshift-service-ca\") pod \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\" (UID: \"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe\") "
Oct 30 01:42:01.908104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:01.908078    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-openshift-service-ca" pod "1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe" (UID: "1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:42:01.915348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:01.915315    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-maas-api-tls" pod "1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe" (UID: "1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:42:01.915736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:01.915704    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-metrics-certs" pod "1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe" (UID: "1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:42:01.915827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:01.915776    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-kube-api-access-npn9b" pod "1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe" (UID: "1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe"). InnerVolumeSpecName "kube-api-access-npn9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:02.009006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:02.008962    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-maas-api-tls\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:02.009006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:02.008995    4425 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-metrics-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:02.009006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:02.009009    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npn9b\" (UniqueName: \"kubernetes.io/projected/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-kube-api-access-npn9b\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:02.009348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:02.009024    4425 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe-openshift-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:02.327080 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:02.327037    4425 generic.go:356] "Generic (PLEG): container finished" podID="1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe" containerID="51b6e1817bedcc27b730e8380c435a21ec12edf56e225bccbfc74c9f47262a6a" exitCode=0
Oct 30 01:42:02.327080 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:02.327082    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"
Oct 30 01:42:02.327345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:02.327114    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw" event={"ID":"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe","Type":"ContainerDied","Data":"51b6e1817bedcc27b730e8380c435a21ec12edf56e225bccbfc74c9f47262a6a"}
Oct 30 01:42:02.327345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:02.327198    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw" event={"ID":"1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe","Type":"ContainerDied","Data":"e28479b1674e020d1b18472185d04bb4a6d50674916088b47ae3116bc21a4b6c"}
Oct 30 01:42:02.327345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:02.327224    4425 scope.go:122] "RemoveContainer" containerID="51b6e1817bedcc27b730e8380c435a21ec12edf56e225bccbfc74c9f47262a6a"
Oct 30 01:42:02.341090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:02.341061    4425 scope.go:122] "RemoveContainer" containerID="51b6e1817bedcc27b730e8380c435a21ec12edf56e225bccbfc74c9f47262a6a"
Oct 30 01:42:02.341459 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:42:02.341433    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51b6e1817bedcc27b730e8380c435a21ec12edf56e225bccbfc74c9f47262a6a\": container with ID starting with 51b6e1817bedcc27b730e8380c435a21ec12edf56e225bccbfc74c9f47262a6a not found: ID does not exist" containerID="51b6e1817bedcc27b730e8380c435a21ec12edf56e225bccbfc74c9f47262a6a"
Oct 30 01:42:02.341548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:02.341470    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51b6e1817bedcc27b730e8380c435a21ec12edf56e225bccbfc74c9f47262a6a"} err="failed to get container status \"51b6e1817bedcc27b730e8380c435a21ec12edf56e225bccbfc74c9f47262a6a\": rpc error: code = NotFound desc = could not find container \"51b6e1817bedcc27b730e8380c435a21ec12edf56e225bccbfc74c9f47262a6a\": container with ID starting with 51b6e1817bedcc27b730e8380c435a21ec12edf56e225bccbfc74c9f47262a6a not found: ID does not exist"
Oct 30 01:42:02.377545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:02.377506    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"]
Oct 30 01:42:02.386589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:02.386546    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bd287f-8b8fd4f97-m5gfw"]
Oct 30 01:42:03.426427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:03.426391    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe" path="/var/lib/kubelet/pods/1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe/volumes"
Oct 30 01:42:04.507384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:04.507347    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"]
Oct 30 01:42:04.541088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:04.541043    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"]
Oct 30 01:42:05.937097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:05.937047    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js" podUID="863be03f-aa38-4aeb-8f71-9ebd0685c391" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:42:05.937567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:05.937061    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh" podUID="29432241-060e-48e7-bb45-578a8391964b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:42:06.937742 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:06.937703    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"]
Oct 30 01:42:06.938244 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:06.937970    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l" podUID="81dd80d6-4327-4622-937a-39c52afd0fef" containerName="istio-proxy" containerID="cri-o://a431e23bc8d7f2b885d05ebec4840e68539984606077db3234af4e3a8bd0bc56" gracePeriod=30
Oct 30 01:42:07.558428 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:42:07.558378    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:42:09.508117 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:09.508074    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq" podUID="11991a7b-d429-4f6e-a8a5-bc8352924d46" containerName="payload-processing" containerID="cri-o://e195f4fd8555d89721334a389cad5f9b24853ce17b405a88d880a7c892c1caee" gracePeriod=25
Oct 30 01:42:09.541844 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:09.541780    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979" podUID="01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9" containerName="payload-pre-processing" containerID="cri-o://61c877f0fd6203e13d36eee730dd3f4dbcf751de605fa15ae331ed8724095b46" gracePeriod=25
Oct 30 01:42:10.588003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.587965    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"]
Oct 30 01:42:10.588965 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.588944    4425 cpu_manager.go:414] "containerMap: removing container" podUID="1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe" containerName="maas-api"
Oct 30 01:42:10.589021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.588966    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe" containerName="maas-api"
Oct 30 01:42:10.589021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.588993    4425 cpu_manager.go:414] "containerMap: removing container" podUID="5f308cb4-16bb-4c79-89da-3ad7f2633de8" containerName="main"
Oct 30 01:42:10.589021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.588999    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f308cb4-16bb-4c79-89da-3ad7f2633de8" containerName="main"
Oct 30 01:42:10.589136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.589023    4425 cpu_manager.go:414] "containerMap: removing container" podUID="02874d4b-fe79-4247-b420-369003cd1d23" containerName="istio-proxy"
Oct 30 01:42:10.589136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.589028    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02874d4b-fe79-4247-b420-369003cd1d23" containerName="istio-proxy"
Oct 30 01:42:10.589136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.589036    4425 cpu_manager.go:414] "containerMap: removing container" podUID="4ebba8f1-f202-41c9-b321-559dcfad5c72" containerName="revoke-keys"
Oct 30 01:42:10.589136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.589041    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebba8f1-f202-41c9-b321-559dcfad5c72" containerName="revoke-keys"
Oct 30 01:42:10.589321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.589232    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="02874d4b-fe79-4247-b420-369003cd1d23" containerName="istio-proxy"
Oct 30 01:42:10.589321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.589244    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ebba8f1-f202-41c9-b321-559dcfad5c72" containerName="revoke-keys"
Oct 30 01:42:10.589321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.589260    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0419d9-c9bd-4c5a-962a-a8771b8e0fbe" containerName="maas-api"
Oct 30 01:42:10.589321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.589271    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f308cb4-16bb-4c79-89da-3ad7f2633de8" containerName="main"
Oct 30 01:42:10.593564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.593547    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"
Oct 30 01:42:10.611671 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.611636    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"]
Oct 30 01:42:10.662832 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.662799    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:42:10.663032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.662980    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:42:10.746568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.746531    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96dcf37-d552-45cf-9d9e-448af64f7595-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-190407-k9gwm\" (UID: \"a96dcf37-d552-45cf-9d9e-448af64f7595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"
Oct 30 01:42:10.746789 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.746768    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96dcf37-d552-45cf-9d9e-448af64f7595-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-190407-k9gwm\" (UID: \"a96dcf37-d552-45cf-9d9e-448af64f7595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"
Oct 30 01:42:10.746838 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.746811    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkdq6\" (UniqueName: \"kubernetes.io/projected/a96dcf37-d552-45cf-9d9e-448af64f7595-kube-api-access-xkdq6\") pod \"maas-api-revoke-keys-e2e-inf-b-190407-k9gwm\" (UID: \"a96dcf37-d552-45cf-9d9e-448af64f7595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"
Oct 30 01:42:10.847989 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.847893    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96dcf37-d552-45cf-9d9e-448af64f7595-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-190407-k9gwm\" (UID: \"a96dcf37-d552-45cf-9d9e-448af64f7595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"
Oct 30 01:42:10.847989 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.847949    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkdq6\" (UniqueName: \"kubernetes.io/projected/a96dcf37-d552-45cf-9d9e-448af64f7595-kube-api-access-xkdq6\") pod \"maas-api-revoke-keys-e2e-inf-b-190407-k9gwm\" (UID: \"a96dcf37-d552-45cf-9d9e-448af64f7595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"
Oct 30 01:42:10.848270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.848102    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96dcf37-d552-45cf-9d9e-448af64f7595-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-190407-k9gwm\" (UID: \"a96dcf37-d552-45cf-9d9e-448af64f7595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"
Oct 30 01:42:10.848693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.848668    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96dcf37-d552-45cf-9d9e-448af64f7595-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-190407-k9gwm\" (UID: \"a96dcf37-d552-45cf-9d9e-448af64f7595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"
Oct 30 01:42:10.850646 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.850623    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96dcf37-d552-45cf-9d9e-448af64f7595-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-190407-k9gwm\" (UID: \"a96dcf37-d552-45cf-9d9e-448af64f7595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"
Oct 30 01:42:10.873473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.873444    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkdq6\" (UniqueName: \"kubernetes.io/projected/a96dcf37-d552-45cf-9d9e-448af64f7595-kube-api-access-xkdq6\") pod \"maas-api-revoke-keys-e2e-inf-b-190407-k9gwm\" (UID: \"a96dcf37-d552-45cf-9d9e-448af64f7595\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"
Oct 30 01:42:10.909262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:10.909223    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"
Oct 30 01:42:11.222060 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:11.222021    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"]
Oct 30 01:42:11.223348 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:42:11.223317    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96dcf37_d552_45cf_9d9e_448af64f7595.slice/crio-65d58409f0ad26ac33ff527d183961a568fab489457d3596f968eb6d2b37cd00 WatchSource:0}: Error finding container 65d58409f0ad26ac33ff527d183961a568fab489457d3596f968eb6d2b37cd00: Status 404 returned error can't find the container with id 65d58409f0ad26ac33ff527d183961a568fab489457d3596f968eb6d2b37cd00
Oct 30 01:42:11.425200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:11.425135    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm" event={"ID":"a96dcf37-d552-45cf-9d9e-448af64f7595","Type":"ContainerStarted","Data":"65d58409f0ad26ac33ff527d183961a568fab489457d3596f968eb6d2b37cd00"}
Oct 30 01:42:12.239410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.239381    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:42:12.371754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.371721    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81dd80d6-4327-4622-937a-39c52afd0fef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81dd80d6-4327-4622-937a-39c52afd0fef-istio-podinfo\") pod \"81dd80d6-4327-4622-937a-39c52afd0fef\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") "
Oct 30 01:42:12.371897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.371789    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-credential-socket\") pod \"81dd80d6-4327-4622-937a-39c52afd0fef\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") "
Oct 30 01:42:12.371897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.371824    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-istio-envoy\") pod \"81dd80d6-4327-4622-937a-39c52afd0fef\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") "
Oct 30 01:42:12.372026 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.371927    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-workload-socket\") pod \"81dd80d6-4327-4622-937a-39c52afd0fef\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") "
Oct 30 01:42:12.372026 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.371961    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81dd80d6-4327-4622-937a-39c52afd0fef-istio-token\" (UniqueName: \"kubernetes.io/projected/81dd80d6-4327-4622-937a-39c52afd0fef-istio-token\") pod \"81dd80d6-4327-4622-937a-39c52afd0fef\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") "
Oct 30 01:42:12.372026 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.372005    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-workload-certs\") pod \"81dd80d6-4327-4622-937a-39c52afd0fef\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") "
Oct 30 01:42:12.372224 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.372044    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-istio-data\") pod \"81dd80d6-4327-4622-937a-39c52afd0fef\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") "
Oct 30 01:42:12.372224 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.372074    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81dd80d6-4327-4622-937a-39c52afd0fef-kube-api-access-r2wlm\" (UniqueName: \"kubernetes.io/projected/81dd80d6-4327-4622-937a-39c52afd0fef-kube-api-access-r2wlm\") pod \"81dd80d6-4327-4622-937a-39c52afd0fef\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") "
Oct 30 01:42:12.372224 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.372130    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-credential-socket" pod "81dd80d6-4327-4622-937a-39c52afd0fef" (UID: "81dd80d6-4327-4622-937a-39c52afd0fef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:12.372224 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.372151    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81dd80d6-4327-4622-937a-39c52afd0fef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81dd80d6-4327-4622-937a-39c52afd0fef-istiod-ca-cert\") pod \"81dd80d6-4327-4622-937a-39c52afd0fef\" (UID: \"81dd80d6-4327-4622-937a-39c52afd0fef\") "
Oct 30 01:42:12.372449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.372341    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-istio-data" pod "81dd80d6-4327-4622-937a-39c52afd0fef" (UID: "81dd80d6-4327-4622-937a-39c52afd0fef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:12.372449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.372356    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-workload-certs" pod "81dd80d6-4327-4622-937a-39c52afd0fef" (UID: "81dd80d6-4327-4622-937a-39c52afd0fef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:12.372449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.372374    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-workload-socket" pod "81dd80d6-4327-4622-937a-39c52afd0fef" (UID: "81dd80d6-4327-4622-937a-39c52afd0fef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:12.372626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.372588    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81dd80d6-4327-4622-937a-39c52afd0fef-istiod-ca-cert" pod "81dd80d6-4327-4622-937a-39c52afd0fef" (UID: "81dd80d6-4327-4622-937a-39c52afd0fef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:42:12.372907 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.372886    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.372988 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.372913    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.372988 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.372926    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.372988 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.372940    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.372988 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.372953    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81dd80d6-4327-4622-937a-39c52afd0fef-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.374612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.374591    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81dd80d6-4327-4622-937a-39c52afd0fef-istio-token" pod "81dd80d6-4327-4622-937a-39c52afd0fef" (UID: "81dd80d6-4327-4622-937a-39c52afd0fef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:12.389553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.389524    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81dd80d6-4327-4622-937a-39c52afd0fef-kube-api-access-r2wlm" pod "81dd80d6-4327-4622-937a-39c52afd0fef" (UID: "81dd80d6-4327-4622-937a-39c52afd0fef"). InnerVolumeSpecName "kube-api-access-r2wlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:12.400585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.400555    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81dd80d6-4327-4622-937a-39c52afd0fef-istio-podinfo" pod "81dd80d6-4327-4622-937a-39c52afd0fef" (UID: "81dd80d6-4327-4622-937a-39c52afd0fef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:42:12.400743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.400610    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-istio-envoy" pod "81dd80d6-4327-4622-937a-39c52afd0fef" (UID: "81dd80d6-4327-4622-937a-39c52afd0fef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:42:12.425133 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.425101    4425 generic.go:356] "Generic (PLEG): container finished" podID="81dd80d6-4327-4622-937a-39c52afd0fef" containerID="a431e23bc8d7f2b885d05ebec4840e68539984606077db3234af4e3a8bd0bc56" exitCode=0
Oct 30 01:42:12.425351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.425147    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"
Oct 30 01:42:12.425351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.425192    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l" event={"ID":"81dd80d6-4327-4622-937a-39c52afd0fef","Type":"ContainerDied","Data":"a431e23bc8d7f2b885d05ebec4840e68539984606077db3234af4e3a8bd0bc56"}
Oct 30 01:42:12.425351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.425249    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l" event={"ID":"81dd80d6-4327-4622-937a-39c52afd0fef","Type":"ContainerDied","Data":"daf830440a9fd65e1e78ce4024acd47f6703db5b68f749dfdcefd6c19e3a8eb7"}
Oct 30 01:42:12.425351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.425281    4425 scope.go:122] "RemoveContainer" containerID="a431e23bc8d7f2b885d05ebec4840e68539984606077db3234af4e3a8bd0bc56"
Oct 30 01:42:12.427387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.427362    4425 generic.go:356] "Generic (PLEG): container finished" podID="a96dcf37-d552-45cf-9d9e-448af64f7595" containerID="7b1e3ff21f9030dc6de4b2defb72f035ec50fa4154b84f6af483d2369138c3ae" exitCode=0
Oct 30 01:42:12.427513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.427449    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm" event={"ID":"a96dcf37-d552-45cf-9d9e-448af64f7595","Type":"ContainerDied","Data":"7b1e3ff21f9030dc6de4b2defb72f035ec50fa4154b84f6af483d2369138c3ae"}
Oct 30 01:42:12.439772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.439689    4425 scope.go:122] "RemoveContainer" containerID="a431e23bc8d7f2b885d05ebec4840e68539984606077db3234af4e3a8bd0bc56"
Oct 30 01:42:12.440736 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:42:12.440362    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a431e23bc8d7f2b885d05ebec4840e68539984606077db3234af4e3a8bd0bc56\": container with ID starting with a431e23bc8d7f2b885d05ebec4840e68539984606077db3234af4e3a8bd0bc56 not found: ID does not exist" containerID="a431e23bc8d7f2b885d05ebec4840e68539984606077db3234af4e3a8bd0bc56"
Oct 30 01:42:12.440919 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.440751    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a431e23bc8d7f2b885d05ebec4840e68539984606077db3234af4e3a8bd0bc56"} err="failed to get container status \"a431e23bc8d7f2b885d05ebec4840e68539984606077db3234af4e3a8bd0bc56\": rpc error: code = NotFound desc = could not find container \"a431e23bc8d7f2b885d05ebec4840e68539984606077db3234af4e3a8bd0bc56\": container with ID starting with a431e23bc8d7f2b885d05ebec4840e68539984606077db3234af4e3a8bd0bc56 not found: ID does not exist"
Oct 30 01:42:12.474629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.474546    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81dd80d6-4327-4622-937a-39c52afd0fef-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.474629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.474575    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81dd80d6-4327-4622-937a-39c52afd0fef-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.474629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.474585    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2wlm\" (UniqueName: \"kubernetes.io/projected/81dd80d6-4327-4622-937a-39c52afd0fef-kube-api-access-r2wlm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.474629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.474594    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81dd80d6-4327-4622-937a-39c52afd0fef-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:12.556392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.556331    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"]
Oct 30 01:42:12.561000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:12.560973    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-kdm6l"]
Oct 30 01:42:13.426188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:13.426137    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81dd80d6-4327-4622-937a-39c52afd0fef" path="/var/lib/kubelet/pods/81dd80d6-4327-4622-937a-39c52afd0fef/volumes"
Oct 30 01:42:13.624329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:13.624296    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"
Oct 30 01:42:13.793697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:13.793649    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a96dcf37-d552-45cf-9d9e-448af64f7595-kube-api-access-xkdq6\" (UniqueName: \"kubernetes.io/projected/a96dcf37-d552-45cf-9d9e-448af64f7595-kube-api-access-xkdq6\") pod \"a96dcf37-d552-45cf-9d9e-448af64f7595\" (UID: \"a96dcf37-d552-45cf-9d9e-448af64f7595\") "
Oct 30 01:42:13.793945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:13.793924    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a96dcf37-d552-45cf-9d9e-448af64f7595-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96dcf37-d552-45cf-9d9e-448af64f7595-maas-api-openshift-ca\") pod \"a96dcf37-d552-45cf-9d9e-448af64f7595\" (UID: \"a96dcf37-d552-45cf-9d9e-448af64f7595\") "
Oct 30 01:42:13.794071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:13.794053    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a96dcf37-d552-45cf-9d9e-448af64f7595-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96dcf37-d552-45cf-9d9e-448af64f7595-maas-api-xks-ca\") pod \"a96dcf37-d552-45cf-9d9e-448af64f7595\" (UID: \"a96dcf37-d552-45cf-9d9e-448af64f7595\") "
Oct 30 01:42:13.794436 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:13.794408    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96dcf37-d552-45cf-9d9e-448af64f7595-maas-api-openshift-ca" pod "a96dcf37-d552-45cf-9d9e-448af64f7595" (UID: "a96dcf37-d552-45cf-9d9e-448af64f7595"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:42:13.794755 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:13.794683    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96dcf37-d552-45cf-9d9e-448af64f7595-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:13.796321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:13.796297    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a96dcf37-d552-45cf-9d9e-448af64f7595-maas-api-xks-ca" pod "a96dcf37-d552-45cf-9d9e-448af64f7595" (UID: "a96dcf37-d552-45cf-9d9e-448af64f7595"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:42:13.796504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:13.796482    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96dcf37-d552-45cf-9d9e-448af64f7595-kube-api-access-xkdq6" pod "a96dcf37-d552-45cf-9d9e-448af64f7595" (UID: "a96dcf37-d552-45cf-9d9e-448af64f7595"). InnerVolumeSpecName "kube-api-access-xkdq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:13.896694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:13.896653    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkdq6\" (UniqueName: \"kubernetes.io/projected/a96dcf37-d552-45cf-9d9e-448af64f7595-kube-api-access-xkdq6\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:13.896694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:13.896689    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96dcf37-d552-45cf-9d9e-448af64f7595-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:14.446486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:14.446403    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm" event={"ID":"a96dcf37-d552-45cf-9d9e-448af64f7595","Type":"ContainerDied","Data":"65d58409f0ad26ac33ff527d183961a568fab489457d3596f968eb6d2b37cd00"}
Oct 30 01:42:14.446486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:14.446462    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65d58409f0ad26ac33ff527d183961a568fab489457d3596f968eb6d2b37cd00"
Oct 30 01:42:14.446486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:14.446413    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"
Oct 30 01:42:15.937025 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:15.936985    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js" podUID="863be03f-aa38-4aeb-8f71-9ebd0685c391" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:42:15.937573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:15.936985    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh" podUID="29432241-060e-48e7-bb45-578a8391964b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:42:16.368299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.368267    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"
Oct 30 01:42:16.389377 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.389355    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"
Oct 30 01:42:16.433091 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.433052    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/863be03f-aa38-4aeb-8f71-9ebd0685c391-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/863be03f-aa38-4aeb-8f71-9ebd0685c391-plugins-config-volume\") pod \"863be03f-aa38-4aeb-8f71-9ebd0685c391\" (UID: \"863be03f-aa38-4aeb-8f71-9ebd0685c391\") "
Oct 30 01:42:16.433091 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.433095    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/863be03f-aa38-4aeb-8f71-9ebd0685c391-kube-api-access-rskr7\" (UniqueName: \"kubernetes.io/projected/863be03f-aa38-4aeb-8f71-9ebd0685c391-kube-api-access-rskr7\") pod \"863be03f-aa38-4aeb-8f71-9ebd0685c391\" (UID: \"863be03f-aa38-4aeb-8f71-9ebd0685c391\") "
Oct 30 01:42:16.433401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.433269    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29432241-060e-48e7-bb45-578a8391964b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29432241-060e-48e7-bb45-578a8391964b-plugins-config-volume\") pod \"29432241-060e-48e7-bb45-578a8391964b\" (UID: \"29432241-060e-48e7-bb45-578a8391964b\") "
Oct 30 01:42:16.433401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.433325    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29432241-060e-48e7-bb45-578a8391964b-kube-api-access-pc566\" (UniqueName: \"kubernetes.io/projected/29432241-060e-48e7-bb45-578a8391964b-kube-api-access-pc566\") pod \"29432241-060e-48e7-bb45-578a8391964b\" (UID: \"29432241-060e-48e7-bb45-578a8391964b\") "
Oct 30 01:42:16.433606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.433579    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863be03f-aa38-4aeb-8f71-9ebd0685c391-plugins-config-volume" pod "863be03f-aa38-4aeb-8f71-9ebd0685c391" (UID: "863be03f-aa38-4aeb-8f71-9ebd0685c391"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:42:16.433690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.433646    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29432241-060e-48e7-bb45-578a8391964b-plugins-config-volume" pod "29432241-060e-48e7-bb45-578a8391964b" (UID: "29432241-060e-48e7-bb45-578a8391964b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:42:16.434109 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.433972    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/863be03f-aa38-4aeb-8f71-9ebd0685c391-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:16.434109 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.433997    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29432241-060e-48e7-bb45-578a8391964b-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:16.435752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.435725    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863be03f-aa38-4aeb-8f71-9ebd0685c391-kube-api-access-rskr7" pod "863be03f-aa38-4aeb-8f71-9ebd0685c391" (UID: "863be03f-aa38-4aeb-8f71-9ebd0685c391"). InnerVolumeSpecName "kube-api-access-rskr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:16.435887 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.435867    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29432241-060e-48e7-bb45-578a8391964b-kube-api-access-pc566" pod "29432241-060e-48e7-bb45-578a8391964b" (UID: "29432241-060e-48e7-bb45-578a8391964b"). InnerVolumeSpecName "kube-api-access-pc566". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:16.468047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.468013    4425 generic.go:356] "Generic (PLEG): container finished" podID="863be03f-aa38-4aeb-8f71-9ebd0685c391" containerID="a4564e34ba0fff98d71b70936012c2e1545ea8a243a4c9f6e1f8b2d6f6423e8d" exitCode=0
Oct 30 01:42:16.468269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.468061    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"
Oct 30 01:42:16.468269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.468103    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js" event={"ID":"863be03f-aa38-4aeb-8f71-9ebd0685c391","Type":"ContainerDied","Data":"a4564e34ba0fff98d71b70936012c2e1545ea8a243a4c9f6e1f8b2d6f6423e8d"}
Oct 30 01:42:16.468269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.468199    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js" event={"ID":"863be03f-aa38-4aeb-8f71-9ebd0685c391","Type":"ContainerDied","Data":"b6bbbeb93d8a2ad19d20f8af32d129d19f369739590a85e660bb9a3cce177529"}
Oct 30 01:42:16.468269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.468229    4425 scope.go:122] "RemoveContainer" containerID="a4564e34ba0fff98d71b70936012c2e1545ea8a243a4c9f6e1f8b2d6f6423e8d"
Oct 30 01:42:16.470089 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.470062    4425 generic.go:356] "Generic (PLEG): container finished" podID="29432241-060e-48e7-bb45-578a8391964b" containerID="a5346c749393159b6e5526971cd495b693553885d1d391e7a79f750fcecf97d3" exitCode=0
Oct 30 01:42:16.470233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.470117    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"
Oct 30 01:42:16.470233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.470123    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh" event={"ID":"29432241-060e-48e7-bb45-578a8391964b","Type":"ContainerDied","Data":"a5346c749393159b6e5526971cd495b693553885d1d391e7a79f750fcecf97d3"}
Oct 30 01:42:16.470412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.470230    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh" event={"ID":"29432241-060e-48e7-bb45-578a8391964b","Type":"ContainerDied","Data":"29b6086d123e8e4b5f8968695b1969bac36cf5322c97865549a3b279be8512e3"}
Oct 30 01:42:16.488103 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.488079    4425 scope.go:122] "RemoveContainer" containerID="a4564e34ba0fff98d71b70936012c2e1545ea8a243a4c9f6e1f8b2d6f6423e8d"
Oct 30 01:42:16.488651 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:42:16.488492    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4564e34ba0fff98d71b70936012c2e1545ea8a243a4c9f6e1f8b2d6f6423e8d\": container with ID starting with a4564e34ba0fff98d71b70936012c2e1545ea8a243a4c9f6e1f8b2d6f6423e8d not found: ID does not exist" containerID="a4564e34ba0fff98d71b70936012c2e1545ea8a243a4c9f6e1f8b2d6f6423e8d"
Oct 30 01:42:16.488651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.488533    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4564e34ba0fff98d71b70936012c2e1545ea8a243a4c9f6e1f8b2d6f6423e8d"} err="failed to get container status \"a4564e34ba0fff98d71b70936012c2e1545ea8a243a4c9f6e1f8b2d6f6423e8d\": rpc error: code = NotFound desc = could not find container \"a4564e34ba0fff98d71b70936012c2e1545ea8a243a4c9f6e1f8b2d6f6423e8d\": container with ID starting with a4564e34ba0fff98d71b70936012c2e1545ea8a243a4c9f6e1f8b2d6f6423e8d not found: ID does not exist"
Oct 30 01:42:16.488651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.488558    4425 scope.go:122] "RemoveContainer" containerID="a5346c749393159b6e5526971cd495b693553885d1d391e7a79f750fcecf97d3"
Oct 30 01:42:16.500165 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.500132    4425 scope.go:122] "RemoveContainer" containerID="a5346c749393159b6e5526971cd495b693553885d1d391e7a79f750fcecf97d3"
Oct 30 01:42:16.500567 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:42:16.500537    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5346c749393159b6e5526971cd495b693553885d1d391e7a79f750fcecf97d3\": container with ID starting with a5346c749393159b6e5526971cd495b693553885d1d391e7a79f750fcecf97d3 not found: ID does not exist" containerID="a5346c749393159b6e5526971cd495b693553885d1d391e7a79f750fcecf97d3"
Oct 30 01:42:16.500677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.500580    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5346c749393159b6e5526971cd495b693553885d1d391e7a79f750fcecf97d3"} err="failed to get container status \"a5346c749393159b6e5526971cd495b693553885d1d391e7a79f750fcecf97d3\": rpc error: code = NotFound desc = could not find container \"a5346c749393159b6e5526971cd495b693553885d1d391e7a79f750fcecf97d3\": container with ID starting with a5346c749393159b6e5526971cd495b693553885d1d391e7a79f750fcecf97d3 not found: ID does not exist"
Oct 30 01:42:16.518891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.518857    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"]
Oct 30 01:42:16.527163 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.527127    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bd287f-77ccf5bb4c-9l6js"]
Oct 30 01:42:16.534882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.534856    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc566\" (UniqueName: \"kubernetes.io/projected/29432241-060e-48e7-bb45-578a8391964b-kube-api-access-pc566\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:16.534882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.534881    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rskr7\" (UniqueName: \"kubernetes.io/projected/863be03f-aa38-4aeb-8f71-9ebd0685c391-kube-api-access-rskr7\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:16.575387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.575349    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"]
Oct 30 01:42:16.580007 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.579976    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bd287f-574bc5586-rlgvh"]
Oct 30 01:42:16.849196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:16.849122    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq" podUID="11991a7b-d429-4f6e-a8a5-bc8352924d46" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:42:17.425897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:17.425859    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29432241-060e-48e7-bb45-578a8391964b" path="/var/lib/kubelet/pods/29432241-060e-48e7-bb45-578a8391964b/volumes"
Oct 30 01:42:17.426421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:17.426389    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="863be03f-aa38-4aeb-8f71-9ebd0685c391" path="/var/lib/kubelet/pods/863be03f-aa38-4aeb-8f71-9ebd0685c391/volumes"
Oct 30 01:42:17.858676 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:17.858633    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979" podUID="01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:42:20.996350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:20.996304    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"]
Oct 30 01:42:21.025780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:21.025743    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"]
Oct 30 01:42:25.997409 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:25.997362    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p" podUID="260bf897-bd50-4c1e-8e09-ebf7971e0a7e" containerName="payload-processing" containerID="cri-o://edf9779c299ad90508c1f49c9576831a82bdd0356f3af1d2667d158f090c9df0" gracePeriod=25
Oct 30 01:42:26.026896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:26.026839    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn" podUID="6c0efd4d-3b76-4b1a-a856-fde91ea1fb80" containerName="payload-pre-processing" containerID="cri-o://3824912c441812c8d2076e735ff9764f80838dbef70f43bc0f0983c59274de2b" gracePeriod=25
Oct 30 01:42:26.849221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:26.849146    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq" podUID="11991a7b-d429-4f6e-a8a5-bc8352924d46" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:42:27.859454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:27.859415    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979" podUID="01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:42:27.942890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:27.942850    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p" podUID="260bf897-bd50-4c1e-8e09-ebf7971e0a7e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:42:28.954281 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:28.954242    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn" podUID="6c0efd4d-3b76-4b1a-a856-fde91ea1fb80" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:42:29.834749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:29.834718    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"
Oct 30 01:42:29.862095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:29.862068    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"
Oct 30 01:42:29.921943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:29.921909    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9-kube-api-access-xngpx\" (UniqueName: \"kubernetes.io/projected/01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9-kube-api-access-xngpx\") pod \"01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9\" (UID: \"01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9\") "
Oct 30 01:42:29.922201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:29.922113    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9-plugins-config-volume\") pod \"01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9\" (UID: \"01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9\") "
Oct 30 01:42:29.922201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:29.922140    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11991a7b-d429-4f6e-a8a5-bc8352924d46-kube-api-access-6rwjp\" (UniqueName: \"kubernetes.io/projected/11991a7b-d429-4f6e-a8a5-bc8352924d46-kube-api-access-6rwjp\") pod \"11991a7b-d429-4f6e-a8a5-bc8352924d46\" (UID: \"11991a7b-d429-4f6e-a8a5-bc8352924d46\") "
Oct 30 01:42:29.922354 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:29.922264    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11991a7b-d429-4f6e-a8a5-bc8352924d46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11991a7b-d429-4f6e-a8a5-bc8352924d46-plugins-config-volume\") pod \"11991a7b-d429-4f6e-a8a5-bc8352924d46\" (UID: \"11991a7b-d429-4f6e-a8a5-bc8352924d46\") "
Oct 30 01:42:29.922551 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:29.922520    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9-plugins-config-volume" pod "01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9" (UID: "01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:42:29.922759 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:29.922733    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11991a7b-d429-4f6e-a8a5-bc8352924d46-plugins-config-volume" pod "11991a7b-d429-4f6e-a8a5-bc8352924d46" (UID: "11991a7b-d429-4f6e-a8a5-bc8352924d46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:42:29.922914 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:29.922898    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:29.922914 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:29.922915    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11991a7b-d429-4f6e-a8a5-bc8352924d46-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:29.924417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:29.924396    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9-kube-api-access-xngpx" pod "01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9" (UID: "01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9"). InnerVolumeSpecName "kube-api-access-xngpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:29.924519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:29.924424    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11991a7b-d429-4f6e-a8a5-bc8352924d46-kube-api-access-6rwjp" pod "11991a7b-d429-4f6e-a8a5-bc8352924d46" (UID: "11991a7b-d429-4f6e-a8a5-bc8352924d46"). InnerVolumeSpecName "kube-api-access-6rwjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:30.024441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.024406    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xngpx\" (UniqueName: \"kubernetes.io/projected/01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9-kube-api-access-xngpx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:30.024441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.024435    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rwjp\" (UniqueName: \"kubernetes.io/projected/11991a7b-d429-4f6e-a8a5-bc8352924d46-kube-api-access-6rwjp\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:30.598816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.598777    4425 generic.go:356] "Generic (PLEG): container finished" podID="11991a7b-d429-4f6e-a8a5-bc8352924d46" containerID="e195f4fd8555d89721334a389cad5f9b24853ce17b405a88d880a7c892c1caee" exitCode=0
Oct 30 01:42:30.599017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.598824    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"
Oct 30 01:42:30.599017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.598852    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq" event={"ID":"11991a7b-d429-4f6e-a8a5-bc8352924d46","Type":"ContainerDied","Data":"e195f4fd8555d89721334a389cad5f9b24853ce17b405a88d880a7c892c1caee"}
Oct 30 01:42:30.599017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.598908    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq" event={"ID":"11991a7b-d429-4f6e-a8a5-bc8352924d46","Type":"ContainerDied","Data":"2e667fb41a0db07f50067efa9b8c10f76a1b33cb5ef4ef74cc379e54587cf9a9"}
Oct 30 01:42:30.599017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.598933    4425 scope.go:122] "RemoveContainer" containerID="e195f4fd8555d89721334a389cad5f9b24853ce17b405a88d880a7c892c1caee"
Oct 30 01:42:30.600663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.600628    4425 generic.go:356] "Generic (PLEG): container finished" podID="01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9" containerID="61c877f0fd6203e13d36eee730dd3f4dbcf751de605fa15ae331ed8724095b46" exitCode=0
Oct 30 01:42:30.600766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.600668    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"
Oct 30 01:42:30.600766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.600691    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979" event={"ID":"01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9","Type":"ContainerDied","Data":"61c877f0fd6203e13d36eee730dd3f4dbcf751de605fa15ae331ed8724095b46"}
Oct 30 01:42:30.600766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.600747    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979" event={"ID":"01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9","Type":"ContainerDied","Data":"ec4045428d6e787222c9d330287df3378a872d984b573e3f29a3dc916e97bb6a"}
Oct 30 01:42:30.611586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.611567    4425 scope.go:122] "RemoveContainer" containerID="e195f4fd8555d89721334a389cad5f9b24853ce17b405a88d880a7c892c1caee"
Oct 30 01:42:30.611874 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:42:30.611855    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e195f4fd8555d89721334a389cad5f9b24853ce17b405a88d880a7c892c1caee\": container with ID starting with e195f4fd8555d89721334a389cad5f9b24853ce17b405a88d880a7c892c1caee not found: ID does not exist" containerID="e195f4fd8555d89721334a389cad5f9b24853ce17b405a88d880a7c892c1caee"
Oct 30 01:42:30.611952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.611881    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e195f4fd8555d89721334a389cad5f9b24853ce17b405a88d880a7c892c1caee"} err="failed to get container status \"e195f4fd8555d89721334a389cad5f9b24853ce17b405a88d880a7c892c1caee\": rpc error: code = NotFound desc = could not find container \"e195f4fd8555d89721334a389cad5f9b24853ce17b405a88d880a7c892c1caee\": container with ID starting with e195f4fd8555d89721334a389cad5f9b24853ce17b405a88d880a7c892c1caee not found: ID does not exist"
Oct 30 01:42:30.611952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.611897    4425 scope.go:122] "RemoveContainer" containerID="61c877f0fd6203e13d36eee730dd3f4dbcf751de605fa15ae331ed8724095b46"
Oct 30 01:42:30.623539 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.623521    4425 scope.go:122] "RemoveContainer" containerID="61c877f0fd6203e13d36eee730dd3f4dbcf751de605fa15ae331ed8724095b46"
Oct 30 01:42:30.623882 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:42:30.623827    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c877f0fd6203e13d36eee730dd3f4dbcf751de605fa15ae331ed8724095b46\": container with ID starting with 61c877f0fd6203e13d36eee730dd3f4dbcf751de605fa15ae331ed8724095b46 not found: ID does not exist" containerID="61c877f0fd6203e13d36eee730dd3f4dbcf751de605fa15ae331ed8724095b46"
Oct 30 01:42:30.623934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.623893    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c877f0fd6203e13d36eee730dd3f4dbcf751de605fa15ae331ed8724095b46"} err="failed to get container status \"61c877f0fd6203e13d36eee730dd3f4dbcf751de605fa15ae331ed8724095b46\": rpc error: code = NotFound desc = could not find container \"61c877f0fd6203e13d36eee730dd3f4dbcf751de605fa15ae331ed8724095b46\": container with ID starting with 61c877f0fd6203e13d36eee730dd3f4dbcf751de605fa15ae331ed8724095b46 not found: ID does not exist"
Oct 30 01:42:30.648216 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.648168    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"]
Oct 30 01:42:30.654006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.653976    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-415c6b-fc4ccc6fd-nmwfq"]
Oct 30 01:42:30.690244 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.690207    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"]
Oct 30 01:42:30.696409 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:30.696382    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-415c6b-6686d49dd7-bk979"]
Oct 30 01:42:31.426316 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:31.426283    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9" path="/var/lib/kubelet/pods/01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9/volumes"
Oct 30 01:42:31.426766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:31.426751    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11991a7b-d429-4f6e-a8a5-bc8352924d46" path="/var/lib/kubelet/pods/11991a7b-d429-4f6e-a8a5-bc8352924d46/volumes"
Oct 30 01:42:35.683681 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:35.683643    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:42:35.839097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:35.839067    4425 scope.go:122] "RemoveContainer" containerID="c5ecb05b9252eaf3bff7f626816afa030b3c17157fb5e198d07d7978c3ea1dd8"
Oct 30 01:42:35.850215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:35.850186    4425 scope.go:122] "RemoveContainer" containerID="17d3836300c616eec1354758d16c1a18e12ed8d5e5fcaf8d827cf526cb10c0a9"
Oct 30 01:42:37.558590 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:42:37.558551    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:42:37.943245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:37.943109    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p" podUID="260bf897-bd50-4c1e-8e09-ebf7971e0a7e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:42:38.954781 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:38.954740    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn" podUID="6c0efd4d-3b76-4b1a-a856-fde91ea1fb80" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:42:46.326365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.326336    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"
Oct 30 01:42:46.375372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.375336    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"
Oct 30 01:42:46.457919 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.457880    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260bf897-bd50-4c1e-8e09-ebf7971e0a7e-kube-api-access-qp9dw\" (UniqueName: \"kubernetes.io/projected/260bf897-bd50-4c1e-8e09-ebf7971e0a7e-kube-api-access-qp9dw\") pod \"260bf897-bd50-4c1e-8e09-ebf7971e0a7e\" (UID: \"260bf897-bd50-4c1e-8e09-ebf7971e0a7e\") "
Oct 30 01:42:46.458243 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.458202    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c0efd4d-3b76-4b1a-a856-fde91ea1fb80-kube-api-access-r9444\" (UniqueName: \"kubernetes.io/projected/6c0efd4d-3b76-4b1a-a856-fde91ea1fb80-kube-api-access-r9444\") pod \"6c0efd4d-3b76-4b1a-a856-fde91ea1fb80\" (UID: \"6c0efd4d-3b76-4b1a-a856-fde91ea1fb80\") "
Oct 30 01:42:46.458412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.458389    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c0efd4d-3b76-4b1a-a856-fde91ea1fb80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c0efd4d-3b76-4b1a-a856-fde91ea1fb80-plugins-config-volume\") pod \"6c0efd4d-3b76-4b1a-a856-fde91ea1fb80\" (UID: \"6c0efd4d-3b76-4b1a-a856-fde91ea1fb80\") "
Oct 30 01:42:46.459108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.458531    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260bf897-bd50-4c1e-8e09-ebf7971e0a7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260bf897-bd50-4c1e-8e09-ebf7971e0a7e-plugins-config-volume\") pod \"260bf897-bd50-4c1e-8e09-ebf7971e0a7e\" (UID: \"260bf897-bd50-4c1e-8e09-ebf7971e0a7e\") "
Oct 30 01:42:46.459108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.459002    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c0efd4d-3b76-4b1a-a856-fde91ea1fb80-plugins-config-volume" pod "6c0efd4d-3b76-4b1a-a856-fde91ea1fb80" (UID: "6c0efd4d-3b76-4b1a-a856-fde91ea1fb80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:42:46.459108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.459077    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260bf897-bd50-4c1e-8e09-ebf7971e0a7e-plugins-config-volume" pod "260bf897-bd50-4c1e-8e09-ebf7971e0a7e" (UID: "260bf897-bd50-4c1e-8e09-ebf7971e0a7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:42:46.459510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.459241    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c0efd4d-3b76-4b1a-a856-fde91ea1fb80-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:46.459510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.459254    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260bf897-bd50-4c1e-8e09-ebf7971e0a7e-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:46.460846 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.460815    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260bf897-bd50-4c1e-8e09-ebf7971e0a7e-kube-api-access-qp9dw" pod "260bf897-bd50-4c1e-8e09-ebf7971e0a7e" (UID: "260bf897-bd50-4c1e-8e09-ebf7971e0a7e"). InnerVolumeSpecName "kube-api-access-qp9dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:46.462078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.462054    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c0efd4d-3b76-4b1a-a856-fde91ea1fb80-kube-api-access-r9444" pod "6c0efd4d-3b76-4b1a-a856-fde91ea1fb80" (UID: "6c0efd4d-3b76-4b1a-a856-fde91ea1fb80"). InnerVolumeSpecName "kube-api-access-r9444". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:46.561074 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.560976    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9444\" (UniqueName: \"kubernetes.io/projected/6c0efd4d-3b76-4b1a-a856-fde91ea1fb80-kube-api-access-r9444\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:46.561074 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.561016    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp9dw\" (UniqueName: \"kubernetes.io/projected/260bf897-bd50-4c1e-8e09-ebf7971e0a7e-kube-api-access-qp9dw\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:46.749028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.748988    4425 generic.go:356] "Generic (PLEG): container finished" podID="260bf897-bd50-4c1e-8e09-ebf7971e0a7e" containerID="edf9779c299ad90508c1f49c9576831a82bdd0356f3af1d2667d158f090c9df0" exitCode=0
Oct 30 01:42:46.749240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.749037    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"
Oct 30 01:42:46.749240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.749071    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p" event={"ID":"260bf897-bd50-4c1e-8e09-ebf7971e0a7e","Type":"ContainerDied","Data":"edf9779c299ad90508c1f49c9576831a82bdd0356f3af1d2667d158f090c9df0"}
Oct 30 01:42:46.749240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.749126    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p" event={"ID":"260bf897-bd50-4c1e-8e09-ebf7971e0a7e","Type":"ContainerDied","Data":"fdce4e360bcf7b88c2bee93069c12ae302d4abcaea1fcff25c1ff86dbcbe70c6"}
Oct 30 01:42:46.749240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.749150    4425 scope.go:122] "RemoveContainer" containerID="edf9779c299ad90508c1f49c9576831a82bdd0356f3af1d2667d158f090c9df0"
Oct 30 01:42:46.750790 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.750769    4425 generic.go:356] "Generic (PLEG): container finished" podID="6c0efd4d-3b76-4b1a-a856-fde91ea1fb80" containerID="3824912c441812c8d2076e735ff9764f80838dbef70f43bc0f0983c59274de2b" exitCode=0
Oct 30 01:42:46.750897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.750865    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"
Oct 30 01:42:46.750946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.750901    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn" event={"ID":"6c0efd4d-3b76-4b1a-a856-fde91ea1fb80","Type":"ContainerDied","Data":"3824912c441812c8d2076e735ff9764f80838dbef70f43bc0f0983c59274de2b"}
Oct 30 01:42:46.750994 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.750950    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn" event={"ID":"6c0efd4d-3b76-4b1a-a856-fde91ea1fb80","Type":"ContainerDied","Data":"6c92258fa5b7e6df77e625c73782be68268636a7e8de024eec17f29609558a0c"}
Oct 30 01:42:46.763435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.763407    4425 scope.go:122] "RemoveContainer" containerID="edf9779c299ad90508c1f49c9576831a82bdd0356f3af1d2667d158f090c9df0"
Oct 30 01:42:46.763842 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:42:46.763817    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf9779c299ad90508c1f49c9576831a82bdd0356f3af1d2667d158f090c9df0\": container with ID starting with edf9779c299ad90508c1f49c9576831a82bdd0356f3af1d2667d158f090c9df0 not found: ID does not exist" containerID="edf9779c299ad90508c1f49c9576831a82bdd0356f3af1d2667d158f090c9df0"
Oct 30 01:42:46.763935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.763846    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf9779c299ad90508c1f49c9576831a82bdd0356f3af1d2667d158f090c9df0"} err="failed to get container status \"edf9779c299ad90508c1f49c9576831a82bdd0356f3af1d2667d158f090c9df0\": rpc error: code = NotFound desc = could not find container \"edf9779c299ad90508c1f49c9576831a82bdd0356f3af1d2667d158f090c9df0\": container with ID starting with edf9779c299ad90508c1f49c9576831a82bdd0356f3af1d2667d158f090c9df0 not found: ID does not exist"
Oct 30 01:42:46.763935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.763865    4425 scope.go:122] "RemoveContainer" containerID="3824912c441812c8d2076e735ff9764f80838dbef70f43bc0f0983c59274de2b"
Oct 30 01:42:46.775245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.775191    4425 scope.go:122] "RemoveContainer" containerID="3824912c441812c8d2076e735ff9764f80838dbef70f43bc0f0983c59274de2b"
Oct 30 01:42:46.775512 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:42:46.775491    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3824912c441812c8d2076e735ff9764f80838dbef70f43bc0f0983c59274de2b\": container with ID starting with 3824912c441812c8d2076e735ff9764f80838dbef70f43bc0f0983c59274de2b not found: ID does not exist" containerID="3824912c441812c8d2076e735ff9764f80838dbef70f43bc0f0983c59274de2b"
Oct 30 01:42:46.775571 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.775520    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3824912c441812c8d2076e735ff9764f80838dbef70f43bc0f0983c59274de2b"} err="failed to get container status \"3824912c441812c8d2076e735ff9764f80838dbef70f43bc0f0983c59274de2b\": rpc error: code = NotFound desc = could not find container \"3824912c441812c8d2076e735ff9764f80838dbef70f43bc0f0983c59274de2b\": container with ID starting with 3824912c441812c8d2076e735ff9764f80838dbef70f43bc0f0983c59274de2b not found: ID does not exist"
Oct 30 01:42:46.847609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.847531    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"]
Oct 30 01:42:46.857146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.856921    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-190407-6466bdc47b-jzv4p"]
Oct 30 01:42:46.894548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.894508    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"]
Oct 30 01:42:46.903668 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:46.903637    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-190407-57fdf85f8c-cxvtn"]
Oct 30 01:42:47.426094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:47.426058    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260bf897-bd50-4c1e-8e09-ebf7971e0a7e" path="/var/lib/kubelet/pods/260bf897-bd50-4c1e-8e09-ebf7971e0a7e/volumes"
Oct 30 01:42:47.426575 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:47.426559    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c0efd4d-3b76-4b1a-a856-fde91ea1fb80" path="/var/lib/kubelet/pods/6c0efd4d-3b76-4b1a-a856-fde91ea1fb80/volumes"
Oct 30 01:42:50.204384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.204348    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"]
Oct 30 01:42:50.204874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.204626    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp" podUID="2994479c-7570-40e8-b898-e7e5404f22c1" containerName="maas-api" containerID="cri-o://f7b46a68b1d2c42459a3d0fc2657d2526c525ac7ad7dbb1efdf09a59339e26f7" gracePeriod=30
Oct 30 01:42:50.302577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.302529    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp" podUID="2994479c-7570-40e8-b898-e7e5404f22c1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.63:8443/health\": dial tcp 10.133.1.63:8443: connect: connection refused"
Oct 30 01:42:50.550708 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.550675    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:42:50.624178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.623603    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2994479c-7570-40e8-b898-e7e5404f22c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2994479c-7570-40e8-b898-e7e5404f22c1-maas-api-tls\") pod \"2994479c-7570-40e8-b898-e7e5404f22c1\" (UID: \"2994479c-7570-40e8-b898-e7e5404f22c1\") "
Oct 30 01:42:50.624178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.623704    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2994479c-7570-40e8-b898-e7e5404f22c1-kube-api-access-mdrjp\" (UniqueName: \"kubernetes.io/projected/2994479c-7570-40e8-b898-e7e5404f22c1-kube-api-access-mdrjp\") pod \"2994479c-7570-40e8-b898-e7e5404f22c1\" (UID: \"2994479c-7570-40e8-b898-e7e5404f22c1\") "
Oct 30 01:42:50.635221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.625901    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2994479c-7570-40e8-b898-e7e5404f22c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2994479c-7570-40e8-b898-e7e5404f22c1-metrics-certs\") pod \"2994479c-7570-40e8-b898-e7e5404f22c1\" (UID: \"2994479c-7570-40e8-b898-e7e5404f22c1\") "
Oct 30 01:42:50.635221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.626214    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2994479c-7570-40e8-b898-e7e5404f22c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2994479c-7570-40e8-b898-e7e5404f22c1-openshift-service-ca\") pod \"2994479c-7570-40e8-b898-e7e5404f22c1\" (UID: \"2994479c-7570-40e8-b898-e7e5404f22c1\") "
Oct 30 01:42:50.635221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.627321    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2994479c-7570-40e8-b898-e7e5404f22c1-openshift-service-ca" pod "2994479c-7570-40e8-b898-e7e5404f22c1" (UID: "2994479c-7570-40e8-b898-e7e5404f22c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:42:50.635221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.633687    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2994479c-7570-40e8-b898-e7e5404f22c1-kube-api-access-mdrjp" pod "2994479c-7570-40e8-b898-e7e5404f22c1" (UID: "2994479c-7570-40e8-b898-e7e5404f22c1"). InnerVolumeSpecName "kube-api-access-mdrjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:42:50.635221 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.633815    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2994479c-7570-40e8-b898-e7e5404f22c1-maas-api-tls" pod "2994479c-7570-40e8-b898-e7e5404f22c1" (UID: "2994479c-7570-40e8-b898-e7e5404f22c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:42:50.635696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.635665    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2994479c-7570-40e8-b898-e7e5404f22c1-metrics-certs" pod "2994479c-7570-40e8-b898-e7e5404f22c1" (UID: "2994479c-7570-40e8-b898-e7e5404f22c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:42:50.728320 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.728281    4425 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2994479c-7570-40e8-b898-e7e5404f22c1-metrics-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:50.728320 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.728314    4425 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2994479c-7570-40e8-b898-e7e5404f22c1-openshift-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:50.728320 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.728324    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2994479c-7570-40e8-b898-e7e5404f22c1-maas-api-tls\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:50.728320 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.728332    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdrjp\" (UniqueName: \"kubernetes.io/projected/2994479c-7570-40e8-b898-e7e5404f22c1-kube-api-access-mdrjp\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:42:50.788264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.788228    4425 generic.go:356] "Generic (PLEG): container finished" podID="2994479c-7570-40e8-b898-e7e5404f22c1" containerID="f7b46a68b1d2c42459a3d0fc2657d2526c525ac7ad7dbb1efdf09a59339e26f7" exitCode=0
Oct 30 01:42:50.788497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.788290    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"
Oct 30 01:42:50.788497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.788318    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp" event={"ID":"2994479c-7570-40e8-b898-e7e5404f22c1","Type":"ContainerDied","Data":"f7b46a68b1d2c42459a3d0fc2657d2526c525ac7ad7dbb1efdf09a59339e26f7"}
Oct 30 01:42:50.788497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.788375    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp" event={"ID":"2994479c-7570-40e8-b898-e7e5404f22c1","Type":"ContainerDied","Data":"0d62a9f4e1f4fe655fee3d7e88aedbb8c2ce6b580493183a26966ca3382b4371"}
Oct 30 01:42:50.788497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.788398    4425 scope.go:122] "RemoveContainer" containerID="f7b46a68b1d2c42459a3d0fc2657d2526c525ac7ad7dbb1efdf09a59339e26f7"
Oct 30 01:42:50.805431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.805140    4425 scope.go:122] "RemoveContainer" containerID="f7b46a68b1d2c42459a3d0fc2657d2526c525ac7ad7dbb1efdf09a59339e26f7"
Oct 30 01:42:50.805637 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:42:50.805605    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b46a68b1d2c42459a3d0fc2657d2526c525ac7ad7dbb1efdf09a59339e26f7\": container with ID starting with f7b46a68b1d2c42459a3d0fc2657d2526c525ac7ad7dbb1efdf09a59339e26f7 not found: ID does not exist" containerID="f7b46a68b1d2c42459a3d0fc2657d2526c525ac7ad7dbb1efdf09a59339e26f7"
Oct 30 01:42:50.805702 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.805650    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b46a68b1d2c42459a3d0fc2657d2526c525ac7ad7dbb1efdf09a59339e26f7"} err="failed to get container status \"f7b46a68b1d2c42459a3d0fc2657d2526c525ac7ad7dbb1efdf09a59339e26f7\": rpc error: code = NotFound desc = could not find container \"f7b46a68b1d2c42459a3d0fc2657d2526c525ac7ad7dbb1efdf09a59339e26f7\": container with ID starting with f7b46a68b1d2c42459a3d0fc2657d2526c525ac7ad7dbb1efdf09a59339e26f7 not found: ID does not exist"
Oct 30 01:42:50.844559 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.844524    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"]
Oct 30 01:42:50.852451 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.852419    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-415c6b-75c8958f79-22tzp"]
Oct 30 01:42:50.968626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.968570    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"]
Oct 30 01:42:50.973044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:50.973007    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-415c6b-tbcfj"]
Oct 30 01:42:51.426284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:51.426252    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2994479c-7570-40e8-b898-e7e5404f22c1" path="/var/lib/kubelet/pods/2994479c-7570-40e8-b898-e7e5404f22c1/volumes"
Oct 30 01:42:51.426767 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:51.426753    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ebba8f1-f202-41c9-b321-559dcfad5c72" path="/var/lib/kubelet/pods/4ebba8f1-f202-41c9-b321-559dcfad5c72/volumes"
Oct 30 01:42:59.259090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.259051    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"]
Oct 30 01:42:59.259955 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.259937    4425 cpu_manager.go:414] "containerMap: removing container" podUID="81dd80d6-4327-4622-937a-39c52afd0fef" containerName="istio-proxy"
Oct 30 01:42:59.260012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.259956    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81dd80d6-4327-4622-937a-39c52afd0fef" containerName="istio-proxy"
Oct 30 01:42:59.260012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.259971    4425 cpu_manager.go:414] "containerMap: removing container" podUID="260bf897-bd50-4c1e-8e09-ebf7971e0a7e" containerName="payload-processing"
Oct 30 01:42:59.260012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.259977    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260bf897-bd50-4c1e-8e09-ebf7971e0a7e" containerName="payload-processing"
Oct 30 01:42:59.260012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.259988    4425 cpu_manager.go:414] "containerMap: removing container" podUID="2994479c-7570-40e8-b898-e7e5404f22c1" containerName="maas-api"
Oct 30 01:42:59.260012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.259993    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2994479c-7570-40e8-b898-e7e5404f22c1" containerName="maas-api"
Oct 30 01:42:59.260012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260000    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a96dcf37-d552-45cf-9d9e-448af64f7595" containerName="revoke-keys"
Oct 30 01:42:59.260012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260004    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96dcf37-d552-45cf-9d9e-448af64f7595" containerName="revoke-keys"
Oct 30 01:42:59.260012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260014    4425 cpu_manager.go:414] "containerMap: removing container" podUID="29432241-060e-48e7-bb45-578a8391964b" containerName="payload-processing"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260019    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29432241-060e-48e7-bb45-578a8391964b" containerName="payload-processing"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260028    4425 cpu_manager.go:414] "containerMap: removing container" podUID="863be03f-aa38-4aeb-8f71-9ebd0685c391" containerName="payload-pre-processing"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260033    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="863be03f-aa38-4aeb-8f71-9ebd0685c391" containerName="payload-pre-processing"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260044    4425 cpu_manager.go:414] "containerMap: removing container" podUID="11991a7b-d429-4f6e-a8a5-bc8352924d46" containerName="payload-processing"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260048    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11991a7b-d429-4f6e-a8a5-bc8352924d46" containerName="payload-processing"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260057    4425 cpu_manager.go:414] "containerMap: removing container" podUID="6c0efd4d-3b76-4b1a-a856-fde91ea1fb80" containerName="payload-pre-processing"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260062    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c0efd4d-3b76-4b1a-a856-fde91ea1fb80" containerName="payload-pre-processing"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260070    4425 cpu_manager.go:414] "containerMap: removing container" podUID="01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9" containerName="payload-pre-processing"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260074    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9" containerName="payload-pre-processing"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260233    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="81dd80d6-4327-4622-937a-39c52afd0fef" containerName="istio-proxy"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260247    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="260bf897-bd50-4c1e-8e09-ebf7971e0a7e" containerName="payload-processing"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260254    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2994479c-7570-40e8-b898-e7e5404f22c1" containerName="maas-api"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260259    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a96dcf37-d552-45cf-9d9e-448af64f7595" containerName="revoke-keys"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260266    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="29432241-060e-48e7-bb45-578a8391964b" containerName="payload-processing"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260273    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="863be03f-aa38-4aeb-8f71-9ebd0685c391" containerName="payload-pre-processing"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260278    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="11991a7b-d429-4f6e-a8a5-bc8352924d46" containerName="payload-processing"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260284    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c0efd4d-3b76-4b1a-a856-fde91ea1fb80" containerName="payload-pre-processing"
Oct 30 01:42:59.260339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.260290    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="01fbdfda-10b6-4b8d-9c44-9a5ac177e7b9" containerName="payload-pre-processing"
Oct 30 01:42:59.265748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.265725    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"
Oct 30 01:42:59.290925 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.290887    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:42:59.291124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.290991    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:42:59.301208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.299504    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"]
Oct 30 01:42:59.347172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.347131    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr\" (UID: \"97a38f99-0f6f-4f5d-a7a4-412c85e63c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"
Oct 30 01:42:59.347379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.347325    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg9t2\" (UniqueName: \"kubernetes.io/projected/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-kube-api-access-hg9t2\") pod \"maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr\" (UID: \"97a38f99-0f6f-4f5d-a7a4-412c85e63c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"
Oct 30 01:42:59.347570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.347540    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr\" (UID: \"97a38f99-0f6f-4f5d-a7a4-412c85e63c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"
Oct 30 01:42:59.448533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.448487    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg9t2\" (UniqueName: \"kubernetes.io/projected/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-kube-api-access-hg9t2\") pod \"maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr\" (UID: \"97a38f99-0f6f-4f5d-a7a4-412c85e63c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"
Oct 30 01:42:59.448732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.448698    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr\" (UID: \"97a38f99-0f6f-4f5d-a7a4-412c85e63c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"
Oct 30 01:42:59.448814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.448795    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr\" (UID: \"97a38f99-0f6f-4f5d-a7a4-412c85e63c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"
Oct 30 01:42:59.449408 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.449383    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr\" (UID: \"97a38f99-0f6f-4f5d-a7a4-412c85e63c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"
Oct 30 01:42:59.451388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.451362    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr\" (UID: \"97a38f99-0f6f-4f5d-a7a4-412c85e63c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"
Oct 30 01:42:59.473281 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.473245    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg9t2\" (UniqueName: \"kubernetes.io/projected/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-kube-api-access-hg9t2\") pod \"maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr\" (UID: \"97a38f99-0f6f-4f5d-a7a4-412c85e63c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"
Oct 30 01:42:59.581588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.581553    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"
Oct 30 01:42:59.893493 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:42:59.893458    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"]
Oct 30 01:42:59.893683 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:42:59.893662    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a38f99_0f6f_4f5d_a7a4_412c85e63c14.slice/crio-6759a747c958087210bc7160227f10605f7e1b6d392076ca733d437580719258 WatchSource:0}: Error finding container 6759a747c958087210bc7160227f10605f7e1b6d392076ca733d437580719258: Status 404 returned error can't find the container with id 6759a747c958087210bc7160227f10605f7e1b6d392076ca733d437580719258
Oct 30 01:43:00.874407 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:00.874364    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr" event={"ID":"97a38f99-0f6f-4f5d-a7a4-412c85e63c14","Type":"ContainerStarted","Data":"52f2786eba813dc15ac96ac341732f775e97085e8d7f5a81de1eb864846b083c"}
Oct 30 01:43:00.874910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:00.874429    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr" event={"ID":"97a38f99-0f6f-4f5d-a7a4-412c85e63c14","Type":"ContainerStarted","Data":"6759a747c958087210bc7160227f10605f7e1b6d392076ca733d437580719258"}
Oct 30 01:43:00.915704 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:00.915641    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr" podStartSLOduration=1.915624857 podStartE2EDuration="1.915624857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:00.912036564 +0000 UTC m=+3689.661691874" watchObservedRunningTime="2026-10-30 01:43:00.915624857 +0000 UTC m=+3689.665280198"
Oct 30 01:43:01.884952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:01.884912    4425 generic.go:356] "Generic (PLEG): container finished" podID="97a38f99-0f6f-4f5d-a7a4-412c85e63c14" containerID="52f2786eba813dc15ac96ac341732f775e97085e8d7f5a81de1eb864846b083c" exitCode=0
Oct 30 01:43:01.885385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:01.885001    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr" event={"ID":"97a38f99-0f6f-4f5d-a7a4-412c85e63c14","Type":"ContainerDied","Data":"52f2786eba813dc15ac96ac341732f775e97085e8d7f5a81de1eb864846b083c"}
Oct 30 01:43:03.078171 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:03.078123    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"
Oct 30 01:43:03.099707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:03.099676    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-maas-api-xks-ca\") pod \"97a38f99-0f6f-4f5d-a7a4-412c85e63c14\" (UID: \"97a38f99-0f6f-4f5d-a7a4-412c85e63c14\") "
Oct 30 01:43:03.099891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:03.099816    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-kube-api-access-hg9t2\" (UniqueName: \"kubernetes.io/projected/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-kube-api-access-hg9t2\") pod \"97a38f99-0f6f-4f5d-a7a4-412c85e63c14\" (UID: \"97a38f99-0f6f-4f5d-a7a4-412c85e63c14\") "
Oct 30 01:43:03.099975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:03.099904    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-maas-api-openshift-ca\") pod \"97a38f99-0f6f-4f5d-a7a4-412c85e63c14\" (UID: \"97a38f99-0f6f-4f5d-a7a4-412c85e63c14\") "
Oct 30 01:43:03.100306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:03.100278    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-maas-api-openshift-ca" pod "97a38f99-0f6f-4f5d-a7a4-412c85e63c14" (UID: "97a38f99-0f6f-4f5d-a7a4-412c85e63c14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:43:03.100689 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:03.100665    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:03.101982 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:03.101957    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-maas-api-xks-ca" pod "97a38f99-0f6f-4f5d-a7a4-412c85e63c14" (UID: "97a38f99-0f6f-4f5d-a7a4-412c85e63c14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:43:03.102076 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:03.102018    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-kube-api-access-hg9t2" pod "97a38f99-0f6f-4f5d-a7a4-412c85e63c14" (UID: "97a38f99-0f6f-4f5d-a7a4-412c85e63c14"). InnerVolumeSpecName "kube-api-access-hg9t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:43:03.201851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:03.201761    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:03.201851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:03.201797    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg9t2\" (UniqueName: \"kubernetes.io/projected/97a38f99-0f6f-4f5d-a7a4-412c85e63c14-kube-api-access-hg9t2\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:03.903637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:03.903605    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"
Oct 30 01:43:03.903881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:03.903642    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr" event={"ID":"97a38f99-0f6f-4f5d-a7a4-412c85e63c14","Type":"ContainerDied","Data":"6759a747c958087210bc7160227f10605f7e1b6d392076ca733d437580719258"}
Oct 30 01:43:03.903881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:03.903699    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6759a747c958087210bc7160227f10605f7e1b6d392076ca733d437580719258"
Oct 30 01:43:07.557547 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:43:07.557501    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:43:12.620264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:12.620229    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"]
Oct 30 01:43:12.621021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:12.620986    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg" podUID="1651d619-a962-4574-8f03-92e0dda05db5" containerName="maas-api" containerID="cri-o://8eb7f8eee21a77f3f92e096106d01a4d113eee84309f9504e53ae11d4b33b362" gracePeriod=30
Oct 30 01:43:12.923974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:12.923941    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:43:12.990056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:12.990007    4425 generic.go:356] "Generic (PLEG): container finished" podID="1651d619-a962-4574-8f03-92e0dda05db5" containerID="8eb7f8eee21a77f3f92e096106d01a4d113eee84309f9504e53ae11d4b33b362" exitCode=0
Oct 30 01:43:12.990294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:12.990073    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"
Oct 30 01:43:12.990294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:12.990254    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg" event={"ID":"1651d619-a962-4574-8f03-92e0dda05db5","Type":"ContainerDied","Data":"8eb7f8eee21a77f3f92e096106d01a4d113eee84309f9504e53ae11d4b33b362"}
Oct 30 01:43:12.990453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:12.990341    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg" event={"ID":"1651d619-a962-4574-8f03-92e0dda05db5","Type":"ContainerDied","Data":"6571dbffc13ae43b30c33ad345f8a089e13c96dc68811580f486859fb0761e13"}
Oct 30 01:43:12.990453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:12.990373    4425 scope.go:122] "RemoveContainer" containerID="8eb7f8eee21a77f3f92e096106d01a4d113eee84309f9504e53ae11d4b33b362"
Oct 30 01:43:13.004428 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.004404    4425 scope.go:122] "RemoveContainer" containerID="8eb7f8eee21a77f3f92e096106d01a4d113eee84309f9504e53ae11d4b33b362"
Oct 30 01:43:13.004860 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:43:13.004835    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb7f8eee21a77f3f92e096106d01a4d113eee84309f9504e53ae11d4b33b362\": container with ID starting with 8eb7f8eee21a77f3f92e096106d01a4d113eee84309f9504e53ae11d4b33b362 not found: ID does not exist" containerID="8eb7f8eee21a77f3f92e096106d01a4d113eee84309f9504e53ae11d4b33b362"
Oct 30 01:43:13.004995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.004868    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb7f8eee21a77f3f92e096106d01a4d113eee84309f9504e53ae11d4b33b362"} err="failed to get container status \"8eb7f8eee21a77f3f92e096106d01a4d113eee84309f9504e53ae11d4b33b362\": rpc error: code = NotFound desc = could not find container \"8eb7f8eee21a77f3f92e096106d01a4d113eee84309f9504e53ae11d4b33b362\": container with ID starting with 8eb7f8eee21a77f3f92e096106d01a4d113eee84309f9504e53ae11d4b33b362 not found: ID does not exist"
Oct 30 01:43:13.060915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.060877    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1651d619-a962-4574-8f03-92e0dda05db5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1651d619-a962-4574-8f03-92e0dda05db5-metrics-certs\") pod \"1651d619-a962-4574-8f03-92e0dda05db5\" (UID: \"1651d619-a962-4574-8f03-92e0dda05db5\") "
Oct 30 01:43:13.061110 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.061063    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1651d619-a962-4574-8f03-92e0dda05db5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1651d619-a962-4574-8f03-92e0dda05db5-openshift-service-ca\") pod \"1651d619-a962-4574-8f03-92e0dda05db5\" (UID: \"1651d619-a962-4574-8f03-92e0dda05db5\") "
Oct 30 01:43:13.061241 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.061130    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1651d619-a962-4574-8f03-92e0dda05db5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1651d619-a962-4574-8f03-92e0dda05db5-maas-api-tls\") pod \"1651d619-a962-4574-8f03-92e0dda05db5\" (UID: \"1651d619-a962-4574-8f03-92e0dda05db5\") "
Oct 30 01:43:13.061241 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.061212    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1651d619-a962-4574-8f03-92e0dda05db5-kube-api-access-dj8tl\" (UniqueName: \"kubernetes.io/projected/1651d619-a962-4574-8f03-92e0dda05db5-kube-api-access-dj8tl\") pod \"1651d619-a962-4574-8f03-92e0dda05db5\" (UID: \"1651d619-a962-4574-8f03-92e0dda05db5\") "
Oct 30 01:43:13.061532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.061504    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1651d619-a962-4574-8f03-92e0dda05db5-openshift-service-ca" pod "1651d619-a962-4574-8f03-92e0dda05db5" (UID: "1651d619-a962-4574-8f03-92e0dda05db5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:43:13.061914 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.061861    4425 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1651d619-a962-4574-8f03-92e0dda05db5-openshift-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:13.064294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.064264    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1651d619-a962-4574-8f03-92e0dda05db5-maas-api-tls" pod "1651d619-a962-4574-8f03-92e0dda05db5" (UID: "1651d619-a962-4574-8f03-92e0dda05db5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:43:13.064478 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.064453    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1651d619-a962-4574-8f03-92e0dda05db5-kube-api-access-dj8tl" pod "1651d619-a962-4574-8f03-92e0dda05db5" (UID: "1651d619-a962-4574-8f03-92e0dda05db5"). InnerVolumeSpecName "kube-api-access-dj8tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:43:13.064565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.064497    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1651d619-a962-4574-8f03-92e0dda05db5-metrics-certs" pod "1651d619-a962-4574-8f03-92e0dda05db5" (UID: "1651d619-a962-4574-8f03-92e0dda05db5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:43:13.164212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.164060    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1651d619-a962-4574-8f03-92e0dda05db5-maas-api-tls\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:13.164212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.164104    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj8tl\" (UniqueName: \"kubernetes.io/projected/1651d619-a962-4574-8f03-92e0dda05db5-kube-api-access-dj8tl\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:13.164212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.164119    4425 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1651d619-a962-4574-8f03-92e0dda05db5-metrics-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:13.335923 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.335887    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"]
Oct 30 01:43:13.348073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.348035    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-190407-6db54969d8-nq2gg"]
Oct 30 01:43:13.426961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.426879    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1651d619-a962-4574-8f03-92e0dda05db5" path="/var/lib/kubelet/pods/1651d619-a962-4574-8f03-92e0dda05db5/volumes"
Oct 30 01:43:13.427299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.427283    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"]
Oct 30 01:43:13.428648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:13.428617    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-190407-k9gwm"]
Oct 30 01:43:15.425404 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:15.425366    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a96dcf37-d552-45cf-9d9e-448af64f7595" path="/var/lib/kubelet/pods/a96dcf37-d552-45cf-9d9e-448af64f7595/volumes"
Oct 30 01:43:29.320007 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.319969    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"]
Oct 30 01:43:29.320875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.320855    4425 cpu_manager.go:414] "containerMap: removing container" podUID="97a38f99-0f6f-4f5d-a7a4-412c85e63c14" containerName="revoke-keys"
Oct 30 01:43:29.320926 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.320877    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a38f99-0f6f-4f5d-a7a4-412c85e63c14" containerName="revoke-keys"
Oct 30 01:43:29.320926 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.320908    4425 cpu_manager.go:414] "containerMap: removing container" podUID="1651d619-a962-4574-8f03-92e0dda05db5" containerName="maas-api"
Oct 30 01:43:29.320926 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.320913    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1651d619-a962-4574-8f03-92e0dda05db5" containerName="maas-api"
Oct 30 01:43:29.321046 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.321036    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="1651d619-a962-4574-8f03-92e0dda05db5" containerName="maas-api"
Oct 30 01:43:29.321082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.321056    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a38f99-0f6f-4f5d-a7a4-412c85e63c14" containerName="revoke-keys"
Oct 30 01:43:29.326730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.326709    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"
Oct 30 01:43:29.339896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.339865    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:43:29.349884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.349836    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:43:29.358795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.358757    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"]
Oct 30 01:43:29.394765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.394725    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d798ab33-6a65-4b02-bf19-28a3f2a1b320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf3dfd77-7fp2k\" (UID: \"d798ab33-6a65-4b02-bf19-28a3f2a1b320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"
Oct 30 01:43:29.394765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.394769    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d798ab33-6a65-4b02-bf19-28a3f2a1b320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf3dfd77-7fp2k\" (UID: \"d798ab33-6a65-4b02-bf19-28a3f2a1b320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"
Oct 30 01:43:29.395117 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.395093    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrv5s\" (UniqueName: \"kubernetes.io/projected/d798ab33-6a65-4b02-bf19-28a3f2a1b320-kube-api-access-qrv5s\") pod \"maas-api-revoke-sub-cf3dfd77-7fp2k\" (UID: \"d798ab33-6a65-4b02-bf19-28a3f2a1b320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"
Oct 30 01:43:29.496078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.496038    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrv5s\" (UniqueName: \"kubernetes.io/projected/d798ab33-6a65-4b02-bf19-28a3f2a1b320-kube-api-access-qrv5s\") pod \"maas-api-revoke-sub-cf3dfd77-7fp2k\" (UID: \"d798ab33-6a65-4b02-bf19-28a3f2a1b320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"
Oct 30 01:43:29.496299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.496128    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d798ab33-6a65-4b02-bf19-28a3f2a1b320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf3dfd77-7fp2k\" (UID: \"d798ab33-6a65-4b02-bf19-28a3f2a1b320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"
Oct 30 01:43:29.496299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.496148    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d798ab33-6a65-4b02-bf19-28a3f2a1b320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf3dfd77-7fp2k\" (UID: \"d798ab33-6a65-4b02-bf19-28a3f2a1b320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"
Oct 30 01:43:29.496872 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.496849    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d798ab33-6a65-4b02-bf19-28a3f2a1b320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf3dfd77-7fp2k\" (UID: \"d798ab33-6a65-4b02-bf19-28a3f2a1b320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"
Oct 30 01:43:29.498645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.498620    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d798ab33-6a65-4b02-bf19-28a3f2a1b320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf3dfd77-7fp2k\" (UID: \"d798ab33-6a65-4b02-bf19-28a3f2a1b320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"
Oct 30 01:43:29.517525 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.517493    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrv5s\" (UniqueName: \"kubernetes.io/projected/d798ab33-6a65-4b02-bf19-28a3f2a1b320-kube-api-access-qrv5s\") pod \"maas-api-revoke-sub-cf3dfd77-7fp2k\" (UID: \"d798ab33-6a65-4b02-bf19-28a3f2a1b320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"
Oct 30 01:43:29.643481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.643393    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"
Oct 30 01:43:29.746533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.746491    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"]
Oct 30 01:43:29.752028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.752002    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"
Oct 30 01:43:29.779294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.779264    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"]
Oct 30 01:43:29.905677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.905576    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zb2r\" (UniqueName: \"kubernetes.io/projected/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-kube-api-access-7zb2r\") pod \"maas-api-revoke-sub-b3c3ae11-m2jwn\" (UID: \"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"
Oct 30 01:43:29.905911 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.905738    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3c3ae11-m2jwn\" (UID: \"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"
Oct 30 01:43:29.905911 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.905796    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3c3ae11-m2jwn\" (UID: \"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"
Oct 30 01:43:29.954395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:29.954354    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"]
Oct 30 01:43:29.954786 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:43:29.954762    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd798ab33_6a65_4b02_bf19_28a3f2a1b320.slice/crio-45c742661d0d15bae63f3e701fd59cca74d595efb232bcfb89284a99ef97b1ed WatchSource:0}: Error finding container 45c742661d0d15bae63f3e701fd59cca74d595efb232bcfb89284a99ef97b1ed: Status 404 returned error can't find the container with id 45c742661d0d15bae63f3e701fd59cca74d595efb232bcfb89284a99ef97b1ed
Oct 30 01:43:30.007582 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:30.007544    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zb2r\" (UniqueName: \"kubernetes.io/projected/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-kube-api-access-7zb2r\") pod \"maas-api-revoke-sub-b3c3ae11-m2jwn\" (UID: \"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"
Oct 30 01:43:30.007746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:30.007656    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3c3ae11-m2jwn\" (UID: \"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"
Oct 30 01:43:30.007746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:30.007701    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3c3ae11-m2jwn\" (UID: \"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"
Oct 30 01:43:30.008337 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:30.008317    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3c3ae11-m2jwn\" (UID: \"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"
Oct 30 01:43:30.010015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:30.009998    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3c3ae11-m2jwn\" (UID: \"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"
Oct 30 01:43:30.030208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:30.030181    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zb2r\" (UniqueName: \"kubernetes.io/projected/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-kube-api-access-7zb2r\") pod \"maas-api-revoke-sub-b3c3ae11-m2jwn\" (UID: \"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"
Oct 30 01:43:30.078151 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:30.078115    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"
Oct 30 01:43:30.143419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:30.142508    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k" event={"ID":"d798ab33-6a65-4b02-bf19-28a3f2a1b320","Type":"ContainerStarted","Data":"45c742661d0d15bae63f3e701fd59cca74d595efb232bcfb89284a99ef97b1ed"}
Oct 30 01:43:30.397249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:30.397217    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"]
Oct 30 01:43:30.399183 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:43:30.399127    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e1b5f1_badf_4ac4_9152_1f7489f1a8c6.slice/crio-d81834baf788c0df7b8c5cfc4773c1f66dc39af9ff2af29e8115666de9ac3c2f WatchSource:0}: Error finding container d81834baf788c0df7b8c5cfc4773c1f66dc39af9ff2af29e8115666de9ac3c2f: Status 404 returned error can't find the container with id d81834baf788c0df7b8c5cfc4773c1f66dc39af9ff2af29e8115666de9ac3c2f
Oct 30 01:43:31.152164 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:31.152112    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn" event={"ID":"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6","Type":"ContainerStarted","Data":"87406ea0119cb37d61f0c02534bd0e03026f8516475cfcf4b9f0ecb5037bd002"}
Oct 30 01:43:31.152430 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:31.152402    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn" event={"ID":"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6","Type":"ContainerStarted","Data":"d81834baf788c0df7b8c5cfc4773c1f66dc39af9ff2af29e8115666de9ac3c2f"}
Oct 30 01:43:31.154212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:31.154183    4425 generic.go:356] "Generic (PLEG): container finished" podID="d798ab33-6a65-4b02-bf19-28a3f2a1b320" containerID="2a644c80518964f21504f9a32e65616e6d2b6de01ae8b9bb430d34440e393c9b" exitCode=0
Oct 30 01:43:31.154321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:31.154221    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k" event={"ID":"d798ab33-6a65-4b02-bf19-28a3f2a1b320","Type":"ContainerDied","Data":"2a644c80518964f21504f9a32e65616e6d2b6de01ae8b9bb430d34440e393c9b"}
Oct 30 01:43:31.192207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:31.192126    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn" podStartSLOduration=2.192107796 podStartE2EDuration="2.192107796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:31.18852731 +0000 UTC m=+3719.938182632" watchObservedRunningTime="2026-10-30 01:43:31.192107796 +0000 UTC m=+3719.941763109"
Oct 30 01:43:32.163911 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:32.163874    4425 generic.go:356] "Generic (PLEG): container finished" podID="a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6" containerID="87406ea0119cb37d61f0c02534bd0e03026f8516475cfcf4b9f0ecb5037bd002" exitCode=0
Oct 30 01:43:32.164436 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:32.163964    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn" event={"ID":"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6","Type":"ContainerDied","Data":"87406ea0119cb37d61f0c02534bd0e03026f8516475cfcf4b9f0ecb5037bd002"}
Oct 30 01:43:32.340296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:32.340268    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"
Oct 30 01:43:32.441370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:32.441284    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d798ab33-6a65-4b02-bf19-28a3f2a1b320-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d798ab33-6a65-4b02-bf19-28a3f2a1b320-maas-api-openshift-ca\") pod \"d798ab33-6a65-4b02-bf19-28a3f2a1b320\" (UID: \"d798ab33-6a65-4b02-bf19-28a3f2a1b320\") "
Oct 30 01:43:32.441370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:32.441327    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d798ab33-6a65-4b02-bf19-28a3f2a1b320-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d798ab33-6a65-4b02-bf19-28a3f2a1b320-maas-api-xks-ca\") pod \"d798ab33-6a65-4b02-bf19-28a3f2a1b320\" (UID: \"d798ab33-6a65-4b02-bf19-28a3f2a1b320\") "
Oct 30 01:43:32.441624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:32.441373    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d798ab33-6a65-4b02-bf19-28a3f2a1b320-kube-api-access-qrv5s\" (UniqueName: \"kubernetes.io/projected/d798ab33-6a65-4b02-bf19-28a3f2a1b320-kube-api-access-qrv5s\") pod \"d798ab33-6a65-4b02-bf19-28a3f2a1b320\" (UID: \"d798ab33-6a65-4b02-bf19-28a3f2a1b320\") "
Oct 30 01:43:32.441702 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:32.441653    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d798ab33-6a65-4b02-bf19-28a3f2a1b320-maas-api-openshift-ca" pod "d798ab33-6a65-4b02-bf19-28a3f2a1b320" (UID: "d798ab33-6a65-4b02-bf19-28a3f2a1b320"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:43:32.441922 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:32.441903    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d798ab33-6a65-4b02-bf19-28a3f2a1b320-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:32.443567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:32.443546    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d798ab33-6a65-4b02-bf19-28a3f2a1b320-maas-api-xks-ca" pod "d798ab33-6a65-4b02-bf19-28a3f2a1b320" (UID: "d798ab33-6a65-4b02-bf19-28a3f2a1b320"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:43:32.443567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:32.443553    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d798ab33-6a65-4b02-bf19-28a3f2a1b320-kube-api-access-qrv5s" pod "d798ab33-6a65-4b02-bf19-28a3f2a1b320" (UID: "d798ab33-6a65-4b02-bf19-28a3f2a1b320"). InnerVolumeSpecName "kube-api-access-qrv5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:43:32.543928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:32.543878    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d798ab33-6a65-4b02-bf19-28a3f2a1b320-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:32.543928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:32.543917    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrv5s\" (UniqueName: \"kubernetes.io/projected/d798ab33-6a65-4b02-bf19-28a3f2a1b320-kube-api-access-qrv5s\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:33.173258 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:33.173222    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"
Oct 30 01:43:33.173711 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:33.173256    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k" event={"ID":"d798ab33-6a65-4b02-bf19-28a3f2a1b320","Type":"ContainerDied","Data":"45c742661d0d15bae63f3e701fd59cca74d595efb232bcfb89284a99ef97b1ed"}
Oct 30 01:43:33.173711 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:33.173316    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45c742661d0d15bae63f3e701fd59cca74d595efb232bcfb89284a99ef97b1ed"
Oct 30 01:43:33.344912 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:33.344883    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"
Oct 30 01:43:33.455297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:33.455208    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-maas-api-openshift-ca\") pod \"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6\" (UID: \"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6\") "
Oct 30 01:43:33.455481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:33.455300    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-maas-api-xks-ca\") pod \"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6\" (UID: \"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6\") "
Oct 30 01:43:33.455481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:33.455376    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-kube-api-access-7zb2r\" (UniqueName: \"kubernetes.io/projected/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-kube-api-access-7zb2r\") pod \"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6\" (UID: \"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6\") "
Oct 30 01:43:33.455593 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:33.455523    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-maas-api-openshift-ca" pod "a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6" (UID: "a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:43:33.455951 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:33.455926    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:33.457286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:33.457263    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-maas-api-xks-ca" pod "a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6" (UID: "a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:43:33.457433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:33.457413    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-kube-api-access-7zb2r" pod "a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6" (UID: "a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6"). InnerVolumeSpecName "kube-api-access-7zb2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:43:33.557529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:33.557487    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:33.557529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:33.557516    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zb2r\" (UniqueName: \"kubernetes.io/projected/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6-kube-api-access-7zb2r\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:34.183385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:34.183356    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"
Oct 30 01:43:34.183385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:34.183380    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn" event={"ID":"a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6","Type":"ContainerDied","Data":"d81834baf788c0df7b8c5cfc4773c1f66dc39af9ff2af29e8115666de9ac3c2f"}
Oct 30 01:43:34.183986 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:34.183430    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d81834baf788c0df7b8c5cfc4773c1f66dc39af9ff2af29e8115666de9ac3c2f"
Oct 30 01:43:37.556732 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:43:37.556693    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:43:40.299599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:40.299562    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"]
Oct 30 01:43:40.888177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:40.888108    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"]
Oct 30 01:43:40.889696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:40.889653    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6" containerName="revoke-keys"
Oct 30 01:43:40.889696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:40.889687    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6" containerName="revoke-keys"
Oct 30 01:43:40.889898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:40.889762    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d798ab33-6a65-4b02-bf19-28a3f2a1b320" containerName="revoke-keys"
Oct 30 01:43:40.889898 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:40.889771    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d798ab33-6a65-4b02-bf19-28a3f2a1b320" containerName="revoke-keys"
Oct 30 01:43:40.890000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:40.889981    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6" containerName="revoke-keys"
Oct 30 01:43:40.890057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:40.890017    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d798ab33-6a65-4b02-bf19-28a3f2a1b320" containerName="revoke-keys"
Oct 30 01:43:40.894312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:40.894290    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"
Oct 30 01:43:40.915140 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:40.912990    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:43:40.919941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:40.919904    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"]
Oct 30 01:43:40.924198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:40.922119    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:43:40.962107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:40.962051    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/351e25a6-877b-47ad-8206-b383bf0bad41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f73a184-bgv6h\" (UID: \"351e25a6-877b-47ad-8206-b383bf0bad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"
Oct 30 01:43:40.962354 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:40.962330    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/351e25a6-877b-47ad-8206-b383bf0bad41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f73a184-bgv6h\" (UID: \"351e25a6-877b-47ad-8206-b383bf0bad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"
Oct 30 01:43:40.962447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:40.962429    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgqql\" (UniqueName: \"kubernetes.io/projected/351e25a6-877b-47ad-8206-b383bf0bad41-kube-api-access-fgqql\") pod \"maas-api-revoke-sub-8f73a184-bgv6h\" (UID: \"351e25a6-877b-47ad-8206-b383bf0bad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"
Oct 30 01:43:41.063503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.063470    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/351e25a6-877b-47ad-8206-b383bf0bad41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f73a184-bgv6h\" (UID: \"351e25a6-877b-47ad-8206-b383bf0bad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"
Oct 30 01:43:41.063686 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.063573    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgqql\" (UniqueName: \"kubernetes.io/projected/351e25a6-877b-47ad-8206-b383bf0bad41-kube-api-access-fgqql\") pod \"maas-api-revoke-sub-8f73a184-bgv6h\" (UID: \"351e25a6-877b-47ad-8206-b383bf0bad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"
Oct 30 01:43:41.063686 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.063637    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/351e25a6-877b-47ad-8206-b383bf0bad41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f73a184-bgv6h\" (UID: \"351e25a6-877b-47ad-8206-b383bf0bad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"
Oct 30 01:43:41.064074 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.064053    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/351e25a6-877b-47ad-8206-b383bf0bad41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f73a184-bgv6h\" (UID: \"351e25a6-877b-47ad-8206-b383bf0bad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"
Oct 30 01:43:41.066057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.066029    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/351e25a6-877b-47ad-8206-b383bf0bad41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f73a184-bgv6h\" (UID: \"351e25a6-877b-47ad-8206-b383bf0bad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"
Oct 30 01:43:41.085354 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.085324    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgqql\" (UniqueName: \"kubernetes.io/projected/351e25a6-877b-47ad-8206-b383bf0bad41-kube-api-access-fgqql\") pod \"maas-api-revoke-sub-8f73a184-bgv6h\" (UID: \"351e25a6-877b-47ad-8206-b383bf0bad41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"
Oct 30 01:43:41.212225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.212096    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"
Oct 30 01:43:41.456976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.456932    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"]
Oct 30 01:43:41.463046 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.462971    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"
Oct 30 01:43:41.485937 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.485902    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"]
Oct 30 01:43:41.543476 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.543445    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"]
Oct 30 01:43:41.543618 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:43:41.543585    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351e25a6_877b_47ad_8206_b383bf0bad41.slice/crio-744744251fe452e225a6e2233a15854ff03902e9448f265414a4ec21b60aa96b WatchSource:0}: Error finding container 744744251fe452e225a6e2233a15854ff03902e9448f265414a4ec21b60aa96b: Status 404 returned error can't find the container with id 744744251fe452e225a6e2233a15854ff03902e9448f265414a4ec21b60aa96b
Oct 30 01:43:41.571238 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.571208    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c94a9366-cnvv4\" (UID: \"9b3214ea-a3b4-43d8-88b7-01d044b63bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"
Oct 30 01:43:41.571363 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.571258    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcml7\" (UniqueName: \"kubernetes.io/projected/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-kube-api-access-wcml7\") pod \"maas-api-revoke-sub-c94a9366-cnvv4\" (UID: \"9b3214ea-a3b4-43d8-88b7-01d044b63bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"
Oct 30 01:43:41.571685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.571654    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c94a9366-cnvv4\" (UID: \"9b3214ea-a3b4-43d8-88b7-01d044b63bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"
Oct 30 01:43:41.672989 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.672935    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c94a9366-cnvv4\" (UID: \"9b3214ea-a3b4-43d8-88b7-01d044b63bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"
Oct 30 01:43:41.673243 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.673073    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c94a9366-cnvv4\" (UID: \"9b3214ea-a3b4-43d8-88b7-01d044b63bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"
Oct 30 01:43:41.673243 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.673116    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcml7\" (UniqueName: \"kubernetes.io/projected/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-kube-api-access-wcml7\") pod \"maas-api-revoke-sub-c94a9366-cnvv4\" (UID: \"9b3214ea-a3b4-43d8-88b7-01d044b63bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"
Oct 30 01:43:41.673891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.673866    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c94a9366-cnvv4\" (UID: \"9b3214ea-a3b4-43d8-88b7-01d044b63bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"
Oct 30 01:43:41.675623 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.675600    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c94a9366-cnvv4\" (UID: \"9b3214ea-a3b4-43d8-88b7-01d044b63bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"
Oct 30 01:43:41.697359 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.697329    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcml7\" (UniqueName: \"kubernetes.io/projected/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-kube-api-access-wcml7\") pod \"maas-api-revoke-sub-c94a9366-cnvv4\" (UID: \"9b3214ea-a3b4-43d8-88b7-01d044b63bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"
Oct 30 01:43:41.784722 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:41.784667    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"
Oct 30 01:43:42.114930 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:43:42.114899    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3214ea_a3b4_43d8_88b7_01d044b63bb4.slice/crio-7e507de8217e44ea77b319780d9e4e98b75fabdadf89e6c56281b6370afd6e10 WatchSource:0}: Error finding container 7e507de8217e44ea77b319780d9e4e98b75fabdadf89e6c56281b6370afd6e10: Status 404 returned error can't find the container with id 7e507de8217e44ea77b319780d9e4e98b75fabdadf89e6c56281b6370afd6e10
Oct 30 01:43:42.115089 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:42.115050    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"]
Oct 30 01:43:42.255145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:42.255104    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4" event={"ID":"9b3214ea-a3b4-43d8-88b7-01d044b63bb4","Type":"ContainerStarted","Data":"7e507de8217e44ea77b319780d9e4e98b75fabdadf89e6c56281b6370afd6e10"}
Oct 30 01:43:42.257756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:42.257724    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h" event={"ID":"351e25a6-877b-47ad-8206-b383bf0bad41","Type":"ContainerStarted","Data":"05a0b5fbaa71e6816eeaed242b9003dbf1d94173988025ad4cecf55222be01d5"}
Oct 30 01:43:42.257895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:42.257772    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h" event={"ID":"351e25a6-877b-47ad-8206-b383bf0bad41","Type":"ContainerStarted","Data":"744744251fe452e225a6e2233a15854ff03902e9448f265414a4ec21b60aa96b"}
Oct 30 01:43:42.295111 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:42.295057    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h" podStartSLOduration=2.29504213 podStartE2EDuration="2.29504213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:42.293523809 +0000 UTC m=+3731.043179131" watchObservedRunningTime="2026-10-30 01:43:42.29504213 +0000 UTC m=+3731.044697499"
Oct 30 01:43:43.268314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:43.268259    4425 generic.go:356] "Generic (PLEG): container finished" podID="9b3214ea-a3b4-43d8-88b7-01d044b63bb4" containerID="7dccfb13bdbe6cf9594df1b7c654a39a003e93e21c3d94d2d7489eece06c6884" exitCode=0
Oct 30 01:43:43.268724 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:43.268346    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4" event={"ID":"9b3214ea-a3b4-43d8-88b7-01d044b63bb4","Type":"ContainerDied","Data":"7dccfb13bdbe6cf9594df1b7c654a39a003e93e21c3d94d2d7489eece06c6884"}
Oct 30 01:43:43.270032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:43.270008    4425 generic.go:356] "Generic (PLEG): container finished" podID="351e25a6-877b-47ad-8206-b383bf0bad41" containerID="05a0b5fbaa71e6816eeaed242b9003dbf1d94173988025ad4cecf55222be01d5" exitCode=0
Oct 30 01:43:43.270116 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:43.270090    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h" event={"ID":"351e25a6-877b-47ad-8206-b383bf0bad41","Type":"ContainerDied","Data":"05a0b5fbaa71e6816eeaed242b9003dbf1d94173988025ad4cecf55222be01d5"}
Oct 30 01:43:44.474634 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.474606    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"
Oct 30 01:43:44.511608 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.511574    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"
Oct 30 01:43:44.522611 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.522580    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351e25a6-877b-47ad-8206-b383bf0bad41-kube-api-access-fgqql\" (UniqueName: \"kubernetes.io/projected/351e25a6-877b-47ad-8206-b383bf0bad41-kube-api-access-fgqql\") pod \"351e25a6-877b-47ad-8206-b383bf0bad41\" (UID: \"351e25a6-877b-47ad-8206-b383bf0bad41\") "
Oct 30 01:43:44.522791 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.522621    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/351e25a6-877b-47ad-8206-b383bf0bad41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/351e25a6-877b-47ad-8206-b383bf0bad41-maas-api-xks-ca\") pod \"351e25a6-877b-47ad-8206-b383bf0bad41\" (UID: \"351e25a6-877b-47ad-8206-b383bf0bad41\") "
Oct 30 01:43:44.522791 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.522707    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/351e25a6-877b-47ad-8206-b383bf0bad41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/351e25a6-877b-47ad-8206-b383bf0bad41-maas-api-openshift-ca\") pod \"351e25a6-877b-47ad-8206-b383bf0bad41\" (UID: \"351e25a6-877b-47ad-8206-b383bf0bad41\") "
Oct 30 01:43:44.523167 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.523125    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351e25a6-877b-47ad-8206-b383bf0bad41-maas-api-openshift-ca" pod "351e25a6-877b-47ad-8206-b383bf0bad41" (UID: "351e25a6-877b-47ad-8206-b383bf0bad41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:43:44.523805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.523762    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/351e25a6-877b-47ad-8206-b383bf0bad41-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:44.525376 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.525348    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351e25a6-877b-47ad-8206-b383bf0bad41-kube-api-access-fgqql" pod "351e25a6-877b-47ad-8206-b383bf0bad41" (UID: "351e25a6-877b-47ad-8206-b383bf0bad41"). InnerVolumeSpecName "kube-api-access-fgqql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:43:44.525376 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.525362    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351e25a6-877b-47ad-8206-b383bf0bad41-maas-api-xks-ca" pod "351e25a6-877b-47ad-8206-b383bf0bad41" (UID: "351e25a6-877b-47ad-8206-b383bf0bad41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:43:44.624198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.624138    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-maas-api-openshift-ca\") pod \"9b3214ea-a3b4-43d8-88b7-01d044b63bb4\" (UID: \"9b3214ea-a3b4-43d8-88b7-01d044b63bb4\") "
Oct 30 01:43:44.624436 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.624328    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-kube-api-access-wcml7\" (UniqueName: \"kubernetes.io/projected/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-kube-api-access-wcml7\") pod \"9b3214ea-a3b4-43d8-88b7-01d044b63bb4\" (UID: \"9b3214ea-a3b4-43d8-88b7-01d044b63bb4\") "
Oct 30 01:43:44.624508 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.624485    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-maas-api-xks-ca\") pod \"9b3214ea-a3b4-43d8-88b7-01d044b63bb4\" (UID: \"9b3214ea-a3b4-43d8-88b7-01d044b63bb4\") "
Oct 30 01:43:44.624620 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.624587    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-maas-api-openshift-ca" pod "9b3214ea-a3b4-43d8-88b7-01d044b63bb4" (UID: "9b3214ea-a3b4-43d8-88b7-01d044b63bb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:43:44.625107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.625087    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgqql\" (UniqueName: \"kubernetes.io/projected/351e25a6-877b-47ad-8206-b383bf0bad41-kube-api-access-fgqql\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:44.625256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.625110    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/351e25a6-877b-47ad-8206-b383bf0bad41-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:44.625256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.625122    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:44.626576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.626544    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-kube-api-access-wcml7" pod "9b3214ea-a3b4-43d8-88b7-01d044b63bb4" (UID: "9b3214ea-a3b4-43d8-88b7-01d044b63bb4"). InnerVolumeSpecName "kube-api-access-wcml7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:43:44.626661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.626600    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-maas-api-xks-ca" pod "9b3214ea-a3b4-43d8-88b7-01d044b63bb4" (UID: "9b3214ea-a3b4-43d8-88b7-01d044b63bb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:43:44.726350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.726310    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcml7\" (UniqueName: \"kubernetes.io/projected/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-kube-api-access-wcml7\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:44.726350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:44.726345    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b3214ea-a3b4-43d8-88b7-01d044b63bb4-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:45.291255 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:45.291223    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"
Oct 30 01:43:45.291482 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:45.291215    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h" event={"ID":"351e25a6-877b-47ad-8206-b383bf0bad41","Type":"ContainerDied","Data":"744744251fe452e225a6e2233a15854ff03902e9448f265414a4ec21b60aa96b"}
Oct 30 01:43:45.291482 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:45.291391    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="744744251fe452e225a6e2233a15854ff03902e9448f265414a4ec21b60aa96b"
Oct 30 01:43:45.293301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:45.293264    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4" event={"ID":"9b3214ea-a3b4-43d8-88b7-01d044b63bb4","Type":"ContainerDied","Data":"7e507de8217e44ea77b319780d9e4e98b75fabdadf89e6c56281b6370afd6e10"}
Oct 30 01:43:45.293455 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:45.293318    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e507de8217e44ea77b319780d9e4e98b75fabdadf89e6c56281b6370afd6e10"
Oct 30 01:43:45.293455 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:45.293338    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"
Oct 30 01:43:51.346528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:51.346488    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:43:55.339379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.339336    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk" podUID="bac33d7a-c77f-4bfd-9881-790ba5559455" containerName="main" containerID="cri-o://08592d6661b9dfe04349e25d4be58be24e8208b6c4388893ac99981b7e7f2e45" gracePeriod=45
Oct 30 01:43:55.407484 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.407447    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"]
Oct 30 01:43:55.692276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.691411    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:43:55.750504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.750458    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"]
Oct 30 01:43:55.751805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.751778    4425 cpu_manager.go:414] "containerMap: removing container" podUID="351e25a6-877b-47ad-8206-b383bf0bad41" containerName="revoke-keys"
Oct 30 01:43:55.751885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.751810    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351e25a6-877b-47ad-8206-b383bf0bad41" containerName="revoke-keys"
Oct 30 01:43:55.751885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.751859    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9b3214ea-a3b4-43d8-88b7-01d044b63bb4" containerName="revoke-keys"
Oct 30 01:43:55.751885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.751867    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b3214ea-a3b4-43d8-88b7-01d044b63bb4" containerName="revoke-keys"
Oct 30 01:43:55.751981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.751912    4425 cpu_manager.go:414] "containerMap: removing container" podUID="bac33d7a-c77f-4bfd-9881-790ba5559455" containerName="main"
Oct 30 01:43:55.751981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.751920    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac33d7a-c77f-4bfd-9881-790ba5559455" containerName="main"
Oct 30 01:43:55.752098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.752086    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="351e25a6-877b-47ad-8206-b383bf0bad41" containerName="revoke-keys"
Oct 30 01:43:55.752136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.752112    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b3214ea-a3b4-43d8-88b7-01d044b63bb4" containerName="revoke-keys"
Oct 30 01:43:55.752136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.752127    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac33d7a-c77f-4bfd-9881-790ba5559455" containerName="main"
Oct 30 01:43:55.756994 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.756968    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"
Oct 30 01:43:55.792585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.792546    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"]
Oct 30 01:43:55.793875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.793835    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:43:55.794433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.794183    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:43:55.803274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.803239    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bac33d7a-c77f-4bfd-9881-790ba5559455-tls-certs\" (UniqueName: \"kubernetes.io/secret/bac33d7a-c77f-4bfd-9881-790ba5559455-tls-certs\") pod \"bac33d7a-c77f-4bfd-9881-790ba5559455\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") "
Oct 30 01:43:55.803435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.803349    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-model-cache\") pod \"bac33d7a-c77f-4bfd-9881-790ba5559455\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") "
Oct 30 01:43:55.803516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.803460    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-home\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-home\") pod \"bac33d7a-c77f-4bfd-9881-790ba5559455\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") "
Oct 30 01:43:55.803516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.803489    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-dshm\") pod \"bac33d7a-c77f-4bfd-9881-790ba5559455\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") "
Oct 30 01:43:55.803637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.803525    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-tmp-dir\") pod \"bac33d7a-c77f-4bfd-9881-790ba5559455\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") "
Oct 30 01:43:55.803637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.803625    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac33d7a-c77f-4bfd-9881-790ba5559455-kube-api-access-bqzfw\" (UniqueName: \"kubernetes.io/projected/bac33d7a-c77f-4bfd-9881-790ba5559455-kube-api-access-bqzfw\") pod \"bac33d7a-c77f-4bfd-9881-790ba5559455\" (UID: \"bac33d7a-c77f-4bfd-9881-790ba5559455\") "
Oct 30 01:43:55.803997 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.803969    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-home" pod "bac33d7a-c77f-4bfd-9881-790ba5559455" (UID: "bac33d7a-c77f-4bfd-9881-790ba5559455"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:43:55.803997 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.803983    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-model-cache" pod "bac33d7a-c77f-4bfd-9881-790ba5559455" (UID: "bac33d7a-c77f-4bfd-9881-790ba5559455"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:43:55.804190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.804018    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq\" (UID: \"e82dcb94-05fe-44ee-99b9-3d22a8d803cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"
Oct 30 01:43:55.804190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.804076    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv9fv\" (UniqueName: \"kubernetes.io/projected/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-kube-api-access-rv9fv\") pod \"maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq\" (UID: \"e82dcb94-05fe-44ee-99b9-3d22a8d803cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"
Oct 30 01:43:55.804312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.804189    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq\" (UID: \"e82dcb94-05fe-44ee-99b9-3d22a8d803cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"
Oct 30 01:43:55.804312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.804242    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-tmp-dir" pod "bac33d7a-c77f-4bfd-9881-790ba5559455" (UID: "bac33d7a-c77f-4bfd-9881-790ba5559455"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:43:55.804546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.804520    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:55.804620 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.804548    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:55.804620 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.804561    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:55.806934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.806906    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac33d7a-c77f-4bfd-9881-790ba5559455-tls-certs" pod "bac33d7a-c77f-4bfd-9881-790ba5559455" (UID: "bac33d7a-c77f-4bfd-9881-790ba5559455"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:43:55.807742 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.807715    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac33d7a-c77f-4bfd-9881-790ba5559455-kube-api-access-bqzfw" pod "bac33d7a-c77f-4bfd-9881-790ba5559455" (UID: "bac33d7a-c77f-4bfd-9881-790ba5559455"). InnerVolumeSpecName "kube-api-access-bqzfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:43:55.808249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.808223    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-dshm" pod "bac33d7a-c77f-4bfd-9881-790ba5559455" (UID: "bac33d7a-c77f-4bfd-9881-790ba5559455"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:43:55.905897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.905799    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv9fv\" (UniqueName: \"kubernetes.io/projected/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-kube-api-access-rv9fv\") pod \"maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq\" (UID: \"e82dcb94-05fe-44ee-99b9-3d22a8d803cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"
Oct 30 01:43:55.905897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.905885    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq\" (UID: \"e82dcb94-05fe-44ee-99b9-3d22a8d803cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"
Oct 30 01:43:55.906123 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.906062    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq\" (UID: \"e82dcb94-05fe-44ee-99b9-3d22a8d803cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"
Oct 30 01:43:55.906123 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.906108    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bac33d7a-c77f-4bfd-9881-790ba5559455-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:55.906123 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.906119    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac33d7a-c77f-4bfd-9881-790ba5559455-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:55.906245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.906128    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqzfw\" (UniqueName: \"kubernetes.io/projected/bac33d7a-c77f-4bfd-9881-790ba5559455-kube-api-access-bqzfw\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:55.906728 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.906701    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq\" (UID: \"e82dcb94-05fe-44ee-99b9-3d22a8d803cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"
Oct 30 01:43:55.908512 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.908493    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq\" (UID: \"e82dcb94-05fe-44ee-99b9-3d22a8d803cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"
Oct 30 01:43:55.929518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:55.929485    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv9fv\" (UniqueName: \"kubernetes.io/projected/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-kube-api-access-rv9fv\") pod \"maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq\" (UID: \"e82dcb94-05fe-44ee-99b9-3d22a8d803cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"
Oct 30 01:43:56.076025 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:56.075989    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"
Oct 30 01:43:56.396237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:56.396197    4425 generic.go:356] "Generic (PLEG): container finished" podID="bac33d7a-c77f-4bfd-9881-790ba5559455" containerID="08592d6661b9dfe04349e25d4be58be24e8208b6c4388893ac99981b7e7f2e45" exitCode=0
Oct 30 01:43:56.396685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:56.396241    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk" event={"ID":"bac33d7a-c77f-4bfd-9881-790ba5559455","Type":"ContainerDied","Data":"08592d6661b9dfe04349e25d4be58be24e8208b6c4388893ac99981b7e7f2e45"}
Oct 30 01:43:56.396685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:56.396259    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"
Oct 30 01:43:56.396685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:56.396296    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk" event={"ID":"bac33d7a-c77f-4bfd-9881-790ba5559455","Type":"ContainerDied","Data":"6ef97a64bf17e04392c7df97af33e644149c44d3d13d99fa569ea3a1c7646cba"}
Oct 30 01:43:56.396685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:56.396320    4425 scope.go:122] "RemoveContainer" containerID="08592d6661b9dfe04349e25d4be58be24e8208b6c4388893ac99981b7e7f2e45"
Oct 30 01:43:56.407856 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:56.407834    4425 scope.go:122] "RemoveContainer" containerID="08592d6661b9dfe04349e25d4be58be24e8208b6c4388893ac99981b7e7f2e45"
Oct 30 01:43:56.408177 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:43:56.408140    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08592d6661b9dfe04349e25d4be58be24e8208b6c4388893ac99981b7e7f2e45\": container with ID starting with 08592d6661b9dfe04349e25d4be58be24e8208b6c4388893ac99981b7e7f2e45 not found: ID does not exist" containerID="08592d6661b9dfe04349e25d4be58be24e8208b6c4388893ac99981b7e7f2e45"
Oct 30 01:43:56.408255 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:56.408192    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08592d6661b9dfe04349e25d4be58be24e8208b6c4388893ac99981b7e7f2e45"} err="failed to get container status \"08592d6661b9dfe04349e25d4be58be24e8208b6c4388893ac99981b7e7f2e45\": rpc error: code = NotFound desc = could not find container \"08592d6661b9dfe04349e25d4be58be24e8208b6c4388893ac99981b7e7f2e45\": container with ID starting with 08592d6661b9dfe04349e25d4be58be24e8208b6c4388893ac99981b7e7f2e45 not found: ID does not exist"
Oct 30 01:43:56.418703 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:56.418678    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"]
Oct 30 01:43:56.420523 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:43:56.420496    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82dcb94_05fe_44ee_99b9_3d22a8d803cc.slice/crio-277b34e11dd386103cf4e7a0cbffc61faa9bf8e64b5c2b8ad1159945fb80e1c0 WatchSource:0}: Error finding container 277b34e11dd386103cf4e7a0cbffc61faa9bf8e64b5c2b8ad1159945fb80e1c0: Status 404 returned error can't find the container with id 277b34e11dd386103cf4e7a0cbffc61faa9bf8e64b5c2b8ad1159945fb80e1c0
Oct 30 01:43:56.477218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:56.477178    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"]
Oct 30 01:43:56.481273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:56.481246    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk"]
Oct 30 01:43:56.573727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:56.573673    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-db4dd2/test-model-db4dd2-kserve-5b7d4677ff-p7jvk" podUID="bac33d7a-c77f-4bfd-9881-790ba5559455" containerName="main" probeResult="failure" output="Get \"https://10.133.1.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:43:57.405794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:57.405762    4425 generic.go:356] "Generic (PLEG): container finished" podID="e82dcb94-05fe-44ee-99b9-3d22a8d803cc" containerID="504d9ff614a544627a7fdde25a0cf8099ed25a5ee11ff5a8136fbbe37763f49b" exitCode=0
Oct 30 01:43:57.406301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:57.405846    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq" event={"ID":"e82dcb94-05fe-44ee-99b9-3d22a8d803cc","Type":"ContainerDied","Data":"504d9ff614a544627a7fdde25a0cf8099ed25a5ee11ff5a8136fbbe37763f49b"}
Oct 30 01:43:57.406301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:57.405902    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq" event={"ID":"e82dcb94-05fe-44ee-99b9-3d22a8d803cc","Type":"ContainerStarted","Data":"277b34e11dd386103cf4e7a0cbffc61faa9bf8e64b5c2b8ad1159945fb80e1c0"}
Oct 30 01:43:57.426360 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:57.426332    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac33d7a-c77f-4bfd-9881-790ba5559455" path="/var/lib/kubelet/pods/bac33d7a-c77f-4bfd-9881-790ba5559455/volumes"
Oct 30 01:43:58.580559 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:58.580531    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"
Oct 30 01:43:58.649730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:58.649696    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-maas-api-openshift-ca\") pod \"e82dcb94-05fe-44ee-99b9-3d22a8d803cc\" (UID: \"e82dcb94-05fe-44ee-99b9-3d22a8d803cc\") "
Oct 30 01:43:58.649928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:58.649873    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-kube-api-access-rv9fv\" (UniqueName: \"kubernetes.io/projected/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-kube-api-access-rv9fv\") pod \"e82dcb94-05fe-44ee-99b9-3d22a8d803cc\" (UID: \"e82dcb94-05fe-44ee-99b9-3d22a8d803cc\") "
Oct 30 01:43:58.650005 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:58.649929    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-maas-api-xks-ca\") pod \"e82dcb94-05fe-44ee-99b9-3d22a8d803cc\" (UID: \"e82dcb94-05fe-44ee-99b9-3d22a8d803cc\") "
Oct 30 01:43:58.650140 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:58.650111    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-maas-api-openshift-ca" pod "e82dcb94-05fe-44ee-99b9-3d22a8d803cc" (UID: "e82dcb94-05fe-44ee-99b9-3d22a8d803cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:43:58.650500 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:58.650477    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:58.652499 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:58.652464    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-maas-api-xks-ca" pod "e82dcb94-05fe-44ee-99b9-3d22a8d803cc" (UID: "e82dcb94-05fe-44ee-99b9-3d22a8d803cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:43:58.652630 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:58.652469    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-kube-api-access-rv9fv" pod "e82dcb94-05fe-44ee-99b9-3d22a8d803cc" (UID: "e82dcb94-05fe-44ee-99b9-3d22a8d803cc"). InnerVolumeSpecName "kube-api-access-rv9fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:43:58.751586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:58.751548    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv9fv\" (UniqueName: \"kubernetes.io/projected/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-kube-api-access-rv9fv\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:58.751586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:58.751583    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82dcb94-05fe-44ee-99b9-3d22a8d803cc-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:43:59.426588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:59.426558    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"
Oct 30 01:43:59.426782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:59.426599    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq" event={"ID":"e82dcb94-05fe-44ee-99b9-3d22a8d803cc","Type":"ContainerDied","Data":"277b34e11dd386103cf4e7a0cbffc61faa9bf8e64b5c2b8ad1159945fb80e1c0"}
Oct 30 01:43:59.426782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:43:59.426652    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="277b34e11dd386103cf4e7a0cbffc61faa9bf8e64b5c2b8ad1159945fb80e1c0"
Oct 30 01:44:01.114785 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.114748    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"]
Oct 30 01:44:01.115454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.115046    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8" podUID="709fe2e0-eb31-41da-bf22-daca403ffbbb" containerName="maas-api" containerID="cri-o://0e5e8d8f05b5691e24012d14a6bb3e0d2c46ed83e24a8c72524c8af26b2d78c1" gracePeriod=30
Oct 30 01:44:01.422879 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.422847    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:44:01.445034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.445003    4425 generic.go:356] "Generic (PLEG): container finished" podID="709fe2e0-eb31-41da-bf22-daca403ffbbb" containerID="0e5e8d8f05b5691e24012d14a6bb3e0d2c46ed83e24a8c72524c8af26b2d78c1" exitCode=0
Oct 30 01:44:01.445270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.445094    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"
Oct 30 01:44:01.445270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.445102    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8" event={"ID":"709fe2e0-eb31-41da-bf22-daca403ffbbb","Type":"ContainerDied","Data":"0e5e8d8f05b5691e24012d14a6bb3e0d2c46ed83e24a8c72524c8af26b2d78c1"}
Oct 30 01:44:01.445270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.445178    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8" event={"ID":"709fe2e0-eb31-41da-bf22-daca403ffbbb","Type":"ContainerDied","Data":"cda95ab63ea555e532d99af5b92ec0f2d0a424969805872bc917a6e2a938e7ea"}
Oct 30 01:44:01.445270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.445216    4425 scope.go:122] "RemoveContainer" containerID="0e5e8d8f05b5691e24012d14a6bb3e0d2c46ed83e24a8c72524c8af26b2d78c1"
Oct 30 01:44:01.458592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.458562    4425 scope.go:122] "RemoveContainer" containerID="0e5e8d8f05b5691e24012d14a6bb3e0d2c46ed83e24a8c72524c8af26b2d78c1"
Oct 30 01:44:01.458911 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:44:01.458887    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5e8d8f05b5691e24012d14a6bb3e0d2c46ed83e24a8c72524c8af26b2d78c1\": container with ID starting with 0e5e8d8f05b5691e24012d14a6bb3e0d2c46ed83e24a8c72524c8af26b2d78c1 not found: ID does not exist" containerID="0e5e8d8f05b5691e24012d14a6bb3e0d2c46ed83e24a8c72524c8af26b2d78c1"
Oct 30 01:44:01.459009 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.458919    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5e8d8f05b5691e24012d14a6bb3e0d2c46ed83e24a8c72524c8af26b2d78c1"} err="failed to get container status \"0e5e8d8f05b5691e24012d14a6bb3e0d2c46ed83e24a8c72524c8af26b2d78c1\": rpc error: code = NotFound desc = could not find container \"0e5e8d8f05b5691e24012d14a6bb3e0d2c46ed83e24a8c72524c8af26b2d78c1\": container with ID starting with 0e5e8d8f05b5691e24012d14a6bb3e0d2c46ed83e24a8c72524c8af26b2d78c1 not found: ID does not exist"
Oct 30 01:44:01.492612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.492581    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/709fe2e0-eb31-41da-bf22-daca403ffbbb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/709fe2e0-eb31-41da-bf22-daca403ffbbb-maas-api-tls\") pod \"709fe2e0-eb31-41da-bf22-daca403ffbbb\" (UID: \"709fe2e0-eb31-41da-bf22-daca403ffbbb\") "
Oct 30 01:44:01.492802 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.492638    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/709fe2e0-eb31-41da-bf22-daca403ffbbb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/709fe2e0-eb31-41da-bf22-daca403ffbbb-openshift-service-ca\") pod \"709fe2e0-eb31-41da-bf22-daca403ffbbb\" (UID: \"709fe2e0-eb31-41da-bf22-daca403ffbbb\") "
Oct 30 01:44:01.492802 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.492670    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709fe2e0-eb31-41da-bf22-daca403ffbbb-kube-api-access-6hfw8\" (UniqueName: \"kubernetes.io/projected/709fe2e0-eb31-41da-bf22-daca403ffbbb-kube-api-access-6hfw8\") pod \"709fe2e0-eb31-41da-bf22-daca403ffbbb\" (UID: \"709fe2e0-eb31-41da-bf22-daca403ffbbb\") "
Oct 30 01:44:01.492917 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.492826    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/709fe2e0-eb31-41da-bf22-daca403ffbbb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/709fe2e0-eb31-41da-bf22-daca403ffbbb-metrics-certs\") pod \"709fe2e0-eb31-41da-bf22-daca403ffbbb\" (UID: \"709fe2e0-eb31-41da-bf22-daca403ffbbb\") "
Oct 30 01:44:01.494623 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.494590    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709fe2e0-eb31-41da-bf22-daca403ffbbb-openshift-service-ca" pod "709fe2e0-eb31-41da-bf22-daca403ffbbb" (UID: "709fe2e0-eb31-41da-bf22-daca403ffbbb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:44:01.497768 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.497740    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709fe2e0-eb31-41da-bf22-daca403ffbbb-kube-api-access-6hfw8" pod "709fe2e0-eb31-41da-bf22-daca403ffbbb" (UID: "709fe2e0-eb31-41da-bf22-daca403ffbbb"). InnerVolumeSpecName "kube-api-access-6hfw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:44:01.497894 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.497855    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/709fe2e0-eb31-41da-bf22-daca403ffbbb-metrics-certs" pod "709fe2e0-eb31-41da-bf22-daca403ffbbb" (UID: "709fe2e0-eb31-41da-bf22-daca403ffbbb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:44:01.501568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.501537    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/709fe2e0-eb31-41da-bf22-daca403ffbbb-maas-api-tls" pod "709fe2e0-eb31-41da-bf22-daca403ffbbb" (UID: "709fe2e0-eb31-41da-bf22-daca403ffbbb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:44:01.594832 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.594793    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/709fe2e0-eb31-41da-bf22-daca403ffbbb-maas-api-tls\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:01.594832 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.594830    4425 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/709fe2e0-eb31-41da-bf22-daca403ffbbb-openshift-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:01.595097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.594845    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hfw8\" (UniqueName: \"kubernetes.io/projected/709fe2e0-eb31-41da-bf22-daca403ffbbb-kube-api-access-6hfw8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:01.595097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.594857    4425 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/709fe2e0-eb31-41da-bf22-daca403ffbbb-metrics-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:01.815102 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.815070    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"]
Oct 30 01:44:01.819024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.819000    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0a2c-6c4cc95df6-l59g8"]
Oct 30 01:44:01.899831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.899801    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"]
Oct 30 01:44:01.915550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:01.915520    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ab0a2c-l2hhr"]
Oct 30 01:44:03.432167 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.432128    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="709fe2e0-eb31-41da-bf22-daca403ffbbb" path="/var/lib/kubelet/pods/709fe2e0-eb31-41da-bf22-daca403ffbbb/volumes"
Oct 30 01:44:03.432852 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.432824    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a38f99-0f6f-4f5d-a7a4-412c85e63c14" path="/var/lib/kubelet/pods/97a38f99-0f6f-4f5d-a7a4-412c85e63c14/volumes"
Oct 30 01:44:03.698570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.698490    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"]
Oct 30 01:44:03.699386 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.699365    4425 cpu_manager.go:414] "containerMap: removing container" podUID="e82dcb94-05fe-44ee-99b9-3d22a8d803cc" containerName="revoke-keys"
Oct 30 01:44:03.699445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.699386    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e82dcb94-05fe-44ee-99b9-3d22a8d803cc" containerName="revoke-keys"
Oct 30 01:44:03.699445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.699408    4425 cpu_manager.go:414] "containerMap: removing container" podUID="709fe2e0-eb31-41da-bf22-daca403ffbbb" containerName="maas-api"
Oct 30 01:44:03.699445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.699414    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709fe2e0-eb31-41da-bf22-daca403ffbbb" containerName="maas-api"
Oct 30 01:44:03.699549 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.699541    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e82dcb94-05fe-44ee-99b9-3d22a8d803cc" containerName="revoke-keys"
Oct 30 01:44:03.699584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.699552    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="709fe2e0-eb31-41da-bf22-daca403ffbbb" containerName="maas-api"
Oct 30 01:44:03.704780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.704763    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"
Oct 30 01:44:03.718679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.718660    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:44:03.723860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.723831    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"]
Oct 30 01:44:03.738305 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.728275    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:44:03.834178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.834114    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3178cd7-499d-4efa-a7a9-26bc93371fe8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9\" (UID: \"f3178cd7-499d-4efa-a7a9-26bc93371fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"
Oct 30 01:44:03.834361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.834321    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-778kv\" (UniqueName: \"kubernetes.io/projected/f3178cd7-499d-4efa-a7a9-26bc93371fe8-kube-api-access-778kv\") pod \"maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9\" (UID: \"f3178cd7-499d-4efa-a7a9-26bc93371fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"
Oct 30 01:44:03.834457 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.834440    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3178cd7-499d-4efa-a7a9-26bc93371fe8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9\" (UID: \"f3178cd7-499d-4efa-a7a9-26bc93371fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"
Oct 30 01:44:03.935714 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.935678    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3178cd7-499d-4efa-a7a9-26bc93371fe8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9\" (UID: \"f3178cd7-499d-4efa-a7a9-26bc93371fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"
Oct 30 01:44:03.935905 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.935808    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3178cd7-499d-4efa-a7a9-26bc93371fe8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9\" (UID: \"f3178cd7-499d-4efa-a7a9-26bc93371fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"
Oct 30 01:44:03.935905 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.935888    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-778kv\" (UniqueName: \"kubernetes.io/projected/f3178cd7-499d-4efa-a7a9-26bc93371fe8-kube-api-access-778kv\") pod \"maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9\" (UID: \"f3178cd7-499d-4efa-a7a9-26bc93371fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"
Oct 30 01:44:03.936608 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.936582    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3178cd7-499d-4efa-a7a9-26bc93371fe8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9\" (UID: \"f3178cd7-499d-4efa-a7a9-26bc93371fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"
Oct 30 01:44:03.938334 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.938310    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3178cd7-499d-4efa-a7a9-26bc93371fe8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9\" (UID: \"f3178cd7-499d-4efa-a7a9-26bc93371fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"
Oct 30 01:44:03.957107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:03.957058    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-778kv\" (UniqueName: \"kubernetes.io/projected/f3178cd7-499d-4efa-a7a9-26bc93371fe8-kube-api-access-778kv\") pod \"maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9\" (UID: \"f3178cd7-499d-4efa-a7a9-26bc93371fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"
Oct 30 01:44:04.021133 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:04.021104    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"
Oct 30 01:44:04.348192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:04.347834    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"]
Oct 30 01:44:04.348739 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:44:04.348708    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3178cd7_499d_4efa_a7a9_26bc93371fe8.slice/crio-93109a4a4a041f66f9e2f304d5d9dae0c80a39f1a8b80466086ce4fa1a8204a5 WatchSource:0}: Error finding container 93109a4a4a041f66f9e2f304d5d9dae0c80a39f1a8b80466086ce4fa1a8204a5: Status 404 returned error can't find the container with id 93109a4a4a041f66f9e2f304d5d9dae0c80a39f1a8b80466086ce4fa1a8204a5
Oct 30 01:44:04.471836 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:04.471801    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9" event={"ID":"f3178cd7-499d-4efa-a7a9-26bc93371fe8","Type":"ContainerStarted","Data":"93109a4a4a041f66f9e2f304d5d9dae0c80a39f1a8b80466086ce4fa1a8204a5"}
Oct 30 01:44:05.480559 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:05.480530    4425 generic.go:356] "Generic (PLEG): container finished" podID="f3178cd7-499d-4efa-a7a9-26bc93371fe8" containerID="77fa4e1ca2ce3bb5690c7ea6a9a666a12cfa0487765cffb5302c23734b6dc642" exitCode=0
Oct 30 01:44:05.480883 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:05.480613    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9" event={"ID":"f3178cd7-499d-4efa-a7a9-26bc93371fe8","Type":"ContainerDied","Data":"77fa4e1ca2ce3bb5690c7ea6a9a666a12cfa0487765cffb5302c23734b6dc642"}
Oct 30 01:44:06.265127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:06.265089    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"]
Oct 30 01:44:06.276400 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:06.276366    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"]
Oct 30 01:44:06.655756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:06.655729    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"
Oct 30 01:44:06.777855 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:06.777818    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3178cd7-499d-4efa-a7a9-26bc93371fe8-kube-api-access-778kv\" (UniqueName: \"kubernetes.io/projected/f3178cd7-499d-4efa-a7a9-26bc93371fe8-kube-api-access-778kv\") pod \"f3178cd7-499d-4efa-a7a9-26bc93371fe8\" (UID: \"f3178cd7-499d-4efa-a7a9-26bc93371fe8\") "
Oct 30 01:44:06.778203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:06.778151    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3178cd7-499d-4efa-a7a9-26bc93371fe8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3178cd7-499d-4efa-a7a9-26bc93371fe8-maas-api-xks-ca\") pod \"f3178cd7-499d-4efa-a7a9-26bc93371fe8\" (UID: \"f3178cd7-499d-4efa-a7a9-26bc93371fe8\") "
Oct 30 01:44:06.778259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:06.778244    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3178cd7-499d-4efa-a7a9-26bc93371fe8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3178cd7-499d-4efa-a7a9-26bc93371fe8-maas-api-openshift-ca\") pod \"f3178cd7-499d-4efa-a7a9-26bc93371fe8\" (UID: \"f3178cd7-499d-4efa-a7a9-26bc93371fe8\") "
Oct 30 01:44:06.778601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:06.778573    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3178cd7-499d-4efa-a7a9-26bc93371fe8-maas-api-openshift-ca" pod "f3178cd7-499d-4efa-a7a9-26bc93371fe8" (UID: "f3178cd7-499d-4efa-a7a9-26bc93371fe8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:44:06.778847 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:06.778827    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3178cd7-499d-4efa-a7a9-26bc93371fe8-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:06.779962 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:06.779942    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3178cd7-499d-4efa-a7a9-26bc93371fe8-kube-api-access-778kv" pod "f3178cd7-499d-4efa-a7a9-26bc93371fe8" (UID: "f3178cd7-499d-4efa-a7a9-26bc93371fe8"). InnerVolumeSpecName "kube-api-access-778kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:44:06.780147 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:06.780126    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3178cd7-499d-4efa-a7a9-26bc93371fe8-maas-api-xks-ca" pod "f3178cd7-499d-4efa-a7a9-26bc93371fe8" (UID: "f3178cd7-499d-4efa-a7a9-26bc93371fe8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:44:06.880126 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:06.880033    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-778kv\" (UniqueName: \"kubernetes.io/projected/f3178cd7-499d-4efa-a7a9-26bc93371fe8-kube-api-access-778kv\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:06.880126 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:06.880067    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3178cd7-499d-4efa-a7a9-26bc93371fe8-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:07.458800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.458770    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"]
Oct 30 01:44:07.459601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.459585    4425 cpu_manager.go:414] "containerMap: removing container" podUID="f3178cd7-499d-4efa-a7a9-26bc93371fe8" containerName="revoke-keys"
Oct 30 01:44:07.459668 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.459603    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3178cd7-499d-4efa-a7a9-26bc93371fe8" containerName="revoke-keys"
Oct 30 01:44:07.459731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.459721    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3178cd7-499d-4efa-a7a9-26bc93371fe8" containerName="revoke-keys"
Oct 30 01:44:07.463761 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.463741    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.486584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.486552    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.486764 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.486743    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llb6s\" (UniqueName: \"kubernetes.io/projected/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-kube-api-access-llb6s\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.486980 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.486955    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.487172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.487128    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.487313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.487216    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.487313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.487273    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.487495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.487421    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.487585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.487534    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.487585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.487564    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.498895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.498862    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"]
Oct 30 01:44:07.508820 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.508794    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9" event={"ID":"f3178cd7-499d-4efa-a7a9-26bc93371fe8","Type":"ContainerDied","Data":"93109a4a4a041f66f9e2f304d5d9dae0c80a39f1a8b80466086ce4fa1a8204a5"}
Oct 30 01:44:07.508936 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.508843    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93109a4a4a041f66f9e2f304d5d9dae0c80a39f1a8b80466086ce4fa1a8204a5"
Oct 30 01:44:07.508936 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.508908    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"
Oct 30 01:44:07.556996 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:44:07.556961    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:44:07.588524 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.588493    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.588688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.588543    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.588688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.588586    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.588688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.588619    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.588688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.588645    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.588688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.588664    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.588959 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.588738    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.588959 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.588767    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llb6s\" (UniqueName: \"kubernetes.io/projected/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-kube-api-access-llb6s\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.588959 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.588862    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.589137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.589035    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.589137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.589069    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.589277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.589225    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.589378 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.589357    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.589968 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.589944    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.591000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.590967    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.591250 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.591234    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.611392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.611369    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.617081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.617061    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llb6s\" (UniqueName: \"kubernetes.io/projected/50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3-kube-api-access-llb6s\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cxntb\" (UID: \"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:07.782940 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:07.782910    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:08.125474 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:44:08.125443    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e0627e_b7b6_4b7e_9adb_46e0d1e0b6b3.slice/crio-631d8aa1a23e4ec0382c86b9a9bdcde6d249eac5d76314a8c971a455b4310bef WatchSource:0}: Error finding container 631d8aa1a23e4ec0382c86b9a9bdcde6d249eac5d76314a8c971a455b4310bef: Status 404 returned error can't find the container with id 631d8aa1a23e4ec0382c86b9a9bdcde6d249eac5d76314a8c971a455b4310bef
Oct 30 01:44:08.126273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:08.126242    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"]
Oct 30 01:44:08.128707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:08.128670    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:44:08.128780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:08.128740    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:44:08.128780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:08.128770    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:44:08.517836 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:08.517806    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb" event={"ID":"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3","Type":"ContainerStarted","Data":"2f1e6d1cf318316bc10bb3dab159bcc4f4ef08323ed94dc9eb3f67d790133035"}
Oct 30 01:44:08.518006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:08.517857    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb" event={"ID":"50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3","Type":"ContainerStarted","Data":"631d8aa1a23e4ec0382c86b9a9bdcde6d249eac5d76314a8c971a455b4310bef"}
Oct 30 01:44:08.565650 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:08.565599    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb" podStartSLOduration=1.565585779 podStartE2EDuration="1.565585779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:08.558684318 +0000 UTC m=+3757.308339628" watchObservedRunningTime="2026-10-30 01:44:08.565585779 +0000 UTC m=+3757.315241171"
Oct 30 01:44:08.783826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:08.783751    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:09.785850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:09.785813    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cxntb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" start-of-body=
Oct 30 01:44:09.786268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:09.785885    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb" podUID="50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused"
Oct 30 01:44:10.440659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.440618    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4" podUID="5b835d5c-e76c-4c40-a74c-ac9b6f5125af" containerName="main" containerID="cri-o://fc5899373694925328833cc7797196f3afd8c05e9b1ed520e8c326b9e3856288" gracePeriod=45
Oct 30 01:44:10.784230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.784194    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:44:10.784394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.784281    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cxntb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" start-of-body=
Oct 30 01:44:10.784394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.784340    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb" podUID="50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused"
Oct 30 01:44:10.836421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.836391    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-tls-certs\") pod \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") "
Oct 30 01:44:10.836902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.836501    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-dshm\") pod \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") "
Oct 30 01:44:10.836902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.836605    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-model-cache\") pod \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") "
Oct 30 01:44:10.836902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.836650    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-kube-api-access-pd5np\" (UniqueName: \"kubernetes.io/projected/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-kube-api-access-pd5np\") pod \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") "
Oct 30 01:44:10.836902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.836725    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-tmp-dir\") pod \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") "
Oct 30 01:44:10.836902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.836798    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-home\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-home\") pod \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\" (UID: \"5b835d5c-e76c-4c40-a74c-ac9b6f5125af\") "
Oct 30 01:44:10.836902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.836890    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-model-cache" pod "5b835d5c-e76c-4c40-a74c-ac9b6f5125af" (UID: "5b835d5c-e76c-4c40-a74c-ac9b6f5125af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:10.837295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.837040    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-tmp-dir" pod "5b835d5c-e76c-4c40-a74c-ac9b6f5125af" (UID: "5b835d5c-e76c-4c40-a74c-ac9b6f5125af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:10.838222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.837210    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-home" pod "5b835d5c-e76c-4c40-a74c-ac9b6f5125af" (UID: "5b835d5c-e76c-4c40-a74c-ac9b6f5125af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:10.838222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.837460    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:10.839351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.839325    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-kube-api-access-pd5np" pod "5b835d5c-e76c-4c40-a74c-ac9b6f5125af" (UID: "5b835d5c-e76c-4c40-a74c-ac9b6f5125af"). InnerVolumeSpecName "kube-api-access-pd5np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:44:10.839494 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.839473    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-dshm" pod "5b835d5c-e76c-4c40-a74c-ac9b6f5125af" (UID: "5b835d5c-e76c-4c40-a74c-ac9b6f5125af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:44:10.841307 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.841277    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-tls-certs" pod "5b835d5c-e76c-4c40-a74c-ac9b6f5125af" (UID: "5b835d5c-e76c-4c40-a74c-ac9b6f5125af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:44:10.939479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.939450    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:10.939479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.939475    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:10.939479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.939483    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:10.939697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.939493    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:10.939697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:10.939502    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd5np\" (UniqueName: \"kubernetes.io/projected/5b835d5c-e76c-4c40-a74c-ac9b6f5125af-kube-api-access-pd5np\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:11.270260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.270213    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n" podUID="ab5a52e6-500d-4e2f-8e40-44b5f595388f" containerName="payload-processing" containerID="cri-o://d45b1ed6bd0100d4c4775eb39289a8b7ebb876a8f195670e4b3f158835bb19d9" gracePeriod=25
Oct 30 01:44:11.277594 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.277552    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp" podUID="e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5" containerName="payload-pre-processing" containerID="cri-o://a5e282886e0696123bb6192b45abce0cfb3bc38f130f656f1c08921903d7cecb" gracePeriod=25
Oct 30 01:44:11.544680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.544607    4425 generic.go:356] "Generic (PLEG): container finished" podID="ab5a52e6-500d-4e2f-8e40-44b5f595388f" containerID="d45b1ed6bd0100d4c4775eb39289a8b7ebb876a8f195670e4b3f158835bb19d9" exitCode=0
Oct 30 01:44:11.544834 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.544685    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n" event={"ID":"ab5a52e6-500d-4e2f-8e40-44b5f595388f","Type":"ContainerDied","Data":"d45b1ed6bd0100d4c4775eb39289a8b7ebb876a8f195670e4b3f158835bb19d9"}
Oct 30 01:44:11.546555 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.546529    4425 generic.go:356] "Generic (PLEG): container finished" podID="5b835d5c-e76c-4c40-a74c-ac9b6f5125af" containerID="fc5899373694925328833cc7797196f3afd8c05e9b1ed520e8c326b9e3856288" exitCode=0
Oct 30 01:44:11.546733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.546589    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"
Oct 30 01:44:11.546733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.546611    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4" event={"ID":"5b835d5c-e76c-4c40-a74c-ac9b6f5125af","Type":"ContainerDied","Data":"fc5899373694925328833cc7797196f3afd8c05e9b1ed520e8c326b9e3856288"}
Oct 30 01:44:11.546733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.546653    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4" event={"ID":"5b835d5c-e76c-4c40-a74c-ac9b6f5125af","Type":"ContainerDied","Data":"667cebab3fe788f71c51ee877542db2c0608fb11b085dc2d900d42dee0b044af"}
Oct 30 01:44:11.546733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.546677    4425 scope.go:122] "RemoveContainer" containerID="fc5899373694925328833cc7797196f3afd8c05e9b1ed520e8c326b9e3856288"
Oct 30 01:44:11.612851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.612829    4425 scope.go:122] "RemoveContainer" containerID="fc5899373694925328833cc7797196f3afd8c05e9b1ed520e8c326b9e3856288"
Oct 30 01:44:11.613727 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:44:11.613689    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc5899373694925328833cc7797196f3afd8c05e9b1ed520e8c326b9e3856288\": container with ID starting with fc5899373694925328833cc7797196f3afd8c05e9b1ed520e8c326b9e3856288 not found: ID does not exist" containerID="fc5899373694925328833cc7797196f3afd8c05e9b1ed520e8c326b9e3856288"
Oct 30 01:44:11.613841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.613731    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc5899373694925328833cc7797196f3afd8c05e9b1ed520e8c326b9e3856288"} err="failed to get container status \"fc5899373694925328833cc7797196f3afd8c05e9b1ed520e8c326b9e3856288\": rpc error: code = NotFound desc = could not find container \"fc5899373694925328833cc7797196f3afd8c05e9b1ed520e8c326b9e3856288\": container with ID starting with fc5899373694925328833cc7797196f3afd8c05e9b1ed520e8c326b9e3856288 not found: ID does not exist"
Oct 30 01:44:11.619607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.619584    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"
Oct 30 01:44:11.642588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.642563    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"]
Oct 30 01:44:11.678696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.678660    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-73423b/test-model-73423b-kserve-799894db4f-scxz4"]
Oct 30 01:44:11.683920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.683894    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5a52e6-500d-4e2f-8e40-44b5f595388f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5a52e6-500d-4e2f-8e40-44b5f595388f-plugins-config-volume\") pod \"ab5a52e6-500d-4e2f-8e40-44b5f595388f\" (UID: \"ab5a52e6-500d-4e2f-8e40-44b5f595388f\") "
Oct 30 01:44:11.684079 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.684060    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5a52e6-500d-4e2f-8e40-44b5f595388f-kube-api-access-k8qmh\" (UniqueName: \"kubernetes.io/projected/ab5a52e6-500d-4e2f-8e40-44b5f595388f-kube-api-access-k8qmh\") pod \"ab5a52e6-500d-4e2f-8e40-44b5f595388f\" (UID: \"ab5a52e6-500d-4e2f-8e40-44b5f595388f\") "
Oct 30 01:44:11.684303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.684279    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5a52e6-500d-4e2f-8e40-44b5f595388f-plugins-config-volume" pod "ab5a52e6-500d-4e2f-8e40-44b5f595388f" (UID: "ab5a52e6-500d-4e2f-8e40-44b5f595388f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:44:11.684637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.684617    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab5a52e6-500d-4e2f-8e40-44b5f595388f-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:11.686194 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.686174    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5a52e6-500d-4e2f-8e40-44b5f595388f-kube-api-access-k8qmh" pod "ab5a52e6-500d-4e2f-8e40-44b5f595388f" (UID: "ab5a52e6-500d-4e2f-8e40-44b5f595388f"). InnerVolumeSpecName "kube-api-access-k8qmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:44:11.783696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.783666    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cxntb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" start-of-body=
Oct 30 01:44:11.783873 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.783727    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb" podUID="50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused"
Oct 30 01:44:11.785534 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:11.785509    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8qmh\" (UniqueName: \"kubernetes.io/projected/ab5a52e6-500d-4e2f-8e40-44b5f595388f-kube-api-access-k8qmh\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:12.582085 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:12.582052    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"
Oct 30 01:44:12.582085 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:12.582076    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n" event={"ID":"ab5a52e6-500d-4e2f-8e40-44b5f595388f","Type":"ContainerDied","Data":"39b1985fe269a336b8c9acb6d0ef3e00b19995c03ab77ae0fd75ec89e8439508"}
Oct 30 01:44:12.582616 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:12.582136    4425 scope.go:122] "RemoveContainer" containerID="d45b1ed6bd0100d4c4775eb39289a8b7ebb876a8f195670e4b3f158835bb19d9"
Oct 30 01:44:12.630984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:12.630961    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"]
Oct 30 01:44:12.640201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:12.640172    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-db4dd2-58dc759584-t5m2n"]
Oct 30 01:44:12.784015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:12.783978    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cxntb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" start-of-body=
Oct 30 01:44:12.784199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:12.784038    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb" podUID="50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused"
Oct 30 01:44:13.194106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:13.194068    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp" podUID="e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:44:13.426448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:13.426419    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b835d5c-e76c-4c40-a74c-ac9b6f5125af" path="/var/lib/kubelet/pods/5b835d5c-e76c-4c40-a74c-ac9b6f5125af/volumes"
Oct 30 01:44:13.426908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:13.426892    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5a52e6-500d-4e2f-8e40-44b5f595388f" path="/var/lib/kubelet/pods/ab5a52e6-500d-4e2f-8e40-44b5f595388f/volumes"
Oct 30 01:44:13.783663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:13.783627    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cxntb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" start-of-body=
Oct 30 01:44:13.784081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:13.783691    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb" podUID="50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused"
Oct 30 01:44:14.278237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:14.277420    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"]
Oct 30 01:44:14.303196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:14.303150    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"]
Oct 30 01:44:14.783570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:14.783542    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cxntb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" start-of-body=
Oct 30 01:44:14.783742 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:14.783594    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb" podUID="50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused"
Oct 30 01:44:15.784111 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:15.784073    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cxntb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused" start-of-body=
Oct 30 01:44:15.784526 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:15.784142    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb" podUID="50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": dial tcp 10.133.1.106:15021: connect: connection refused"
Oct 30 01:44:16.864951 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:16.864915    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:16.865548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:16.865455    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:16.866653 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:16.866619    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb"
Oct 30 01:44:19.278578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:19.278542    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9" podUID="a2157c2a-b5cf-497e-8f76-0d4b290440ec" containerName="payload-processing" containerID="cri-o://7f3116ee376b4339eb5f69b8c675a5b042ffac72c0b493a870215694ac171335" gracePeriod=25
Oct 30 01:44:19.304588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:19.304554    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x" podUID="d47fca09-7546-4877-bd87-3dee036c13c9" containerName="payload-pre-processing" containerID="cri-o://46bc0a71d895e01e29a113d8c97fafb5e609f956aa142c11394adb8030305cc4" gracePeriod=25
Oct 30 01:44:22.479580 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.479544    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"]
Oct 30 01:44:22.480426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.480404    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ab5a52e6-500d-4e2f-8e40-44b5f595388f" containerName="payload-processing"
Oct 30 01:44:22.480426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.480427    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5a52e6-500d-4e2f-8e40-44b5f595388f" containerName="payload-processing"
Oct 30 01:44:22.480562 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.480458    4425 cpu_manager.go:414] "containerMap: removing container" podUID="5b835d5c-e76c-4c40-a74c-ac9b6f5125af" containerName="main"
Oct 30 01:44:22.480562 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.480464    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b835d5c-e76c-4c40-a74c-ac9b6f5125af" containerName="main"
Oct 30 01:44:22.480630 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.480613    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b835d5c-e76c-4c40-a74c-ac9b6f5125af" containerName="main"
Oct 30 01:44:22.480669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.480630    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5a52e6-500d-4e2f-8e40-44b5f595388f" containerName="payload-processing"
Oct 30 01:44:22.486389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.486370    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.520438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.520400    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"]
Oct 30 01:44:22.544514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.544476    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.544667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.544533    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.544725 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.544663    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.544762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.544716    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn6k2\" (UniqueName: \"kubernetes.io/projected/4c009b06-2c6a-4909-81b8-f245378e9c4f-kube-api-access-tn6k2\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.544881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.544863    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.544918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.544902    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.544967 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.544940    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.545199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.545113    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.545356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.545308    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c009b06-2c6a-4909-81b8-f245378e9c4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.646264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.646234    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.646419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.646327    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c009b06-2c6a-4909-81b8-f245378e9c4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.646419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.646392    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.646553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.646422    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.646553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.646469    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.646553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.646495    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn6k2\" (UniqueName: \"kubernetes.io/projected/4c009b06-2c6a-4909-81b8-f245378e9c4f-kube-api-access-tn6k2\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.646727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.646582    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.646727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.646638    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.646727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.646676    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.646890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.646829    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.646950 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.646921    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.647198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.647134    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c009b06-2c6a-4909-81b8-f245378e9c4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.647300 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.647205    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.647574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.647549    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.649333 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.649290    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.649435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.649332    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.682797 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.682774    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.683036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.683019    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn6k2\" (UniqueName: \"kubernetes.io/projected/4c009b06-2c6a-4909-81b8-f245378e9c4f-kube-api-access-tn6k2\") pod \"maas-default-gateway-openshift-default-fcb75c45f-ggj4n\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:22.803337 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:22.803316    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:23.130714 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:23.130688    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"]
Oct 30 01:44:23.131697 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:44:23.131670    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c009b06_2c6a_4909_81b8_f245378e9c4f.slice/crio-eccf5613fdd008bc123f554424cfcb28fb6481c4024dab5148daeb1560e1d806 WatchSource:0}: Error finding container eccf5613fdd008bc123f554424cfcb28fb6481c4024dab5148daeb1560e1d806: Status 404 returned error can't find the container with id eccf5613fdd008bc123f554424cfcb28fb6481c4024dab5148daeb1560e1d806
Oct 30 01:44:23.134439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:23.134405    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:44:23.134519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:23.134475    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:44:23.134519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:23.134502    4425 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 01:44:23.194404 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:23.194375    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp" podUID="e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:44:23.674174 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:23.674124    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n" event={"ID":"4c009b06-2c6a-4909-81b8-f245378e9c4f","Type":"ContainerStarted","Data":"587b617e80e27d95ac15731bf1871492165d1c3b42a1bdfa6bdf7f1a0cca99eb"}
Oct 30 01:44:23.674619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:23.674207    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n" event={"ID":"4c009b06-2c6a-4909-81b8-f245378e9c4f","Type":"ContainerStarted","Data":"eccf5613fdd008bc123f554424cfcb28fb6481c4024dab5148daeb1560e1d806"}
Oct 30 01:44:23.719645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:23.719586    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n" podStartSLOduration=1.719572181 podStartE2EDuration="1.719572181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:23.717663122 +0000 UTC m=+3772.467318443" watchObservedRunningTime="2026-10-30 01:44:23.719572181 +0000 UTC m=+3772.469227489"
Oct 30 01:44:23.803779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:23.803744    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:24.806108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:24.806077    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ggj4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Oct 30 01:44:24.806546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:24.806125    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n" podUID="4c009b06-2c6a-4909-81b8-f245378e9c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Oct 30 01:44:25.213441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:25.213343    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x" podUID="d47fca09-7546-4877-bd87-3dee036c13c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:44:25.213441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:25.213343    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9" podUID="a2157c2a-b5cf-497e-8f76-0d4b290440ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:44:25.804633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:25.804592    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ggj4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Oct 30 01:44:25.804839 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:25.804649    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n" podUID="4c009b06-2c6a-4909-81b8-f245378e9c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Oct 30 01:44:26.803674 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:26.803635    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ggj4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Oct 30 01:44:26.804081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:26.803685    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n" podUID="4c009b06-2c6a-4909-81b8-f245378e9c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Oct 30 01:44:27.804694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:27.804660    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ggj4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Oct 30 01:44:27.805092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:27.804710    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n" podUID="4c009b06-2c6a-4909-81b8-f245378e9c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Oct 30 01:44:28.803931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:28.803900    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ggj4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Oct 30 01:44:28.804118 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:28.803950    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n" podUID="4c009b06-2c6a-4909-81b8-f245378e9c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Oct 30 01:44:29.804667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:29.804633    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ggj4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Oct 30 01:44:29.805052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:29.804711    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n" podUID="4c009b06-2c6a-4909-81b8-f245378e9c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Oct 30 01:44:30.803873 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:30.803837    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-ggj4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused" start-of-body=
Oct 30 01:44:30.804053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:30.803897    4425 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n" podUID="4c009b06-2c6a-4909-81b8-f245378e9c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.107:15021/healthz/ready\": dial tcp 10.133.1.107:15021: connect: connection refused"
Oct 30 01:44:31.600323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.600295    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"
Oct 30 01:44:31.693414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.693386    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5-plugins-config-volume\") pod \"e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5\" (UID: \"e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5\") "
Oct 30 01:44:31.693612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.693439    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5-kube-api-access-c5vvx\" (UniqueName: \"kubernetes.io/projected/e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5-kube-api-access-c5vvx\") pod \"e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5\" (UID: \"e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5\") "
Oct 30 01:44:31.693875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.693844    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5-plugins-config-volume" pod "e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5" (UID: "e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:44:31.695644 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.695611    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5-kube-api-access-c5vvx" pod "e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5" (UID: "e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5"). InnerVolumeSpecName "kube-api-access-c5vvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:44:31.740728 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.740699    4425 generic.go:356] "Generic (PLEG): container finished" podID="e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5" containerID="a5e282886e0696123bb6192b45abce0cfb3bc38f130f656f1c08921903d7cecb" exitCode=0
Oct 30 01:44:31.740851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.740747    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"
Oct 30 01:44:31.740851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.740783    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp" event={"ID":"e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5","Type":"ContainerDied","Data":"a5e282886e0696123bb6192b45abce0cfb3bc38f130f656f1c08921903d7cecb"}
Oct 30 01:44:31.740851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.740837    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp" event={"ID":"e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5","Type":"ContainerDied","Data":"7e8b4ade888ef032ff2b2936f1042adc19ff9a4d97f3f66c9a9c66e1419e779c"}
Oct 30 01:44:31.740983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.740865    4425 scope.go:122] "RemoveContainer" containerID="a5e282886e0696123bb6192b45abce0cfb3bc38f130f656f1c08921903d7cecb"
Oct 30 01:44:31.755758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.755737    4425 scope.go:122] "RemoveContainer" containerID="a5e282886e0696123bb6192b45abce0cfb3bc38f130f656f1c08921903d7cecb"
Oct 30 01:44:31.756059 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:44:31.756038    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e282886e0696123bb6192b45abce0cfb3bc38f130f656f1c08921903d7cecb\": container with ID starting with a5e282886e0696123bb6192b45abce0cfb3bc38f130f656f1c08921903d7cecb not found: ID does not exist" containerID="a5e282886e0696123bb6192b45abce0cfb3bc38f130f656f1c08921903d7cecb"
Oct 30 01:44:31.756119 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.756070    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e282886e0696123bb6192b45abce0cfb3bc38f130f656f1c08921903d7cecb"} err="failed to get container status \"a5e282886e0696123bb6192b45abce0cfb3bc38f130f656f1c08921903d7cecb\": rpc error: code = NotFound desc = could not find container \"a5e282886e0696123bb6192b45abce0cfb3bc38f130f656f1c08921903d7cecb\": container with ID starting with a5e282886e0696123bb6192b45abce0cfb3bc38f130f656f1c08921903d7cecb not found: ID does not exist"
Oct 30 01:44:31.790705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.790676    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"]
Oct 30 01:44:31.794959 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.794931    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:31.795057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.794962    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5vvx\" (UniqueName: \"kubernetes.io/projected/e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5-kube-api-access-c5vvx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:31.799073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.799027    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-db4dd2-6544bcd895-qzkkp"]
Oct 30 01:44:31.808865 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.808845    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:31.809112 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.809093    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:31.809887 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:31.809869    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:44:33.432272 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:33.432083    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5" path="/var/lib/kubelet/pods/e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5/volumes"
Oct 30 01:44:35.213339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:35.213301    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x" podUID="d47fca09-7546-4877-bd87-3dee036c13c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:44:35.213760 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:35.213301    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9" podUID="a2157c2a-b5cf-497e-8f76-0d4b290440ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:44:37.555954 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:44:37.555919    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:44:39.613829 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.613803    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"
Oct 30 01:44:39.617396 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.617375    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"
Oct 30 01:44:39.692837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.692805    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d47fca09-7546-4877-bd87-3dee036c13c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d47fca09-7546-4877-bd87-3dee036c13c9-plugins-config-volume\") pod \"d47fca09-7546-4877-bd87-3dee036c13c9\" (UID: \"d47fca09-7546-4877-bd87-3dee036c13c9\") "
Oct 30 01:44:39.693068 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.692902    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2157c2a-b5cf-497e-8f76-0d4b290440ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2157c2a-b5cf-497e-8f76-0d4b290440ec-plugins-config-volume\") pod \"a2157c2a-b5cf-497e-8f76-0d4b290440ec\" (UID: \"a2157c2a-b5cf-497e-8f76-0d4b290440ec\") "
Oct 30 01:44:39.693068 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.693034    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d47fca09-7546-4877-bd87-3dee036c13c9-kube-api-access-qwpsp\" (UniqueName: \"kubernetes.io/projected/d47fca09-7546-4877-bd87-3dee036c13c9-kube-api-access-qwpsp\") pod \"d47fca09-7546-4877-bd87-3dee036c13c9\" (UID: \"d47fca09-7546-4877-bd87-3dee036c13c9\") "
Oct 30 01:44:39.693327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.693091    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2157c2a-b5cf-497e-8f76-0d4b290440ec-kube-api-access-ndgfj\" (UniqueName: \"kubernetes.io/projected/a2157c2a-b5cf-497e-8f76-0d4b290440ec-kube-api-access-ndgfj\") pod \"a2157c2a-b5cf-497e-8f76-0d4b290440ec\" (UID: \"a2157c2a-b5cf-497e-8f76-0d4b290440ec\") "
Oct 30 01:44:39.694270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.694242    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d47fca09-7546-4877-bd87-3dee036c13c9-plugins-config-volume" pod "d47fca09-7546-4877-bd87-3dee036c13c9" (UID: "d47fca09-7546-4877-bd87-3dee036c13c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:44:39.694435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.694303    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2157c2a-b5cf-497e-8f76-0d4b290440ec-plugins-config-volume" pod "a2157c2a-b5cf-497e-8f76-0d4b290440ec" (UID: "a2157c2a-b5cf-497e-8f76-0d4b290440ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:44:39.696396 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.696362    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47fca09-7546-4877-bd87-3dee036c13c9-kube-api-access-qwpsp" pod "d47fca09-7546-4877-bd87-3dee036c13c9" (UID: "d47fca09-7546-4877-bd87-3dee036c13c9"). InnerVolumeSpecName "kube-api-access-qwpsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:44:39.696602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.696432    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2157c2a-b5cf-497e-8f76-0d4b290440ec-kube-api-access-ndgfj" pod "a2157c2a-b5cf-497e-8f76-0d4b290440ec" (UID: "a2157c2a-b5cf-497e-8f76-0d4b290440ec"). InnerVolumeSpecName "kube-api-access-ndgfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:44:39.795238 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.795202    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d47fca09-7546-4877-bd87-3dee036c13c9-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:39.795238 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.795234    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2157c2a-b5cf-497e-8f76-0d4b290440ec-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:39.795238 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.795244    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwpsp\" (UniqueName: \"kubernetes.io/projected/d47fca09-7546-4877-bd87-3dee036c13c9-kube-api-access-qwpsp\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:39.795468 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.795252    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndgfj\" (UniqueName: \"kubernetes.io/projected/a2157c2a-b5cf-497e-8f76-0d4b290440ec-kube-api-access-ndgfj\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:39.803744 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.803714    4425 generic.go:356] "Generic (PLEG): container finished" podID="d47fca09-7546-4877-bd87-3dee036c13c9" containerID="46bc0a71d895e01e29a113d8c97fafb5e609f956aa142c11394adb8030305cc4" exitCode=0
Oct 30 01:44:39.803884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.803763    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"
Oct 30 01:44:39.803884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.803804    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x" event={"ID":"d47fca09-7546-4877-bd87-3dee036c13c9","Type":"ContainerDied","Data":"46bc0a71d895e01e29a113d8c97fafb5e609f956aa142c11394adb8030305cc4"}
Oct 30 01:44:39.803884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.803854    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x" event={"ID":"d47fca09-7546-4877-bd87-3dee036c13c9","Type":"ContainerDied","Data":"59904834fd77f7bc7afc2ed0f0a64b493500a796f3d29b15c68b958b1b18f10a"}
Oct 30 01:44:39.803884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.803880    4425 scope.go:122] "RemoveContainer" containerID="46bc0a71d895e01e29a113d8c97fafb5e609f956aa142c11394adb8030305cc4"
Oct 30 01:44:39.805563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.805539    4425 generic.go:356] "Generic (PLEG): container finished" podID="a2157c2a-b5cf-497e-8f76-0d4b290440ec" containerID="7f3116ee376b4339eb5f69b8c675a5b042ffac72c0b493a870215694ac171335" exitCode=0
Oct 30 01:44:39.805685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.805569    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9" event={"ID":"a2157c2a-b5cf-497e-8f76-0d4b290440ec","Type":"ContainerDied","Data":"7f3116ee376b4339eb5f69b8c675a5b042ffac72c0b493a870215694ac171335"}
Oct 30 01:44:39.805685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.805601    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"
Oct 30 01:44:39.805794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.805602    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9" event={"ID":"a2157c2a-b5cf-497e-8f76-0d4b290440ec","Type":"ContainerDied","Data":"a387e3642bbab125cc3cd45298af71bed806bc5cb437be4ddaab4597c3c9f169"}
Oct 30 01:44:39.815636 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.815615    4425 scope.go:122] "RemoveContainer" containerID="46bc0a71d895e01e29a113d8c97fafb5e609f956aa142c11394adb8030305cc4"
Oct 30 01:44:39.815887 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:44:39.815869    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46bc0a71d895e01e29a113d8c97fafb5e609f956aa142c11394adb8030305cc4\": container with ID starting with 46bc0a71d895e01e29a113d8c97fafb5e609f956aa142c11394adb8030305cc4 not found: ID does not exist" containerID="46bc0a71d895e01e29a113d8c97fafb5e609f956aa142c11394adb8030305cc4"
Oct 30 01:44:39.815949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.815895    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bc0a71d895e01e29a113d8c97fafb5e609f956aa142c11394adb8030305cc4"} err="failed to get container status \"46bc0a71d895e01e29a113d8c97fafb5e609f956aa142c11394adb8030305cc4\": rpc error: code = NotFound desc = could not find container \"46bc0a71d895e01e29a113d8c97fafb5e609f956aa142c11394adb8030305cc4\": container with ID starting with 46bc0a71d895e01e29a113d8c97fafb5e609f956aa142c11394adb8030305cc4 not found: ID does not exist"
Oct 30 01:44:39.815949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.815912    4425 scope.go:122] "RemoveContainer" containerID="7f3116ee376b4339eb5f69b8c675a5b042ffac72c0b493a870215694ac171335"
Oct 30 01:44:39.826753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.826738    4425 scope.go:122] "RemoveContainer" containerID="7f3116ee376b4339eb5f69b8c675a5b042ffac72c0b493a870215694ac171335"
Oct 30 01:44:39.826997 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:44:39.826975    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3116ee376b4339eb5f69b8c675a5b042ffac72c0b493a870215694ac171335\": container with ID starting with 7f3116ee376b4339eb5f69b8c675a5b042ffac72c0b493a870215694ac171335 not found: ID does not exist" containerID="7f3116ee376b4339eb5f69b8c675a5b042ffac72c0b493a870215694ac171335"
Oct 30 01:44:39.827067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.827007    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3116ee376b4339eb5f69b8c675a5b042ffac72c0b493a870215694ac171335"} err="failed to get container status \"7f3116ee376b4339eb5f69b8c675a5b042ffac72c0b493a870215694ac171335\": rpc error: code = NotFound desc = could not find container \"7f3116ee376b4339eb5f69b8c675a5b042ffac72c0b493a870215694ac171335\": container with ID starting with 7f3116ee376b4339eb5f69b8c675a5b042ffac72c0b493a870215694ac171335 not found: ID does not exist"
Oct 30 01:44:39.862861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.862827    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"]
Oct 30 01:44:39.867029 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.867010    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b29d4a-6b594c6899-6hm2x"]
Oct 30 01:44:39.895196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.895178    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"]
Oct 30 01:44:39.901501 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:39.901482    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b29d4a-7b8955d6c6-xdfl9"]
Oct 30 01:44:41.424404 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:41.424363    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2157c2a-b5cf-497e-8f76-0d4b290440ec" path="/var/lib/kubelet/pods/a2157c2a-b5cf-497e-8f76-0d4b290440ec/volumes"
Oct 30 01:44:41.424917 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:41.424900    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d47fca09-7546-4877-bd87-3dee036c13c9" path="/var/lib/kubelet/pods/d47fca09-7546-4877-bd87-3dee036c13c9/volumes"
Oct 30 01:44:42.680255 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:42.680220    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"]
Oct 30 01:44:42.689576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:42.689549    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db4dd2-n5jsq"]
Oct 30 01:44:43.427722 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:43.427685    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e82dcb94-05fe-44ee-99b9-3d22a8d803cc" path="/var/lib/kubelet/pods/e82dcb94-05fe-44ee-99b9-3d22a8d803cc/volumes"
Oct 30 01:44:44.297892 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.297852    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"]
Oct 30 01:44:44.298415 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.298114    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf" podUID="a75fad1b-cc67-4381-baa7-3f0aa33c44b7" containerName="maas-api" containerID="cri-o://476622b7325aa88e65d94ae10456930ae4c0d0f9e7eaae3629b7f02bd42bade3" gracePeriod=30
Oct 30 01:44:44.605697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.605665    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:44:44.663466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.663316    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-metrics-certs\") pod \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\" (UID: \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\") "
Oct 30 01:44:44.663466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.663416    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-openshift-service-ca\") pod \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\" (UID: \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\") "
Oct 30 01:44:44.663721 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.663560    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-maas-api-tls\") pod \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\" (UID: \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\") "
Oct 30 01:44:44.663721 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.663644    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-kube-api-access-8mb6z\" (UniqueName: \"kubernetes.io/projected/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-kube-api-access-8mb6z\") pod \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\" (UID: \"a75fad1b-cc67-4381-baa7-3f0aa33c44b7\") "
Oct 30 01:44:44.664974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.664942    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-openshift-service-ca" pod "a75fad1b-cc67-4381-baa7-3f0aa33c44b7" (UID: "a75fad1b-cc67-4381-baa7-3f0aa33c44b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:44:44.671588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.671552    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-maas-api-tls" pod "a75fad1b-cc67-4381-baa7-3f0aa33c44b7" (UID: "a75fad1b-cc67-4381-baa7-3f0aa33c44b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:44:44.684049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.684018    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-metrics-certs" pod "a75fad1b-cc67-4381-baa7-3f0aa33c44b7" (UID: "a75fad1b-cc67-4381-baa7-3f0aa33c44b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:44:44.684265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.684059    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-kube-api-access-8mb6z" pod "a75fad1b-cc67-4381-baa7-3f0aa33c44b7" (UID: "a75fad1b-cc67-4381-baa7-3f0aa33c44b7"). InnerVolumeSpecName "kube-api-access-8mb6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:44:44.765042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.765001    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mb6z\" (UniqueName: \"kubernetes.io/projected/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-kube-api-access-8mb6z\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:44.765042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.765034    4425 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-metrics-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:44.765042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.765043    4425 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-openshift-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:44.765042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.765051    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a75fad1b-cc67-4381-baa7-3f0aa33c44b7-maas-api-tls\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:44.848621 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.848526    4425 generic.go:356] "Generic (PLEG): container finished" podID="a75fad1b-cc67-4381-baa7-3f0aa33c44b7" containerID="476622b7325aa88e65d94ae10456930ae4c0d0f9e7eaae3629b7f02bd42bade3" exitCode=0
Oct 30 01:44:44.848621 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.848604    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"
Oct 30 01:44:44.848861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.848604    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf" event={"ID":"a75fad1b-cc67-4381-baa7-3f0aa33c44b7","Type":"ContainerDied","Data":"476622b7325aa88e65d94ae10456930ae4c0d0f9e7eaae3629b7f02bd42bade3"}
Oct 30 01:44:44.848861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.848752    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf" event={"ID":"a75fad1b-cc67-4381-baa7-3f0aa33c44b7","Type":"ContainerDied","Data":"3525a9a11bcb305846feaabfd41b8b6543ce4630610eb04973fe872dfb909ec7"}
Oct 30 01:44:44.848861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.848781    4425 scope.go:122] "RemoveContainer" containerID="476622b7325aa88e65d94ae10456930ae4c0d0f9e7eaae3629b7f02bd42bade3"
Oct 30 01:44:44.872869 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.872844    4425 scope.go:122] "RemoveContainer" containerID="476622b7325aa88e65d94ae10456930ae4c0d0f9e7eaae3629b7f02bd42bade3"
Oct 30 01:44:44.873374 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:44:44.873340    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476622b7325aa88e65d94ae10456930ae4c0d0f9e7eaae3629b7f02bd42bade3\": container with ID starting with 476622b7325aa88e65d94ae10456930ae4c0d0f9e7eaae3629b7f02bd42bade3 not found: ID does not exist" containerID="476622b7325aa88e65d94ae10456930ae4c0d0f9e7eaae3629b7f02bd42bade3"
Oct 30 01:44:44.873500 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.873383    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476622b7325aa88e65d94ae10456930ae4c0d0f9e7eaae3629b7f02bd42bade3"} err="failed to get container status \"476622b7325aa88e65d94ae10456930ae4c0d0f9e7eaae3629b7f02bd42bade3\": rpc error: code = NotFound desc = could not find container \"476622b7325aa88e65d94ae10456930ae4c0d0f9e7eaae3629b7f02bd42bade3\": container with ID starting with 476622b7325aa88e65d94ae10456930ae4c0d0f9e7eaae3629b7f02bd42bade3 not found: ID does not exist"
Oct 30 01:44:44.905838 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.905808    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"]
Oct 30 01:44:44.910280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:44.910254    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b29d4a-7bff6bccdb-7smhf"]
Oct 30 01:44:45.200021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:45.199922    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"]
Oct 30 01:44:45.206394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:45.206361    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b29d4a-qz5d9"]
Oct 30 01:44:45.428697 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:45.428659    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a75fad1b-cc67-4381-baa7-3f0aa33c44b7" path="/var/lib/kubelet/pods/a75fad1b-cc67-4381-baa7-3f0aa33c44b7/volumes"
Oct 30 01:44:45.429417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:45.429392    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3178cd7-499d-4efa-a7a9-26bc93371fe8" path="/var/lib/kubelet/pods/f3178cd7-499d-4efa-a7a9-26bc93371fe8/volumes"
Oct 30 01:44:51.152287 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.152252    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"]
Oct 30 01:44:51.153005 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.152989    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a2157c2a-b5cf-497e-8f76-0d4b290440ec" containerName="payload-processing"
Oct 30 01:44:51.153059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.153007    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2157c2a-b5cf-497e-8f76-0d4b290440ec" containerName="payload-processing"
Oct 30 01:44:51.153059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.153030    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a75fad1b-cc67-4381-baa7-3f0aa33c44b7" containerName="maas-api"
Oct 30 01:44:51.153059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.153036    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75fad1b-cc67-4381-baa7-3f0aa33c44b7" containerName="maas-api"
Oct 30 01:44:51.153059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.153051    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d47fca09-7546-4877-bd87-3dee036c13c9" containerName="payload-pre-processing"
Oct 30 01:44:51.153059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.153057    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d47fca09-7546-4877-bd87-3dee036c13c9" containerName="payload-pre-processing"
Oct 30 01:44:51.153261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.153071    4425 cpu_manager.go:414] "containerMap: removing container" podUID="e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5" containerName="payload-pre-processing"
Oct 30 01:44:51.153261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.153076    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5" containerName="payload-pre-processing"
Oct 30 01:44:51.153261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.153255    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2157c2a-b5cf-497e-8f76-0d4b290440ec" containerName="payload-processing"
Oct 30 01:44:51.153356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.153268    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75fad1b-cc67-4381-baa7-3f0aa33c44b7" containerName="maas-api"
Oct 30 01:44:51.153356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.153276    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d47fca09-7546-4877-bd87-3dee036c13c9" containerName="payload-pre-processing"
Oct 30 01:44:51.153356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.153282    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3adcbf8-6ae0-48a2-8a4e-d914a4c681b5" containerName="payload-pre-processing"
Oct 30 01:44:51.158433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.158411    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"
Oct 30 01:44:51.177384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.176885    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"]
Oct 30 01:44:51.189486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.189458    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:44:51.189616 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.189582    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:44:51.250381 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.250349    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk72f\" (UniqueName: \"kubernetes.io/projected/41c70768-1d75-4208-a8b4-a745ef199520-kube-api-access-fk72f\") pod \"maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv\" (UID: \"41c70768-1d75-4208-a8b4-a745ef199520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"
Oct 30 01:44:51.250522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.250487    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c70768-1d75-4208-a8b4-a745ef199520-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv\" (UID: \"41c70768-1d75-4208-a8b4-a745ef199520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"
Oct 30 01:44:51.250599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.250573    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c70768-1d75-4208-a8b4-a745ef199520-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv\" (UID: \"41c70768-1d75-4208-a8b4-a745ef199520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"
Oct 30 01:44:51.351733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.351703    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk72f\" (UniqueName: \"kubernetes.io/projected/41c70768-1d75-4208-a8b4-a745ef199520-kube-api-access-fk72f\") pod \"maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv\" (UID: \"41c70768-1d75-4208-a8b4-a745ef199520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"
Oct 30 01:44:51.351880 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.351794    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c70768-1d75-4208-a8b4-a745ef199520-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv\" (UID: \"41c70768-1d75-4208-a8b4-a745ef199520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"
Oct 30 01:44:51.351880 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.351853    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c70768-1d75-4208-a8b4-a745ef199520-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv\" (UID: \"41c70768-1d75-4208-a8b4-a745ef199520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"
Oct 30 01:44:51.352567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.352536    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c70768-1d75-4208-a8b4-a745ef199520-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv\" (UID: \"41c70768-1d75-4208-a8b4-a745ef199520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"
Oct 30 01:44:51.354403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.354379    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c70768-1d75-4208-a8b4-a745ef199520-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv\" (UID: \"41c70768-1d75-4208-a8b4-a745ef199520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"
Oct 30 01:44:51.373458 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.373429    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk72f\" (UniqueName: \"kubernetes.io/projected/41c70768-1d75-4208-a8b4-a745ef199520-kube-api-access-fk72f\") pod \"maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv\" (UID: \"41c70768-1d75-4208-a8b4-a745ef199520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"
Oct 30 01:44:51.474042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.473975    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"
Oct 30 01:44:51.791934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.791911    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"]
Oct 30 01:44:51.795090 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:44:51.795061    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c70768_1d75_4208_a8b4_a745ef199520.slice/crio-c6db521d66c7e620b3b7840be288e3bc0b16ef698563108a1f79cfa3f045ae9e WatchSource:0}: Error finding container c6db521d66c7e620b3b7840be288e3bc0b16ef698563108a1f79cfa3f045ae9e: Status 404 returned error can't find the container with id c6db521d66c7e620b3b7840be288e3bc0b16ef698563108a1f79cfa3f045ae9e
Oct 30 01:44:51.908608 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:51.908575    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv" event={"ID":"41c70768-1d75-4208-a8b4-a745ef199520","Type":"ContainerStarted","Data":"c6db521d66c7e620b3b7840be288e3bc0b16ef698563108a1f79cfa3f045ae9e"}
Oct 30 01:44:52.918511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:52.918435    4425 generic.go:356] "Generic (PLEG): container finished" podID="41c70768-1d75-4208-a8b4-a745ef199520" containerID="8eff35400fe1e71bd83a23308f7a5cf60e0085f028586e97572ed2dce638449f" exitCode=0
Oct 30 01:44:52.918964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:52.918512    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv" event={"ID":"41c70768-1d75-4208-a8b4-a745ef199520","Type":"ContainerDied","Data":"8eff35400fe1e71bd83a23308f7a5cf60e0085f028586e97572ed2dce638449f"}
Oct 30 01:44:53.584288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:53.584253    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"]
Oct 30 01:44:53.588742 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:53.588725    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"
Oct 30 01:44:53.608870 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:53.608837    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"]
Oct 30 01:44:53.684100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:53.684070    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492dec49-fde4-4b48-aa34-afccd3610dda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4\" (UID: \"492dec49-fde4-4b48-aa34-afccd3610dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"
Oct 30 01:44:53.684263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:53.684198    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmrpv\" (UniqueName: \"kubernetes.io/projected/492dec49-fde4-4b48-aa34-afccd3610dda-kube-api-access-pmrpv\") pod \"maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4\" (UID: \"492dec49-fde4-4b48-aa34-afccd3610dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"
Oct 30 01:44:53.684306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:53.684289    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492dec49-fde4-4b48-aa34-afccd3610dda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4\" (UID: \"492dec49-fde4-4b48-aa34-afccd3610dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"
Oct 30 01:44:53.785705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:53.785675    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492dec49-fde4-4b48-aa34-afccd3610dda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4\" (UID: \"492dec49-fde4-4b48-aa34-afccd3610dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"
Oct 30 01:44:53.785887 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:53.785785    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmrpv\" (UniqueName: \"kubernetes.io/projected/492dec49-fde4-4b48-aa34-afccd3610dda-kube-api-access-pmrpv\") pod \"maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4\" (UID: \"492dec49-fde4-4b48-aa34-afccd3610dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"
Oct 30 01:44:53.785961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:53.785916    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492dec49-fde4-4b48-aa34-afccd3610dda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4\" (UID: \"492dec49-fde4-4b48-aa34-afccd3610dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"
Oct 30 01:44:53.786572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:53.786551    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492dec49-fde4-4b48-aa34-afccd3610dda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4\" (UID: \"492dec49-fde4-4b48-aa34-afccd3610dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"
Oct 30 01:44:53.788032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:53.788012    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492dec49-fde4-4b48-aa34-afccd3610dda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4\" (UID: \"492dec49-fde4-4b48-aa34-afccd3610dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"
Oct 30 01:44:53.808088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:53.808055    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmrpv\" (UniqueName: \"kubernetes.io/projected/492dec49-fde4-4b48-aa34-afccd3610dda-kube-api-access-pmrpv\") pod \"maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4\" (UID: \"492dec49-fde4-4b48-aa34-afccd3610dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"
Oct 30 01:44:53.903389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:53.903324    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"
Oct 30 01:44:54.160241 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:54.160170    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"
Oct 30 01:44:54.230201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:54.220791    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"]
Oct 30 01:44:54.230201 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:44:54.227332    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492dec49_fde4_4b48_aa34_afccd3610dda.slice/crio-7d4df1dd4bbc5ea5ac331d242a704968f7b191e3972ffe5a3b39bc15078a9f8d WatchSource:0}: Error finding container 7d4df1dd4bbc5ea5ac331d242a704968f7b191e3972ffe5a3b39bc15078a9f8d: Status 404 returned error can't find the container with id 7d4df1dd4bbc5ea5ac331d242a704968f7b191e3972ffe5a3b39bc15078a9f8d
Oct 30 01:44:54.294417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:54.294385    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41c70768-1d75-4208-a8b4-a745ef199520-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c70768-1d75-4208-a8b4-a745ef199520-maas-api-xks-ca\") pod \"41c70768-1d75-4208-a8b4-a745ef199520\" (UID: \"41c70768-1d75-4208-a8b4-a745ef199520\") "
Oct 30 01:44:54.294544 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:54.294524    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c70768-1d75-4208-a8b4-a745ef199520-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c70768-1d75-4208-a8b4-a745ef199520-maas-api-openshift-ca\") pod \"41c70768-1d75-4208-a8b4-a745ef199520\" (UID: \"41c70768-1d75-4208-a8b4-a745ef199520\") "
Oct 30 01:44:54.294629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:54.294613    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c70768-1d75-4208-a8b4-a745ef199520-kube-api-access-fk72f\" (UniqueName: \"kubernetes.io/projected/41c70768-1d75-4208-a8b4-a745ef199520-kube-api-access-fk72f\") pod \"41c70768-1d75-4208-a8b4-a745ef199520\" (UID: \"41c70768-1d75-4208-a8b4-a745ef199520\") "
Oct 30 01:44:54.295149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:54.295113    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c70768-1d75-4208-a8b4-a745ef199520-maas-api-openshift-ca" pod "41c70768-1d75-4208-a8b4-a745ef199520" (UID: "41c70768-1d75-4208-a8b4-a745ef199520"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:44:54.295392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:54.295371    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c70768-1d75-4208-a8b4-a745ef199520-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:54.296795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:54.296771    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c70768-1d75-4208-a8b4-a745ef199520-kube-api-access-fk72f" pod "41c70768-1d75-4208-a8b4-a745ef199520" (UID: "41c70768-1d75-4208-a8b4-a745ef199520"). InnerVolumeSpecName "kube-api-access-fk72f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:44:54.296888 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:54.296838    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c70768-1d75-4208-a8b4-a745ef199520-maas-api-xks-ca" pod "41c70768-1d75-4208-a8b4-a745ef199520" (UID: "41c70768-1d75-4208-a8b4-a745ef199520"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:44:54.396621 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:54.396587    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk72f\" (UniqueName: \"kubernetes.io/projected/41c70768-1d75-4208-a8b4-a745ef199520-kube-api-access-fk72f\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:54.396621 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:54.396619    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c70768-1d75-4208-a8b4-a745ef199520-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:54.936979 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:54.936943    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"
Oct 30 01:44:54.937135 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:54.936965    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv" event={"ID":"41c70768-1d75-4208-a8b4-a745ef199520","Type":"ContainerDied","Data":"c6db521d66c7e620b3b7840be288e3bc0b16ef698563108a1f79cfa3f045ae9e"}
Oct 30 01:44:54.937135 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:54.937025    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6db521d66c7e620b3b7840be288e3bc0b16ef698563108a1f79cfa3f045ae9e"
Oct 30 01:44:54.939072 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:54.939042    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4" event={"ID":"492dec49-fde4-4b48-aa34-afccd3610dda","Type":"ContainerStarted","Data":"ef96c8976cf2876ebd30e6be33071db26a63691403b4e0cd0c80c9321d5129e7"}
Oct 30 01:44:54.939211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:54.939086    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4" event={"ID":"492dec49-fde4-4b48-aa34-afccd3610dda","Type":"ContainerStarted","Data":"7d4df1dd4bbc5ea5ac331d242a704968f7b191e3972ffe5a3b39bc15078a9f8d"}
Oct 30 01:44:54.989701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:54.989653    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4" podStartSLOduration=1.989639162 podStartE2EDuration="1.989639162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:54.984011686 +0000 UTC m=+3803.733666995" watchObservedRunningTime="2026-10-30 01:44:54.989639162 +0000 UTC m=+3803.739294470"
Oct 30 01:44:55.948945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:55.948903    4425 generic.go:356] "Generic (PLEG): container finished" podID="492dec49-fde4-4b48-aa34-afccd3610dda" containerID="ef96c8976cf2876ebd30e6be33071db26a63691403b4e0cd0c80c9321d5129e7" exitCode=0
Oct 30 01:44:55.949424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:55.948979    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4" event={"ID":"492dec49-fde4-4b48-aa34-afccd3610dda","Type":"ContainerDied","Data":"ef96c8976cf2876ebd30e6be33071db26a63691403b4e0cd0c80c9321d5129e7"}
Oct 30 01:44:56.276445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.276410    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"]
Oct 30 01:44:56.277200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.277179    4425 cpu_manager.go:414] "containerMap: removing container" podUID="41c70768-1d75-4208-a8b4-a745ef199520" containerName="revoke-keys"
Oct 30 01:44:56.277247 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.277201    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c70768-1d75-4208-a8b4-a745ef199520" containerName="revoke-keys"
Oct 30 01:44:56.277351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.277342    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c70768-1d75-4208-a8b4-a745ef199520" containerName="revoke-keys"
Oct 30 01:44:56.281350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.281332    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"
Oct 30 01:44:56.321600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.321577    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"]
Oct 30 01:44:56.427795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.427764    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997e8b87-01db-47e2-b9ef-bf86441a027d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e9f9fb5-6k5t5\" (UID: \"997e8b87-01db-47e2-b9ef-bf86441a027d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"
Oct 30 01:44:56.427958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.427840    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997e8b87-01db-47e2-b9ef-bf86441a027d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e9f9fb5-6k5t5\" (UID: \"997e8b87-01db-47e2-b9ef-bf86441a027d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"
Oct 30 01:44:56.427958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.427895    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrjn8\" (UniqueName: \"kubernetes.io/projected/997e8b87-01db-47e2-b9ef-bf86441a027d-kube-api-access-nrjn8\") pod \"maas-api-revoke-sub-7e9f9fb5-6k5t5\" (UID: \"997e8b87-01db-47e2-b9ef-bf86441a027d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"
Oct 30 01:44:56.529028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.528950    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997e8b87-01db-47e2-b9ef-bf86441a027d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e9f9fb5-6k5t5\" (UID: \"997e8b87-01db-47e2-b9ef-bf86441a027d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"
Oct 30 01:44:56.529028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.529010    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997e8b87-01db-47e2-b9ef-bf86441a027d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e9f9fb5-6k5t5\" (UID: \"997e8b87-01db-47e2-b9ef-bf86441a027d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"
Oct 30 01:44:56.529239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.529032    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjn8\" (UniqueName: \"kubernetes.io/projected/997e8b87-01db-47e2-b9ef-bf86441a027d-kube-api-access-nrjn8\") pod \"maas-api-revoke-sub-7e9f9fb5-6k5t5\" (UID: \"997e8b87-01db-47e2-b9ef-bf86441a027d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"
Oct 30 01:44:56.529737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.529716    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997e8b87-01db-47e2-b9ef-bf86441a027d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e9f9fb5-6k5t5\" (UID: \"997e8b87-01db-47e2-b9ef-bf86441a027d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"
Oct 30 01:44:56.531369 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.531347    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997e8b87-01db-47e2-b9ef-bf86441a027d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e9f9fb5-6k5t5\" (UID: \"997e8b87-01db-47e2-b9ef-bf86441a027d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"
Oct 30 01:44:56.558112 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.558088    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrjn8\" (UniqueName: \"kubernetes.io/projected/997e8b87-01db-47e2-b9ef-bf86441a027d-kube-api-access-nrjn8\") pod \"maas-api-revoke-sub-7e9f9fb5-6k5t5\" (UID: \"997e8b87-01db-47e2-b9ef-bf86441a027d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"
Oct 30 01:44:56.595075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.595050    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"
Oct 30 01:44:56.897984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.897950    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"]
Oct 30 01:44:56.960443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:56.960413    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5" event={"ID":"997e8b87-01db-47e2-b9ef-bf86441a027d","Type":"ContainerStarted","Data":"64fc55ddfc4b609af33152bb3ed3c3da001f8abeda323a0bbebcaa66162fda31"}
Oct 30 01:44:57.107213 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:57.107141    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"
Oct 30 01:44:57.136651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:57.136630    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492dec49-fde4-4b48-aa34-afccd3610dda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492dec49-fde4-4b48-aa34-afccd3610dda-maas-api-openshift-ca\") pod \"492dec49-fde4-4b48-aa34-afccd3610dda\" (UID: \"492dec49-fde4-4b48-aa34-afccd3610dda\") "
Oct 30 01:44:57.136765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:57.136727    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492dec49-fde4-4b48-aa34-afccd3610dda-kube-api-access-pmrpv\" (UniqueName: \"kubernetes.io/projected/492dec49-fde4-4b48-aa34-afccd3610dda-kube-api-access-pmrpv\") pod \"492dec49-fde4-4b48-aa34-afccd3610dda\" (UID: \"492dec49-fde4-4b48-aa34-afccd3610dda\") "
Oct 30 01:44:57.136904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:57.136886    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/492dec49-fde4-4b48-aa34-afccd3610dda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492dec49-fde4-4b48-aa34-afccd3610dda-maas-api-xks-ca\") pod \"492dec49-fde4-4b48-aa34-afccd3610dda\" (UID: \"492dec49-fde4-4b48-aa34-afccd3610dda\") "
Oct 30 01:44:57.136960 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:57.136941    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492dec49-fde4-4b48-aa34-afccd3610dda-maas-api-openshift-ca" pod "492dec49-fde4-4b48-aa34-afccd3610dda" (UID: "492dec49-fde4-4b48-aa34-afccd3610dda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:44:57.137438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:57.137415    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492dec49-fde4-4b48-aa34-afccd3610dda-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:57.138750 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:57.138731    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492dec49-fde4-4b48-aa34-afccd3610dda-kube-api-access-pmrpv" pod "492dec49-fde4-4b48-aa34-afccd3610dda" (UID: "492dec49-fde4-4b48-aa34-afccd3610dda"). InnerVolumeSpecName "kube-api-access-pmrpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:44:57.138828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:57.138784    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492dec49-fde4-4b48-aa34-afccd3610dda-maas-api-xks-ca" pod "492dec49-fde4-4b48-aa34-afccd3610dda" (UID: "492dec49-fde4-4b48-aa34-afccd3610dda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:44:57.239301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:57.239219    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmrpv\" (UniqueName: \"kubernetes.io/projected/492dec49-fde4-4b48-aa34-afccd3610dda-kube-api-access-pmrpv\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:57.239301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:57.239246    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492dec49-fde4-4b48-aa34-afccd3610dda-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:57.971876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:57.971738    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"
Oct 30 01:44:57.971876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:57.971769    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4" event={"ID":"492dec49-fde4-4b48-aa34-afccd3610dda","Type":"ContainerDied","Data":"7d4df1dd4bbc5ea5ac331d242a704968f7b191e3972ffe5a3b39bc15078a9f8d"}
Oct 30 01:44:57.971876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:57.971828    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d4df1dd4bbc5ea5ac331d242a704968f7b191e3972ffe5a3b39bc15078a9f8d"
Oct 30 01:44:57.973934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:57.973907    4425 generic.go:356] "Generic (PLEG): container finished" podID="997e8b87-01db-47e2-b9ef-bf86441a027d" containerID="df92f7278efdd867dbcbb341e6ff452c792f9784a505be7288a6a9b1d1600d14" exitCode=0
Oct 30 01:44:57.974082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:57.973995    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5" event={"ID":"997e8b87-01db-47e2-b9ef-bf86441a027d","Type":"ContainerDied","Data":"df92f7278efdd867dbcbb341e6ff452c792f9784a505be7288a6a9b1d1600d14"}
Oct 30 01:44:59.156311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:59.156286    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"
Oct 30 01:44:59.265238 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:59.265211    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/997e8b87-01db-47e2-b9ef-bf86441a027d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997e8b87-01db-47e2-b9ef-bf86441a027d-maas-api-xks-ca\") pod \"997e8b87-01db-47e2-b9ef-bf86441a027d\" (UID: \"997e8b87-01db-47e2-b9ef-bf86441a027d\") "
Oct 30 01:44:59.265403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:59.265325    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997e8b87-01db-47e2-b9ef-bf86441a027d-kube-api-access-nrjn8\" (UniqueName: \"kubernetes.io/projected/997e8b87-01db-47e2-b9ef-bf86441a027d-kube-api-access-nrjn8\") pod \"997e8b87-01db-47e2-b9ef-bf86441a027d\" (UID: \"997e8b87-01db-47e2-b9ef-bf86441a027d\") "
Oct 30 01:44:59.265452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:59.265423    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/997e8b87-01db-47e2-b9ef-bf86441a027d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997e8b87-01db-47e2-b9ef-bf86441a027d-maas-api-openshift-ca\") pod \"997e8b87-01db-47e2-b9ef-bf86441a027d\" (UID: \"997e8b87-01db-47e2-b9ef-bf86441a027d\") "
Oct 30 01:44:59.265757 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:59.265725    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997e8b87-01db-47e2-b9ef-bf86441a027d-maas-api-openshift-ca" pod "997e8b87-01db-47e2-b9ef-bf86441a027d" (UID: "997e8b87-01db-47e2-b9ef-bf86441a027d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:44:59.265906 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:59.265884    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997e8b87-01db-47e2-b9ef-bf86441a027d-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:59.267258 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:59.267230    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/997e8b87-01db-47e2-b9ef-bf86441a027d-maas-api-xks-ca" pod "997e8b87-01db-47e2-b9ef-bf86441a027d" (UID: "997e8b87-01db-47e2-b9ef-bf86441a027d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:44:59.267442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:59.267420    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997e8b87-01db-47e2-b9ef-bf86441a027d-kube-api-access-nrjn8" pod "997e8b87-01db-47e2-b9ef-bf86441a027d" (UID: "997e8b87-01db-47e2-b9ef-bf86441a027d"). InnerVolumeSpecName "kube-api-access-nrjn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:44:59.367673 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:59.367646    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997e8b87-01db-47e2-b9ef-bf86441a027d-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:44:59.367673 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:44:59.367674    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrjn8\" (UniqueName: \"kubernetes.io/projected/997e8b87-01db-47e2-b9ef-bf86441a027d-kube-api-access-nrjn8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:00.001745 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.001705    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"
Oct 30 01:45:00.001924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.001786    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5" event={"ID":"997e8b87-01db-47e2-b9ef-bf86441a027d","Type":"ContainerDied","Data":"64fc55ddfc4b609af33152bb3ed3c3da001f8abeda323a0bbebcaa66162fda31"}
Oct 30 01:45:00.001924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.001865    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64fc55ddfc4b609af33152bb3ed3c3da001f8abeda323a0bbebcaa66162fda31"
Oct 30 01:45:00.154345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.154319    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs"]
Oct 30 01:45:00.155293 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.155268    4425 cpu_manager.go:414] "containerMap: removing container" podUID="492dec49-fde4-4b48-aa34-afccd3610dda" containerName="revoke-keys"
Oct 30 01:45:00.155293 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.155290    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492dec49-fde4-4b48-aa34-afccd3610dda" containerName="revoke-keys"
Oct 30 01:45:00.155482 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.155346    4425 cpu_manager.go:414] "containerMap: removing container" podUID="997e8b87-01db-47e2-b9ef-bf86441a027d" containerName="revoke-keys"
Oct 30 01:45:00.155482 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.155352    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997e8b87-01db-47e2-b9ef-bf86441a027d" containerName="revoke-keys"
Oct 30 01:45:00.155482 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.155475    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="997e8b87-01db-47e2-b9ef-bf86441a027d" containerName="revoke-keys"
Oct 30 01:45:00.155601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.155489    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="492dec49-fde4-4b48-aa34-afccd3610dda" containerName="revoke-keys"
Oct 30 01:45:00.159011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.158993    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj"]
Oct 30 01:45:00.159384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.159099    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs"
Oct 30 01:45:00.163242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.163226    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj"
Oct 30 01:45:00.177884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.177860    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:45:00.178249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.178220    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g954k\" (UniqueName: \"kubernetes.io/projected/336e3514-5287-4632-84f7-2673bc71a606-kube-api-access-g954k\") pod \"maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs\" (UID: \"336e3514-5287-4632-84f7-2673bc71a606\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs"
Oct 30 01:45:00.178387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.178273    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/336e3514-5287-4632-84f7-2673bc71a606-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs\" (UID: \"336e3514-5287-4632-84f7-2673bc71a606\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs"
Oct 30 01:45:00.179289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.179269    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv"]
Oct 30 01:45:00.184312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.184293    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p"]
Oct 30 01:45:00.184423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.184385    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv"
Oct 30 01:45:00.189517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.189497    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf"]
Oct 30 01:45:00.189627 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.189610    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p"
Oct 30 01:45:00.200023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.199999    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj"]
Oct 30 01:45:00.200182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.200115    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf"
Oct 30 01:45:00.200288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.200270    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs"]
Oct 30 01:45:00.210197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.209135    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd"]
Oct 30 01:45:00.237306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.235371    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf"]
Oct 30 01:45:00.237306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.235416    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p"]
Oct 30 01:45:00.237306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.235445    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2"]
Oct 30 01:45:00.238287 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.238264    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd"
Oct 30 01:45:00.243261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.243234    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2"]
Oct 30 01:45:00.243452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.243434    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2"
Oct 30 01:45:00.257453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.255326    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv"]
Oct 30 01:45:00.257453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.255370    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd"]
Oct 30 01:45:00.257453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.255454    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2"
Oct 30 01:45:00.260217 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.260194    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2"]
Oct 30 01:45:00.272728 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.272688    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9"]
Oct 30 01:45:00.278447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.278420    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9"
Oct 30 01:45:00.279209 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.279180    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44cf871-7767-4ad3-a9b4-e70a58a52ec9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2\" (UID: \"d44cf871-7767-4ad3-a9b4-e70a58a52ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2"
Oct 30 01:45:00.279342 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.279226    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e253e33e-8ac1-4658-aaf1-485bb678f1b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p\" (UID: \"e253e33e-8ac1-4658-aaf1-485bb678f1b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p"
Oct 30 01:45:00.279342 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.279245    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77mgx\" (UniqueName: \"kubernetes.io/projected/d44cf871-7767-4ad3-a9b4-e70a58a52ec9-kube-api-access-77mgx\") pod \"maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2\" (UID: \"d44cf871-7767-4ad3-a9b4-e70a58a52ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2"
Oct 30 01:45:00.279342 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.279265    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdvpv\" (UniqueName: \"kubernetes.io/projected/ee57e527-71ff-4a09-8db0-9840a6467fce-kube-api-access-kdvpv\") pod \"maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv\" (UID: \"ee57e527-71ff-4a09-8db0-9840a6467fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv"
Oct 30 01:45:00.279536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.279333    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g954k\" (UniqueName: \"kubernetes.io/projected/336e3514-5287-4632-84f7-2673bc71a606-kube-api-access-g954k\") pod \"maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs\" (UID: \"336e3514-5287-4632-84f7-2673bc71a606\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs"
Oct 30 01:45:00.279536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.279498    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/336e3514-5287-4632-84f7-2673bc71a606-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs\" (UID: \"336e3514-5287-4632-84f7-2673bc71a606\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs"
Oct 30 01:45:00.279665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.279550    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee57e527-71ff-4a09-8db0-9840a6467fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv\" (UID: \"ee57e527-71ff-4a09-8db0-9840a6467fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv"
Oct 30 01:45:00.279665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.279578    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql2cz\" (UniqueName: \"kubernetes.io/projected/ebecef45-aca8-486c-af0c-33bc1380da5d-kube-api-access-ql2cz\") pod \"maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj\" (UID: \"ebecef45-aca8-486c-af0c-33bc1380da5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj"
Oct 30 01:45:00.279665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.279609    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zls8w\" (UniqueName: \"kubernetes.io/projected/357853d9-748f-44af-baa0-523e64187284-kube-api-access-zls8w\") pod \"maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd\" (UID: \"357853d9-748f-44af-baa0-523e64187284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd"
Oct 30 01:45:00.280405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.280377    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/336e3514-5287-4632-84f7-2673bc71a606-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs\" (UID: \"336e3514-5287-4632-84f7-2673bc71a606\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs"
Oct 30 01:45:00.280523 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.280501    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbhzd\" (UniqueName: \"kubernetes.io/projected/af779dae-521f-4f8a-9ad2-e7fe3948dcbc-kube-api-access-kbhzd\") pod \"maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2\" (UID: \"af779dae-521f-4f8a-9ad2-e7fe3948dcbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2"
Oct 30 01:45:00.280665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.280648    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bfmc\" (UniqueName: \"kubernetes.io/projected/caccf6b1-b7c1-437e-a634-84d62cf8d3b4-kube-api-access-6bfmc\") pod \"maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf\" (UID: \"caccf6b1-b7c1-437e-a634-84d62cf8d3b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf"
Oct 30 01:45:00.280744 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.280684    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357853d9-748f-44af-baa0-523e64187284-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd\" (UID: \"357853d9-748f-44af-baa0-523e64187284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd"
Oct 30 01:45:00.280744 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.280737    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebecef45-aca8-486c-af0c-33bc1380da5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj\" (UID: \"ebecef45-aca8-486c-af0c-33bc1380da5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj"
Oct 30 01:45:00.280861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.280790    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d7ps\" (UniqueName: \"kubernetes.io/projected/e253e33e-8ac1-4658-aaf1-485bb678f1b5-kube-api-access-9d7ps\") pod \"maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p\" (UID: \"e253e33e-8ac1-4658-aaf1-485bb678f1b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p"
Oct 30 01:45:00.280861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.280818    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caccf6b1-b7c1-437e-a634-84d62cf8d3b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf\" (UID: \"caccf6b1-b7c1-437e-a634-84d62cf8d3b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf"
Oct 30 01:45:00.280969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.280857    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af779dae-521f-4f8a-9ad2-e7fe3948dcbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2\" (UID: \"af779dae-521f-4f8a-9ad2-e7fe3948dcbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2"
Oct 30 01:45:00.294886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.294376    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2"]
Oct 30 01:45:00.301768 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.301732    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9"]
Oct 30 01:45:00.316826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.316800    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g954k\" (UniqueName: \"kubernetes.io/projected/336e3514-5287-4632-84f7-2673bc71a606-kube-api-access-g954k\") pod \"maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs\" (UID: \"336e3514-5287-4632-84f7-2673bc71a606\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs"
Oct 30 01:45:00.322424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.322391    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm"]
Oct 30 01:45:00.328012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.327987    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d"]
Oct 30 01:45:00.328998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.328794    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm"
Oct 30 01:45:00.333978 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.333955    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d"
Oct 30 01:45:00.340819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.340065    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm"]
Oct 30 01:45:00.354941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.354914    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d"]
Oct 30 01:45:00.369822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.369790    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k"]
Oct 30 01:45:00.377719 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.377692    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k"
Oct 30 01:45:00.383209 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.383175    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af779dae-521f-4f8a-9ad2-e7fe3948dcbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2\" (UID: \"af779dae-521f-4f8a-9ad2-e7fe3948dcbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2"
Oct 30 01:45:00.383360 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.383231    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lskpp\" (UniqueName: \"kubernetes.io/projected/6c1bab87-9a09-48f6-9302-43b0254810e5-kube-api-access-lskpp\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d\" (UID: \"6c1bab87-9a09-48f6-9302-43b0254810e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d"
Oct 30 01:45:00.383360 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.383269    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44cf871-7767-4ad3-a9b4-e70a58a52ec9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2\" (UID: \"d44cf871-7767-4ad3-a9b4-e70a58a52ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2"
Oct 30 01:45:00.383360 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.383302    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1bab87-9a09-48f6-9302-43b0254810e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d\" (UID: \"6c1bab87-9a09-48f6-9302-43b0254810e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d"
Oct 30 01:45:00.383360 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.383341    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6hf7\" (UniqueName: \"kubernetes.io/projected/9a82dd87-6039-44da-b690-297f5449c337-kube-api-access-t6hf7\") pod \"maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9\" (UID: \"9a82dd87-6039-44da-b690-297f5449c337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9"
Oct 30 01:45:00.383612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.383378    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e253e33e-8ac1-4658-aaf1-485bb678f1b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p\" (UID: \"e253e33e-8ac1-4658-aaf1-485bb678f1b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p"
Oct 30 01:45:00.383612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.383405    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77mgx\" (UniqueName: \"kubernetes.io/projected/d44cf871-7767-4ad3-a9b4-e70a58a52ec9-kube-api-access-77mgx\") pod \"maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2\" (UID: \"d44cf871-7767-4ad3-a9b4-e70a58a52ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2"
Oct 30 01:45:00.383612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.383431    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a87024-9da9-4842-afb2-063b447bd15a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm\" (UID: \"a2a87024-9da9-4842-afb2-063b447bd15a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm"
Oct 30 01:45:00.383612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.383462    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdvpv\" (UniqueName: \"kubernetes.io/projected/ee57e527-71ff-4a09-8db0-9840a6467fce-kube-api-access-kdvpv\") pod \"maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv\" (UID: \"ee57e527-71ff-4a09-8db0-9840a6467fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv"
Oct 30 01:45:00.383612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.383559    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee57e527-71ff-4a09-8db0-9840a6467fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv\" (UID: \"ee57e527-71ff-4a09-8db0-9840a6467fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv"
Oct 30 01:45:00.383612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.383601    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql2cz\" (UniqueName: \"kubernetes.io/projected/ebecef45-aca8-486c-af0c-33bc1380da5d-kube-api-access-ql2cz\") pod \"maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj\" (UID: \"ebecef45-aca8-486c-af0c-33bc1380da5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj"
Oct 30 01:45:00.383952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.383649    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zls8w\" (UniqueName: \"kubernetes.io/projected/357853d9-748f-44af-baa0-523e64187284-kube-api-access-zls8w\") pod \"maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd\" (UID: \"357853d9-748f-44af-baa0-523e64187284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd"
Oct 30 01:45:00.383952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.383684    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7prb5\" (UniqueName: \"kubernetes.io/projected/a2a87024-9da9-4842-afb2-063b447bd15a-kube-api-access-7prb5\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm\" (UID: \"a2a87024-9da9-4842-afb2-063b447bd15a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm"
Oct 30 01:45:00.383952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.383724    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbhzd\" (UniqueName: \"kubernetes.io/projected/af779dae-521f-4f8a-9ad2-e7fe3948dcbc-kube-api-access-kbhzd\") pod \"maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2\" (UID: \"af779dae-521f-4f8a-9ad2-e7fe3948dcbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2"
Oct 30 01:45:00.383952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.383819    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a82dd87-6039-44da-b690-297f5449c337-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9\" (UID: \"9a82dd87-6039-44da-b690-297f5449c337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9"
Oct 30 01:45:00.383952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.383910    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bfmc\" (UniqueName: \"kubernetes.io/projected/caccf6b1-b7c1-437e-a634-84d62cf8d3b4-kube-api-access-6bfmc\") pod \"maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf\" (UID: \"caccf6b1-b7c1-437e-a634-84d62cf8d3b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf"
Oct 30 01:45:00.383952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.383942    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357853d9-748f-44af-baa0-523e64187284-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd\" (UID: \"357853d9-748f-44af-baa0-523e64187284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd"
Oct 30 01:45:00.384309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.384001    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebecef45-aca8-486c-af0c-33bc1380da5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj\" (UID: \"ebecef45-aca8-486c-af0c-33bc1380da5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj"
Oct 30 01:45:00.384309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.384062    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d7ps\" (UniqueName: \"kubernetes.io/projected/e253e33e-8ac1-4658-aaf1-485bb678f1b5-kube-api-access-9d7ps\") pod \"maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p\" (UID: \"e253e33e-8ac1-4658-aaf1-485bb678f1b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p"
Oct 30 01:45:00.384309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.384089    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caccf6b1-b7c1-437e-a634-84d62cf8d3b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf\" (UID: \"caccf6b1-b7c1-437e-a634-84d62cf8d3b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf"
Oct 30 01:45:00.385467 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.384968    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caccf6b1-b7c1-437e-a634-84d62cf8d3b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf\" (UID: \"caccf6b1-b7c1-437e-a634-84d62cf8d3b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf"
Oct 30 01:45:00.385467 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.385351    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k"]
Oct 30 01:45:00.387348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.387325    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357853d9-748f-44af-baa0-523e64187284-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd\" (UID: \"357853d9-748f-44af-baa0-523e64187284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd"
Oct 30 01:45:00.388186 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.388149    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebecef45-aca8-486c-af0c-33bc1380da5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj\" (UID: \"ebecef45-aca8-486c-af0c-33bc1380da5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj"
Oct 30 01:45:00.394477 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.390849    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee57e527-71ff-4a09-8db0-9840a6467fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv\" (UID: \"ee57e527-71ff-4a09-8db0-9840a6467fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv"
Oct 30 01:45:00.394477 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.391481    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e253e33e-8ac1-4658-aaf1-485bb678f1b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p\" (UID: \"e253e33e-8ac1-4658-aaf1-485bb678f1b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p"
Oct 30 01:45:00.394626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.394369    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44cf871-7767-4ad3-a9b4-e70a58a52ec9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2\" (UID: \"d44cf871-7767-4ad3-a9b4-e70a58a52ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2"
Oct 30 01:45:00.395020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.394993    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af779dae-521f-4f8a-9ad2-e7fe3948dcbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2\" (UID: \"af779dae-521f-4f8a-9ad2-e7fe3948dcbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2"
Oct 30 01:45:00.403015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.402831    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7"]
Oct 30 01:45:00.413204 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.412783    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7"]
Oct 30 01:45:00.418927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.417324    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7"
Oct 30 01:45:00.419649 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.419626    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zls8w\" (UniqueName: \"kubernetes.io/projected/357853d9-748f-44af-baa0-523e64187284-kube-api-access-zls8w\") pod \"maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd\" (UID: \"357853d9-748f-44af-baa0-523e64187284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd"
Oct 30 01:45:00.421352 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.421329    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bfmc\" (UniqueName: \"kubernetes.io/projected/caccf6b1-b7c1-437e-a634-84d62cf8d3b4-kube-api-access-6bfmc\") pod \"maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf\" (UID: \"caccf6b1-b7c1-437e-a634-84d62cf8d3b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf"
Oct 30 01:45:00.422538 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.422517    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7"]
Oct 30 01:45:00.422698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.422683    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7"
Oct 30 01:45:00.424843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.424813    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d7ps\" (UniqueName: \"kubernetes.io/projected/e253e33e-8ac1-4658-aaf1-485bb678f1b5-kube-api-access-9d7ps\") pod \"maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p\" (UID: \"e253e33e-8ac1-4658-aaf1-485bb678f1b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p"
Oct 30 01:45:00.431351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.431299    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql2cz\" (UniqueName: \"kubernetes.io/projected/ebecef45-aca8-486c-af0c-33bc1380da5d-kube-api-access-ql2cz\") pod \"maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj\" (UID: \"ebecef45-aca8-486c-af0c-33bc1380da5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj"
Oct 30 01:45:00.431447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.431365    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdvpv\" (UniqueName: \"kubernetes.io/projected/ee57e527-71ff-4a09-8db0-9840a6467fce-kube-api-access-kdvpv\") pod \"maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv\" (UID: \"ee57e527-71ff-4a09-8db0-9840a6467fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv"
Oct 30 01:45:00.431826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.431801    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77mgx\" (UniqueName: \"kubernetes.io/projected/d44cf871-7767-4ad3-a9b4-e70a58a52ec9-kube-api-access-77mgx\") pod \"maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2\" (UID: \"d44cf871-7767-4ad3-a9b4-e70a58a52ec9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2"
Oct 30 01:45:00.433059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.433021    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbhzd\" (UniqueName: \"kubernetes.io/projected/af779dae-521f-4f8a-9ad2-e7fe3948dcbc-kube-api-access-kbhzd\") pod \"maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2\" (UID: \"af779dae-521f-4f8a-9ad2-e7fe3948dcbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2"
Oct 30 01:45:00.462035 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.460384    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7"]
Oct 30 01:45:00.475758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.475729    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs"
Oct 30 01:45:00.484912 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.484885    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf3962c-f594-478e-89dc-a0ab9671bb9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7\" (UID: \"3cf3962c-f594-478e-89dc-a0ab9671bb9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7"
Oct 30 01:45:00.485195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.485146    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7prb5\" (UniqueName: \"kubernetes.io/projected/a2a87024-9da9-4842-afb2-063b447bd15a-kube-api-access-7prb5\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm\" (UID: \"a2a87024-9da9-4842-afb2-063b447bd15a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm"
Oct 30 01:45:00.485330 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.485310    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a82dd87-6039-44da-b690-297f5449c337-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9\" (UID: \"9a82dd87-6039-44da-b690-297f5449c337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9"
Oct 30 01:45:00.485412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.485348    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtkdk\" (UniqueName: \"kubernetes.io/projected/3cf3962c-f594-478e-89dc-a0ab9671bb9b-kube-api-access-wtkdk\") pod \"maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7\" (UID: \"3cf3962c-f594-478e-89dc-a0ab9671bb9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7"
Oct 30 01:45:00.485412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.485395    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd1b902-d5db-4613-90ea-30a92c1bf2b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k\" (UID: \"8cd1b902-d5db-4613-90ea-30a92c1bf2b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k"
Oct 30 01:45:00.486301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.486093    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khdh7\" (UniqueName: \"kubernetes.io/projected/8cd1b902-d5db-4613-90ea-30a92c1bf2b0-kube-api-access-khdh7\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k\" (UID: \"8cd1b902-d5db-4613-90ea-30a92c1bf2b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k"
Oct 30 01:45:00.486301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.486252    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2fk5\" (UniqueName: \"kubernetes.io/projected/e77b1d4c-dc08-416b-ad79-6e777474940c-kube-api-access-n2fk5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7\" (UID: \"e77b1d4c-dc08-416b-ad79-6e777474940c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7"
Oct 30 01:45:00.486301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.486292    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lskpp\" (UniqueName: \"kubernetes.io/projected/6c1bab87-9a09-48f6-9302-43b0254810e5-kube-api-access-lskpp\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d\" (UID: \"6c1bab87-9a09-48f6-9302-43b0254810e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d"
Oct 30 01:45:00.486505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.486344    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1bab87-9a09-48f6-9302-43b0254810e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d\" (UID: \"6c1bab87-9a09-48f6-9302-43b0254810e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d"
Oct 30 01:45:00.486505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.486381    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6hf7\" (UniqueName: \"kubernetes.io/projected/9a82dd87-6039-44da-b690-297f5449c337-kube-api-access-t6hf7\") pod \"maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9\" (UID: \"9a82dd87-6039-44da-b690-297f5449c337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9"
Oct 30 01:45:00.486505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.486434    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a87024-9da9-4842-afb2-063b447bd15a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm\" (UID: \"a2a87024-9da9-4842-afb2-063b447bd15a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm"
Oct 30 01:45:00.486505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.486495    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b1d4c-dc08-416b-ad79-6e777474940c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7\" (UID: \"e77b1d4c-dc08-416b-ad79-6e777474940c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7"
Oct 30 01:45:00.488416 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.487554    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1bab87-9a09-48f6-9302-43b0254810e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d\" (UID: \"6c1bab87-9a09-48f6-9302-43b0254810e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d"
Oct 30 01:45:00.488416 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.487954    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a87024-9da9-4842-afb2-063b447bd15a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm\" (UID: \"a2a87024-9da9-4842-afb2-063b447bd15a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm"
Oct 30 01:45:00.488416 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.487997    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj"
Oct 30 01:45:00.488416 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.488250    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a82dd87-6039-44da-b690-297f5449c337-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9\" (UID: \"9a82dd87-6039-44da-b690-297f5449c337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9"
Oct 30 01:45:00.506590 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.506567    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv"
Oct 30 01:45:00.517657 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.517628    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7prb5\" (UniqueName: \"kubernetes.io/projected/a2a87024-9da9-4842-afb2-063b447bd15a-kube-api-access-7prb5\") pod \"maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm\" (UID: \"a2a87024-9da9-4842-afb2-063b447bd15a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm"
Oct 30 01:45:00.519871 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.519843    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lskpp\" (UniqueName: \"kubernetes.io/projected/6c1bab87-9a09-48f6-9302-43b0254810e5-kube-api-access-lskpp\") pod \"maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d\" (UID: \"6c1bab87-9a09-48f6-9302-43b0254810e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d"
Oct 30 01:45:00.522208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.522140    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6hf7\" (UniqueName: \"kubernetes.io/projected/9a82dd87-6039-44da-b690-297f5449c337-kube-api-access-t6hf7\") pod \"maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9\" (UID: \"9a82dd87-6039-44da-b690-297f5449c337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9"
Oct 30 01:45:00.534593 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.534386    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p"
Oct 30 01:45:00.542456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.542296    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf"
Oct 30 01:45:00.568687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.568439    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd"
Oct 30 01:45:00.584659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.584463    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2"
Oct 30 01:45:00.593532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.591175    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtkdk\" (UniqueName: \"kubernetes.io/projected/3cf3962c-f594-478e-89dc-a0ab9671bb9b-kube-api-access-wtkdk\") pod \"maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7\" (UID: \"3cf3962c-f594-478e-89dc-a0ab9671bb9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7"
Oct 30 01:45:00.593532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.591470    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd1b902-d5db-4613-90ea-30a92c1bf2b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k\" (UID: \"8cd1b902-d5db-4613-90ea-30a92c1bf2b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k"
Oct 30 01:45:00.593532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.591595    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khdh7\" (UniqueName: \"kubernetes.io/projected/8cd1b902-d5db-4613-90ea-30a92c1bf2b0-kube-api-access-khdh7\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k\" (UID: \"8cd1b902-d5db-4613-90ea-30a92c1bf2b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k"
Oct 30 01:45:00.593532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.591639    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2fk5\" (UniqueName: \"kubernetes.io/projected/e77b1d4c-dc08-416b-ad79-6e777474940c-kube-api-access-n2fk5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7\" (UID: \"e77b1d4c-dc08-416b-ad79-6e777474940c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7"
Oct 30 01:45:00.593532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.591718    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b1d4c-dc08-416b-ad79-6e777474940c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7\" (UID: \"e77b1d4c-dc08-416b-ad79-6e777474940c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7"
Oct 30 01:45:00.593532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.591805    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf3962c-f594-478e-89dc-a0ab9671bb9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7\" (UID: \"3cf3962c-f594-478e-89dc-a0ab9671bb9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7"
Oct 30 01:45:00.593532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.592779    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd1b902-d5db-4613-90ea-30a92c1bf2b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k\" (UID: \"8cd1b902-d5db-4613-90ea-30a92c1bf2b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k"
Oct 30 01:45:00.593532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.592824    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf3962c-f594-478e-89dc-a0ab9671bb9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7\" (UID: \"3cf3962c-f594-478e-89dc-a0ab9671bb9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7"
Oct 30 01:45:00.593532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.593493    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b1d4c-dc08-416b-ad79-6e777474940c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7\" (UID: \"e77b1d4c-dc08-416b-ad79-6e777474940c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7"
Oct 30 01:45:00.617022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.616990    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2fk5\" (UniqueName: \"kubernetes.io/projected/e77b1d4c-dc08-416b-ad79-6e777474940c-kube-api-access-n2fk5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7\" (UID: \"e77b1d4c-dc08-416b-ad79-6e777474940c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7"
Oct 30 01:45:00.625557 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.623010    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2"
Oct 30 01:45:00.639837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.639769    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtkdk\" (UniqueName: \"kubernetes.io/projected/3cf3962c-f594-478e-89dc-a0ab9671bb9b-kube-api-access-wtkdk\") pod \"maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7\" (UID: \"3cf3962c-f594-478e-89dc-a0ab9671bb9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7"
Oct 30 01:45:00.640690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.640532    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khdh7\" (UniqueName: \"kubernetes.io/projected/8cd1b902-d5db-4613-90ea-30a92c1bf2b0-kube-api-access-khdh7\") pod \"maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k\" (UID: \"8cd1b902-d5db-4613-90ea-30a92c1bf2b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k"
Oct 30 01:45:00.676492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.675241    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9"
Oct 30 01:45:00.701225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.700900    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm"
Oct 30 01:45:00.724963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.724942    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d"
Oct 30 01:45:00.785854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.785817    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k"
Oct 30 01:45:00.807700 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.807655    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs"]
Oct 30 01:45:00.817480 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:45:00.812442    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336e3514_5287_4632_84f7_2673bc71a606.slice/crio-97c07333a34443fff685fdbc45af861ab0f58b0f77d3b48c6f02ddbe4722dc87 WatchSource:0}: Error finding container 97c07333a34443fff685fdbc45af861ab0f58b0f77d3b48c6f02ddbe4722dc87: Status 404 returned error can't find the container with id 97c07333a34443fff685fdbc45af861ab0f58b0f77d3b48c6f02ddbe4722dc87
Oct 30 01:45:00.841273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.841061    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7"
Oct 30 01:45:00.855882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:00.855853    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7"
Oct 30 01:45:01.013877 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:01.013848    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs" event={"ID":"336e3514-5287-4632-84f7-2673bc71a606","Type":"ContainerStarted","Data":"97c07333a34443fff685fdbc45af861ab0f58b0f77d3b48c6f02ddbe4722dc87"}
Oct 30 01:45:01.066368 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:01.066068    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj"]
Oct 30 01:45:01.070332 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:01.070309    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv"]
Oct 30 01:45:01.076100 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:45:01.076075    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebecef45_aca8_486c_af0c_33bc1380da5d.slice/crio-9132d1cee819fbeae251ee75985f5a7adf1cc50c07d2b334a747ab34215d41f7 WatchSource:0}: Error finding container 9132d1cee819fbeae251ee75985f5a7adf1cc50c07d2b334a747ab34215d41f7: Status 404 returned error can't find the container with id 9132d1cee819fbeae251ee75985f5a7adf1cc50c07d2b334a747ab34215d41f7
Oct 30 01:45:01.076379 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:45:01.076341    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee57e527_71ff_4a09_8db0_9840a6467fce.slice/crio-fdd69bf54b8a8b3e86ec910052e5582103e50bdde1bd6890d5ebc2890571229f WatchSource:0}: Error finding container fdd69bf54b8a8b3e86ec910052e5582103e50bdde1bd6890d5ebc2890571229f: Status 404 returned error can't find the container with id fdd69bf54b8a8b3e86ec910052e5582103e50bdde1bd6890d5ebc2890571229f
Oct 30 01:45:01.237636 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:01.237602    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2"]
Oct 30 01:45:01.256472 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:45:01.256437    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf779dae_521f_4f8a_9ad2_e7fe3948dcbc.slice/crio-400225d45a029ec129127cff30b3cf3804ba37e1e0c0c87f6c480a724e515f34 WatchSource:0}: Error finding container 400225d45a029ec129127cff30b3cf3804ba37e1e0c0c87f6c480a724e515f34: Status 404 returned error can't find the container with id 400225d45a029ec129127cff30b3cf3804ba37e1e0c0c87f6c480a724e515f34
Oct 30 01:45:01.271328 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:45:01.271299    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a87024_9da9_4842_afb2_063b447bd15a.slice/crio-39e01a46a38f3d74aa76282b834052050550a006105ac2cd2e2b0f4f7129d177 WatchSource:0}: Error finding container 39e01a46a38f3d74aa76282b834052050550a006105ac2cd2e2b0f4f7129d177: Status 404 returned error can't find the container with id 39e01a46a38f3d74aa76282b834052050550a006105ac2cd2e2b0f4f7129d177
Oct 30 01:45:01.272123 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:01.272100    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm"]
Oct 30 01:45:01.281745 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:01.281719    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9"]
Oct 30 01:45:01.285784 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:45:01.285758    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a82dd87_6039_44da_b690_297f5449c337.slice/crio-28d891bd860e8e947c8ea249602f387cfc5e6e30f950c4230cee346f52ee1dd1 WatchSource:0}: Error finding container 28d891bd860e8e947c8ea249602f387cfc5e6e30f950c4230cee346f52ee1dd1: Status 404 returned error can't find the container with id 28d891bd860e8e947c8ea249602f387cfc5e6e30f950c4230cee346f52ee1dd1
Oct 30 01:45:01.304630 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:01.304603    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p"]
Oct 30 01:45:01.315998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:01.315974    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d"]
Oct 30 01:45:01.321760 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:45:01.321715    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1bab87_9a09_48f6_9302_43b0254810e5.slice/crio-118d333264a6503706b0c5080d49a55ed80776bd50d0d7eee8fbaa9ccd4d3845 WatchSource:0}: Error finding container 118d333264a6503706b0c5080d49a55ed80776bd50d0d7eee8fbaa9ccd4d3845: Status 404 returned error can't find the container with id 118d333264a6503706b0c5080d49a55ed80776bd50d0d7eee8fbaa9ccd4d3845
Oct 30 01:45:01.357091 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:01.357049    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd"]
Oct 30 01:45:01.357404 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:45:01.357373    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357853d9_748f_44af_baa0_523e64187284.slice/crio-d6c61079e7fb299acefbf9d4495b6562d282070788bff1f1db27c1c242b06618 WatchSource:0}: Error finding container d6c61079e7fb299acefbf9d4495b6562d282070788bff1f1db27c1c242b06618: Status 404 returned error can't find the container with id d6c61079e7fb299acefbf9d4495b6562d282070788bff1f1db27c1c242b06618
Oct 30 01:45:01.376965 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:01.376937    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf"]
Oct 30 01:45:01.394289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:01.394269    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2"]
Oct 30 01:45:01.402855 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:01.402836    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k"]
Oct 30 01:45:01.425588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:01.425565    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7"]
Oct 30 01:45:01.439714 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:01.439693    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7"]
Oct 30 01:45:01.456147 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:45:01.456111    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaccf6b1_b7c1_437e_a634_84d62cf8d3b4.slice/crio-a0881d748e2c12ba9f914870a8ddfa7b3aa278e4f2f76ac93aca64ee955f6a12 WatchSource:0}: Error finding container a0881d748e2c12ba9f914870a8ddfa7b3aa278e4f2f76ac93aca64ee955f6a12: Status 404 returned error can't find the container with id a0881d748e2c12ba9f914870a8ddfa7b3aa278e4f2f76ac93aca64ee955f6a12
Oct 30 01:45:01.457241 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:45:01.457215    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44cf871_7767_4ad3_a9b4_e70a58a52ec9.slice/crio-9153eb76c14b25521a881686865c7c5a59e9bbb6443d16195be681102a825158 WatchSource:0}: Error finding container 9153eb76c14b25521a881686865c7c5a59e9bbb6443d16195be681102a825158: Status 404 returned error can't find the container with id 9153eb76c14b25521a881686865c7c5a59e9bbb6443d16195be681102a825158
Oct 30 01:45:01.458653 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:45:01.458623    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd1b902_d5db_4613_90ea_30a92c1bf2b0.slice/crio-39ff05363bd13feded1da7a67169b89492f8b040019fe3115301d8118c05ea07 WatchSource:0}: Error finding container 39ff05363bd13feded1da7a67169b89492f8b040019fe3115301d8118c05ea07: Status 404 returned error can't find the container with id 39ff05363bd13feded1da7a67169b89492f8b040019fe3115301d8118c05ea07
Oct 30 01:45:01.552927 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:45:01.552866    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77b1d4c_dc08_416b_ad79_6e777474940c.slice/crio-025622ab2bf6ec03484a5d3af21a3672c62886daffb45358fc8822287eed4fc7 WatchSource:0}: Error finding container 025622ab2bf6ec03484a5d3af21a3672c62886daffb45358fc8822287eed4fc7: Status 404 returned error can't find the container with id 025622ab2bf6ec03484a5d3af21a3672c62886daffb45358fc8822287eed4fc7
Oct 30 01:45:01.555439 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:45:01.555253    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf3962c_f594_478e_89dc_a0ab9671bb9b.slice/crio-12e5bcb1d4482cf0a4ec1e2e25ffea7bf1e53034c8a22b124f4f1fc37819e842 WatchSource:0}: Error finding container 12e5bcb1d4482cf0a4ec1e2e25ffea7bf1e53034c8a22b124f4f1fc37819e842: Status 404 returned error can't find the container with id 12e5bcb1d4482cf0a4ec1e2e25ffea7bf1e53034c8a22b124f4f1fc37819e842
Oct 30 01:45:02.035079 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.035041    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7" event={"ID":"e77b1d4c-dc08-416b-ad79-6e777474940c","Type":"ContainerStarted","Data":"025622ab2bf6ec03484a5d3af21a3672c62886daffb45358fc8822287eed4fc7"}
Oct 30 01:45:02.036728 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.036695    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9" event={"ID":"9a82dd87-6039-44da-b690-297f5449c337","Type":"ContainerStarted","Data":"3fa1cc8fa4d4d315aecfa49f3fb34b67359686de69de68e18602b694cc984bd8"}
Oct 30 01:45:02.036870 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.036742    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9" event={"ID":"9a82dd87-6039-44da-b690-297f5449c337","Type":"ContainerStarted","Data":"28d891bd860e8e947c8ea249602f387cfc5e6e30f950c4230cee346f52ee1dd1"}
Oct 30 01:45:02.038264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.038242    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv" event={"ID":"ee57e527-71ff-4a09-8db0-9840a6467fce","Type":"ContainerStarted","Data":"0c9506df212ae613e507a41f6e0717aee9948533b15316a6537b6a678a1eaaac"}
Oct 30 01:45:02.038342 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.038282    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv" event={"ID":"ee57e527-71ff-4a09-8db0-9840a6467fce","Type":"ContainerStarted","Data":"fdd69bf54b8a8b3e86ec910052e5582103e50bdde1bd6890d5ebc2890571229f"}
Oct 30 01:45:02.039781 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.039758    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm" event={"ID":"a2a87024-9da9-4842-afb2-063b447bd15a","Type":"ContainerStarted","Data":"99dc5143768b1f0b871327331457f3a9a8d15c9728f49e1112f1702c8cdb5987"}
Oct 30 01:45:02.039863 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.039795    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm" event={"ID":"a2a87024-9da9-4842-afb2-063b447bd15a","Type":"ContainerStarted","Data":"39e01a46a38f3d74aa76282b834052050550a006105ac2cd2e2b0f4f7129d177"}
Oct 30 01:45:02.041303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.041282    4425 generic.go:356] "Generic (PLEG): container finished" podID="336e3514-5287-4632-84f7-2673bc71a606" containerID="e4b8e9aefadef930e621d661525ffb4ff185dc08391110fe57f89fd06f752c9e" exitCode=60
Oct 30 01:45:02.041384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.041346    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs" event={"ID":"336e3514-5287-4632-84f7-2673bc71a606","Type":"ContainerDied","Data":"e4b8e9aefadef930e621d661525ffb4ff185dc08391110fe57f89fd06f752c9e"}
Oct 30 01:45:02.041536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.041520    4425 scope.go:122] "RemoveContainer" containerID="e4b8e9aefadef930e621d661525ffb4ff185dc08391110fe57f89fd06f752c9e"
Oct 30 01:45:02.043168 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.043100    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj" event={"ID":"ebecef45-aca8-486c-af0c-33bc1380da5d","Type":"ContainerStarted","Data":"aa80fd748767b2abd0c63fedaed4e80d88b8b9739fa75c0b445cac5c791c1c29"}
Oct 30 01:45:02.043168 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.043132    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj" event={"ID":"ebecef45-aca8-486c-af0c-33bc1380da5d","Type":"ContainerStarted","Data":"9132d1cee819fbeae251ee75985f5a7adf1cc50c07d2b334a747ab34215d41f7"}
Oct 30 01:45:02.044458 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.044438    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd" event={"ID":"357853d9-748f-44af-baa0-523e64187284","Type":"ContainerStarted","Data":"d6c61079e7fb299acefbf9d4495b6562d282070788bff1f1db27c1c242b06618"}
Oct 30 01:45:02.045751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.045733    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2" event={"ID":"d44cf871-7767-4ad3-a9b4-e70a58a52ec9","Type":"ContainerStarted","Data":"9153eb76c14b25521a881686865c7c5a59e9bbb6443d16195be681102a825158"}
Oct 30 01:45:02.047056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.047035    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k" event={"ID":"8cd1b902-d5db-4613-90ea-30a92c1bf2b0","Type":"ContainerStarted","Data":"39ff05363bd13feded1da7a67169b89492f8b040019fe3115301d8118c05ea07"}
Oct 30 01:45:02.048611 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.048586    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d" event={"ID":"6c1bab87-9a09-48f6-9302-43b0254810e5","Type":"ContainerStarted","Data":"118d333264a6503706b0c5080d49a55ed80776bd50d0d7eee8fbaa9ccd4d3845"}
Oct 30 01:45:02.060133 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.060106    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7" event={"ID":"3cf3962c-f594-478e-89dc-a0ab9671bb9b","Type":"ContainerStarted","Data":"12e5bcb1d4482cf0a4ec1e2e25ffea7bf1e53034c8a22b124f4f1fc37819e842"}
Oct 30 01:45:02.063515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.063490    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2" event={"ID":"af779dae-521f-4f8a-9ad2-e7fe3948dcbc","Type":"ContainerStarted","Data":"c3ea30508e025481b904db1b726655deda5c1a00668222fd51d2dc17d335fd9c"}
Oct 30 01:45:02.063595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.063536    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2" event={"ID":"af779dae-521f-4f8a-9ad2-e7fe3948dcbc","Type":"ContainerStarted","Data":"400225d45a029ec129127cff30b3cf3804ba37e1e0c0c87f6c480a724e515f34"}
Oct 30 01:45:02.065681 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.065656    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf" event={"ID":"caccf6b1-b7c1-437e-a634-84d62cf8d3b4","Type":"ContainerStarted","Data":"a0881d748e2c12ba9f914870a8ddfa7b3aa278e4f2f76ac93aca64ee955f6a12"}
Oct 30 01:45:02.094962 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.092334    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9" podStartSLOduration=2.092316927 podStartE2EDuration="2.092316927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:02.083762131 +0000 UTC m=+3810.833417441" watchObservedRunningTime="2026-10-30 01:45:02.092316927 +0000 UTC m=+3810.841972236"
Oct 30 01:45:02.102236 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.101812    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p" event={"ID":"e253e33e-8ac1-4658-aaf1-485bb678f1b5","Type":"ContainerStarted","Data":"802a05980d501b17b8d559dc383ef8eacf099101273b45766fcd49fc05d296cd"}
Oct 30 01:45:02.128799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.128747    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv" podStartSLOduration=2.128727809 podStartE2EDuration="2.128727809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:02.122119851 +0000 UTC m=+3810.871775161" watchObservedRunningTime="2026-10-30 01:45:02.128727809 +0000 UTC m=+3810.878383120"
Oct 30 01:45:02.162821 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.162777    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm" podStartSLOduration=2.162757203 podStartE2EDuration="2.162757203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:02.157997819 +0000 UTC m=+3810.907653129" watchObservedRunningTime="2026-10-30 01:45:02.162757203 +0000 UTC m=+3810.912412513"
Oct 30 01:45:02.238993 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.238871    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj" podStartSLOduration=2.238853458 podStartE2EDuration="2.238853458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:02.235643474 +0000 UTC m=+3810.985298783" watchObservedRunningTime="2026-10-30 01:45:02.238853458 +0000 UTC m=+3810.988508769"
Oct 30 01:45:02.292101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:02.292042    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2" podStartSLOduration=2.292023013 podStartE2EDuration="2.292023013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:02.284653507 +0000 UTC m=+3811.034308816" watchObservedRunningTime="2026-10-30 01:45:02.292023013 +0000 UTC m=+3811.041678332"
Oct 30 01:45:03.112507 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.112477    4425 generic.go:356] "Generic (PLEG): container finished" podID="caccf6b1-b7c1-437e-a634-84d62cf8d3b4" containerID="824d7a16e1cb753da6dca71c561275804cc2fcebabb5ec5f4539eb419fab750c" exitCode=60
Oct 30 01:45:03.112673 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.112581    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf" event={"ID":"caccf6b1-b7c1-437e-a634-84d62cf8d3b4","Type":"ContainerDied","Data":"824d7a16e1cb753da6dca71c561275804cc2fcebabb5ec5f4539eb419fab750c"}
Oct 30 01:45:03.112953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.112933    4425 scope.go:122] "RemoveContainer" containerID="824d7a16e1cb753da6dca71c561275804cc2fcebabb5ec5f4539eb419fab750c"
Oct 30 01:45:03.114411 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.114390    4425 generic.go:356] "Generic (PLEG): container finished" podID="e253e33e-8ac1-4658-aaf1-485bb678f1b5" containerID="f715b06ae853cb2029c87aeca46d9ffc8b78a12792d2827e895aa5ad04b1f5f4" exitCode=60
Oct 30 01:45:03.114506 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.114439    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p" event={"ID":"e253e33e-8ac1-4658-aaf1-485bb678f1b5","Type":"ContainerDied","Data":"f715b06ae853cb2029c87aeca46d9ffc8b78a12792d2827e895aa5ad04b1f5f4"}
Oct 30 01:45:03.114710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.114695    4425 scope.go:122] "RemoveContainer" containerID="f715b06ae853cb2029c87aeca46d9ffc8b78a12792d2827e895aa5ad04b1f5f4"
Oct 30 01:45:03.116966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.116864    4425 generic.go:356] "Generic (PLEG): container finished" podID="e77b1d4c-dc08-416b-ad79-6e777474940c" containerID="da5845cad102e2663183054f5a2c146b8ce8d67f25c83f7203e98f601e09700d" exitCode=60
Oct 30 01:45:03.116966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.116896    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7" event={"ID":"e77b1d4c-dc08-416b-ad79-6e777474940c","Type":"ContainerDied","Data":"da5845cad102e2663183054f5a2c146b8ce8d67f25c83f7203e98f601e09700d"}
Oct 30 01:45:03.117216 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.117194    4425 scope.go:122] "RemoveContainer" containerID="da5845cad102e2663183054f5a2c146b8ce8d67f25c83f7203e98f601e09700d"
Oct 30 01:45:03.119648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.119408    4425 generic.go:356] "Generic (PLEG): container finished" podID="9a82dd87-6039-44da-b690-297f5449c337" containerID="3fa1cc8fa4d4d315aecfa49f3fb34b67359686de69de68e18602b694cc984bd8" exitCode=60
Oct 30 01:45:03.119648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.119469    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9" event={"ID":"9a82dd87-6039-44da-b690-297f5449c337","Type":"ContainerDied","Data":"3fa1cc8fa4d4d315aecfa49f3fb34b67359686de69de68e18602b694cc984bd8"}
Oct 30 01:45:03.119768 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.119669    4425 scope.go:122] "RemoveContainer" containerID="3fa1cc8fa4d4d315aecfa49f3fb34b67359686de69de68e18602b694cc984bd8"
Oct 30 01:45:03.121485 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.121464    4425 generic.go:356] "Generic (PLEG): container finished" podID="ee57e527-71ff-4a09-8db0-9840a6467fce" containerID="0c9506df212ae613e507a41f6e0717aee9948533b15316a6537b6a678a1eaaac" exitCode=60
Oct 30 01:45:03.121595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.121543    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv" event={"ID":"ee57e527-71ff-4a09-8db0-9840a6467fce","Type":"ContainerDied","Data":"0c9506df212ae613e507a41f6e0717aee9948533b15316a6537b6a678a1eaaac"}
Oct 30 01:45:03.121770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.121755    4425 scope.go:122] "RemoveContainer" containerID="0c9506df212ae613e507a41f6e0717aee9948533b15316a6537b6a678a1eaaac"
Oct 30 01:45:03.123299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.123278    4425 generic.go:356] "Generic (PLEG): container finished" podID="a2a87024-9da9-4842-afb2-063b447bd15a" containerID="99dc5143768b1f0b871327331457f3a9a8d15c9728f49e1112f1702c8cdb5987" exitCode=60
Oct 30 01:45:03.123400 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.123347    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm" event={"ID":"a2a87024-9da9-4842-afb2-063b447bd15a","Type":"ContainerDied","Data":"99dc5143768b1f0b871327331457f3a9a8d15c9728f49e1112f1702c8cdb5987"}
Oct 30 01:45:03.123632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.123613    4425 scope.go:122] "RemoveContainer" containerID="99dc5143768b1f0b871327331457f3a9a8d15c9728f49e1112f1702c8cdb5987"
Oct 30 01:45:03.125792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.125771    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs" event={"ID":"336e3514-5287-4632-84f7-2673bc71a606","Type":"ContainerStarted","Data":"4e9b0ad353abaf08a8b651dada3d9321cb9eca5e74972a0a4e5e7430e03e97c8"}
Oct 30 01:45:03.128440 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.128367    4425 generic.go:356] "Generic (PLEG): container finished" podID="ebecef45-aca8-486c-af0c-33bc1380da5d" containerID="aa80fd748767b2abd0c63fedaed4e80d88b8b9739fa75c0b445cac5c791c1c29" exitCode=60
Oct 30 01:45:03.128504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.128473    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj" event={"ID":"ebecef45-aca8-486c-af0c-33bc1380da5d","Type":"ContainerDied","Data":"aa80fd748767b2abd0c63fedaed4e80d88b8b9739fa75c0b445cac5c791c1c29"}
Oct 30 01:45:03.128792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.128772    4425 scope.go:122] "RemoveContainer" containerID="aa80fd748767b2abd0c63fedaed4e80d88b8b9739fa75c0b445cac5c791c1c29"
Oct 30 01:45:03.130644 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.130623    4425 generic.go:356] "Generic (PLEG): container finished" podID="357853d9-748f-44af-baa0-523e64187284" containerID="cc5ee4ad3b24f70ca43915e7c76f154d7cbf8ebdf7264d1c94052953a8709503" exitCode=60
Oct 30 01:45:03.130764 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.130679    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd" event={"ID":"357853d9-748f-44af-baa0-523e64187284","Type":"ContainerDied","Data":"cc5ee4ad3b24f70ca43915e7c76f154d7cbf8ebdf7264d1c94052953a8709503"}
Oct 30 01:45:03.130922 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.130900    4425 scope.go:122] "RemoveContainer" containerID="cc5ee4ad3b24f70ca43915e7c76f154d7cbf8ebdf7264d1c94052953a8709503"
Oct 30 01:45:03.132519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.132498    4425 generic.go:356] "Generic (PLEG): container finished" podID="d44cf871-7767-4ad3-a9b4-e70a58a52ec9" containerID="c9c6a985c8664cf9fd8fa69531127bce2b1574c1da5e039534f03d62d086d004" exitCode=60
Oct 30 01:45:03.132611 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.132567    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2" event={"ID":"d44cf871-7767-4ad3-a9b4-e70a58a52ec9","Type":"ContainerDied","Data":"c9c6a985c8664cf9fd8fa69531127bce2b1574c1da5e039534f03d62d086d004"}
Oct 30 01:45:03.132749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.132736    4425 scope.go:122] "RemoveContainer" containerID="c9c6a985c8664cf9fd8fa69531127bce2b1574c1da5e039534f03d62d086d004"
Oct 30 01:45:03.134295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.134269    4425 generic.go:356] "Generic (PLEG): container finished" podID="8cd1b902-d5db-4613-90ea-30a92c1bf2b0" containerID="c1b58d856ac3b29bd3a63e846d882db045b46e839703e9d9627c47408ab96ae4" exitCode=60
Oct 30 01:45:03.134557 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.134537    4425 scope.go:122] "RemoveContainer" containerID="c1b58d856ac3b29bd3a63e846d882db045b46e839703e9d9627c47408ab96ae4"
Oct 30 01:45:03.134654 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.134592    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k" event={"ID":"8cd1b902-d5db-4613-90ea-30a92c1bf2b0","Type":"ContainerDied","Data":"c1b58d856ac3b29bd3a63e846d882db045b46e839703e9d9627c47408ab96ae4"}
Oct 30 01:45:03.135887 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.135866    4425 generic.go:356] "Generic (PLEG): container finished" podID="6c1bab87-9a09-48f6-9302-43b0254810e5" containerID="533667ebcace24d81929cc921fae13668518d73e459461e199d747958cb71f92" exitCode=60
Oct 30 01:45:03.136030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.135913    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d" event={"ID":"6c1bab87-9a09-48f6-9302-43b0254810e5","Type":"ContainerDied","Data":"533667ebcace24d81929cc921fae13668518d73e459461e199d747958cb71f92"}
Oct 30 01:45:03.136218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.136198    4425 scope.go:122] "RemoveContainer" containerID="533667ebcace24d81929cc921fae13668518d73e459461e199d747958cb71f92"
Oct 30 01:45:03.140040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.139414    4425 generic.go:356] "Generic (PLEG): container finished" podID="3cf3962c-f594-478e-89dc-a0ab9671bb9b" containerID="79b3ccf610de7281e9ae75833fb6b4ef5025ffff1d5cd154a613641e27f89dd4" exitCode=60
Oct 30 01:45:03.140040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.139543    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7" event={"ID":"3cf3962c-f594-478e-89dc-a0ab9671bb9b","Type":"ContainerDied","Data":"79b3ccf610de7281e9ae75833fb6b4ef5025ffff1d5cd154a613641e27f89dd4"}
Oct 30 01:45:03.140418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.140188    4425 scope.go:122] "RemoveContainer" containerID="79b3ccf610de7281e9ae75833fb6b4ef5025ffff1d5cd154a613641e27f89dd4"
Oct 30 01:45:03.142608 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.142587    4425 generic.go:356] "Generic (PLEG): container finished" podID="af779dae-521f-4f8a-9ad2-e7fe3948dcbc" containerID="c3ea30508e025481b904db1b726655deda5c1a00668222fd51d2dc17d335fd9c" exitCode=60
Oct 30 01:45:03.142721 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.142635    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2" event={"ID":"af779dae-521f-4f8a-9ad2-e7fe3948dcbc","Type":"ContainerDied","Data":"c3ea30508e025481b904db1b726655deda5c1a00668222fd51d2dc17d335fd9c"}
Oct 30 01:45:03.142916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.142896    4425 scope.go:122] "RemoveContainer" containerID="c3ea30508e025481b904db1b726655deda5c1a00668222fd51d2dc17d335fd9c"
Oct 30 01:45:03.417423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:03.417330    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs" podStartSLOduration=3.417317162 podStartE2EDuration="3.417317162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:03.413847274 +0000 UTC m=+3812.163502579" watchObservedRunningTime="2026-10-30 01:45:03.417317162 +0000 UTC m=+3812.166972467"
Oct 30 01:45:04.078902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.078878    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"]
Oct 30 01:45:04.158487 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.158453    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2" event={"ID":"d44cf871-7767-4ad3-a9b4-e70a58a52ec9","Type":"ContainerStarted","Data":"2412ba6025f4f27a3515e1b2652864fd9e9442f36c209c168f266b313e10e896"}
Oct 30 01:45:04.162286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.162226    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k" event={"ID":"8cd1b902-d5db-4613-90ea-30a92c1bf2b0","Type":"ContainerStarted","Data":"d8db9937bdd5b1ef3dcd5b27384d157f831f9aeecb9ee6bab4d9c3183982bf93"}
Oct 30 01:45:04.165578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.165548    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d" event={"ID":"6c1bab87-9a09-48f6-9302-43b0254810e5","Type":"ContainerStarted","Data":"a86342ec83950b5cb5ab68bf86a1fd9e0e90ce104d91c1865ddce4ce36fdda82"}
Oct 30 01:45:04.172341 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.171940    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7" event={"ID":"3cf3962c-f594-478e-89dc-a0ab9671bb9b","Type":"ContainerStarted","Data":"b940451841c081d35304a5f09bd597c9bc5e9fe81fb4a4bb993bf2dd0a46efb8"}
Oct 30 01:45:04.178829 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.178806    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf" event={"ID":"caccf6b1-b7c1-437e-a634-84d62cf8d3b4","Type":"ContainerStarted","Data":"b041af6c4bb1711c0a35a75ca8951e42cea483e7c3cf0fbbea5955e48b75b894"}
Oct 30 01:45:04.198833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.181956    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p" event={"ID":"e253e33e-8ac1-4658-aaf1-485bb678f1b5","Type":"ContainerStarted","Data":"06d3f09481afccdd2863a5b5a5d7b03bba010d6313c95623c63f64d4cfccf10d"}
Oct 30 01:45:04.198833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.184135    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7" event={"ID":"e77b1d4c-dc08-416b-ad79-6e777474940c","Type":"ContainerStarted","Data":"113a15f876239dbc47cf75ebd796c82e22d35650bc0ee5db282a58197759a80a"}
Oct 30 01:45:04.198833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.187216    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9" event={"ID":"9a82dd87-6039-44da-b690-297f5449c337","Type":"ContainerStarted","Data":"2d3371a41423945d879100cc9269f5cb6617d8200b5f38dd7481a56dddba988b"}
Oct 30 01:45:04.198833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.191749    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv" event={"ID":"ee57e527-71ff-4a09-8db0-9840a6467fce","Type":"ContainerStarted","Data":"894605233a89a544380aa7b282437375745b111a887a86c6f04114b057a0fb03"}
Oct 30 01:45:04.198833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.194010    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm" event={"ID":"a2a87024-9da9-4842-afb2-063b447bd15a","Type":"ContainerStarted","Data":"6ed0e2bb1812dda3a8bd9ab01a8862a734143da0786c2956d486148103039138"}
Oct 30 01:45:04.198833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.197037    4425 generic.go:356] "Generic (PLEG): container finished" podID="336e3514-5287-4632-84f7-2673bc71a606" containerID="4e9b0ad353abaf08a8b651dada3d9321cb9eca5e74972a0a4e5e7430e03e97c8" exitCode=60
Oct 30 01:45:04.198833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.197117    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs" event={"ID":"336e3514-5287-4632-84f7-2673bc71a606","Type":"ContainerDied","Data":"4e9b0ad353abaf08a8b651dada3d9321cb9eca5e74972a0a4e5e7430e03e97c8"}
Oct 30 01:45:04.198833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.197180    4425 scope.go:122] "RemoveContainer" containerID="e4b8e9aefadef930e621d661525ffb4ff185dc08391110fe57f89fd06f752c9e"
Oct 30 01:45:04.198833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.197551    4425 scope.go:122] "RemoveContainer" containerID="4e9b0ad353abaf08a8b651dada3d9321cb9eca5e74972a0a4e5e7430e03e97c8"
Oct 30 01:45:04.198833 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:04.197796    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs_odh-ai-gateway-infra(336e3514-5287-4632-84f7-2673bc71a606)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs" podUID="336e3514-5287-4632-84f7-2673bc71a606"
Oct 30 01:45:04.200020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.199973    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2" podStartSLOduration=4.199956532 podStartE2EDuration="4.199956532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:04.19660906 +0000 UTC m=+3812.946264369" watchObservedRunningTime="2026-10-30 01:45:04.199956532 +0000 UTC m=+3812.949611843"
Oct 30 01:45:04.206720 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.206647    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj" event={"ID":"ebecef45-aca8-486c-af0c-33bc1380da5d","Type":"ContainerStarted","Data":"083313e8ae95fc1061277759b63caf4d78509885425cd6e4b3e1882d60ec2e88"}
Oct 30 01:45:04.242121 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.242080    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p" podStartSLOduration=4.242067688 podStartE2EDuration="4.242067688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:04.241126913 +0000 UTC m=+3812.990782223" watchObservedRunningTime="2026-10-30 01:45:04.242067688 +0000 UTC m=+3812.991722997"
Oct 30 01:45:04.339465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.339360    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7" podStartSLOduration=4.339346656 podStartE2EDuration="4.339346656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:04.334365636 +0000 UTC m=+3813.084020946" watchObservedRunningTime="2026-10-30 01:45:04.339346656 +0000 UTC m=+3813.089001964"
Oct 30 01:45:04.385339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.385273    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7" podStartSLOduration=4.385230995 podStartE2EDuration="4.385230995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:04.384203473 +0000 UTC m=+3813.133858782" watchObservedRunningTime="2026-10-30 01:45:04.385230995 +0000 UTC m=+3813.134886305"
Oct 30 01:45:04.479756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.479708    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf" podStartSLOduration=4.479694719 podStartE2EDuration="4.479694719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:04.471059097 +0000 UTC m=+3813.220714406" watchObservedRunningTime="2026-10-30 01:45:04.479694719 +0000 UTC m=+3813.229350028"
Oct 30 01:45:04.616340 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.616256    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k" podStartSLOduration=4.616241874 podStartE2EDuration="4.616241874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:04.61296264 +0000 UTC m=+3813.362617949" watchObservedRunningTime="2026-10-30 01:45:04.616241874 +0000 UTC m=+3813.365897184"
Oct 30 01:45:04.738311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:04.738265    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d" podStartSLOduration=4.738252584 podStartE2EDuration="4.738252584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:04.737071798 +0000 UTC m=+3813.486727117" watchObservedRunningTime="2026-10-30 01:45:04.738252584 +0000 UTC m=+3813.487907892"
Oct 30 01:45:05.224258 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.224147    4425 generic.go:356] "Generic (PLEG): container finished" podID="ee57e527-71ff-4a09-8db0-9840a6467fce" containerID="894605233a89a544380aa7b282437375745b111a887a86c6f04114b057a0fb03" exitCode=60
Oct 30 01:45:05.224405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.224293    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv" event={"ID":"ee57e527-71ff-4a09-8db0-9840a6467fce","Type":"ContainerDied","Data":"894605233a89a544380aa7b282437375745b111a887a86c6f04114b057a0fb03"}
Oct 30 01:45:05.224405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.224347    4425 scope.go:122] "RemoveContainer" containerID="0c9506df212ae613e507a41f6e0717aee9948533b15316a6537b6a678a1eaaac"
Oct 30 01:45:05.224768 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.224753    4425 scope.go:122] "RemoveContainer" containerID="894605233a89a544380aa7b282437375745b111a887a86c6f04114b057a0fb03"
Oct 30 01:45:05.225097 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:05.225075    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv_odh-ai-gateway-infra(ee57e527-71ff-4a09-8db0-9840a6467fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv" podUID="ee57e527-71ff-4a09-8db0-9840a6467fce"
Oct 30 01:45:05.227690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.227661    4425 generic.go:356] "Generic (PLEG): container finished" podID="a2a87024-9da9-4842-afb2-063b447bd15a" containerID="6ed0e2bb1812dda3a8bd9ab01a8862a734143da0786c2956d486148103039138" exitCode=60
Oct 30 01:45:05.227837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.227735    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm" event={"ID":"a2a87024-9da9-4842-afb2-063b447bd15a","Type":"ContainerDied","Data":"6ed0e2bb1812dda3a8bd9ab01a8862a734143da0786c2956d486148103039138"}
Oct 30 01:45:05.227998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.227980    4425 scope.go:122] "RemoveContainer" containerID="6ed0e2bb1812dda3a8bd9ab01a8862a734143da0786c2956d486148103039138"
Oct 30 01:45:05.228308 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:05.228282    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm_odh-ai-gateway-infra(a2a87024-9da9-4842-afb2-063b447bd15a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm" podUID="a2a87024-9da9-4842-afb2-063b447bd15a"
Oct 30 01:45:05.229833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.229813    4425 scope.go:122] "RemoveContainer" containerID="4e9b0ad353abaf08a8b651dada3d9321cb9eca5e74972a0a4e5e7430e03e97c8"
Oct 30 01:45:05.230115 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:05.230090    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs_odh-ai-gateway-infra(336e3514-5287-4632-84f7-2673bc71a606)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs" podUID="336e3514-5287-4632-84f7-2673bc71a606"
Oct 30 01:45:05.231493 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.231474    4425 generic.go:356] "Generic (PLEG): container finished" podID="ebecef45-aca8-486c-af0c-33bc1380da5d" containerID="083313e8ae95fc1061277759b63caf4d78509885425cd6e4b3e1882d60ec2e88" exitCode=60
Oct 30 01:45:05.231603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.231549    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj" event={"ID":"ebecef45-aca8-486c-af0c-33bc1380da5d","Type":"ContainerDied","Data":"083313e8ae95fc1061277759b63caf4d78509885425cd6e4b3e1882d60ec2e88"}
Oct 30 01:45:05.231786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.231768    4425 scope.go:122] "RemoveContainer" containerID="083313e8ae95fc1061277759b63caf4d78509885425cd6e4b3e1882d60ec2e88"
Oct 30 01:45:05.232042 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:05.232022    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj_odh-ai-gateway-infra(ebecef45-aca8-486c-af0c-33bc1380da5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj" podUID="ebecef45-aca8-486c-af0c-33bc1380da5d"
Oct 30 01:45:05.233834 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.233808    4425 generic.go:356] "Generic (PLEG): container finished" podID="357853d9-748f-44af-baa0-523e64187284" containerID="fddbc4fe922f699f598f0bab7e53463e80ce373747b0281c8a79b86affcef87a" exitCode=60
Oct 30 01:45:05.233932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.233886    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd" event={"ID":"357853d9-748f-44af-baa0-523e64187284","Type":"ContainerDied","Data":"fddbc4fe922f699f598f0bab7e53463e80ce373747b0281c8a79b86affcef87a"}
Oct 30 01:45:05.234144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.234124    4425 scope.go:122] "RemoveContainer" containerID="fddbc4fe922f699f598f0bab7e53463e80ce373747b0281c8a79b86affcef87a"
Oct 30 01:45:05.234423 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:05.234404    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd_odh-ai-gateway-infra(357853d9-748f-44af-baa0-523e64187284)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd" podUID="357853d9-748f-44af-baa0-523e64187284"
Oct 30 01:45:05.235949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.235930    4425 generic.go:356] "Generic (PLEG): container finished" podID="d44cf871-7767-4ad3-a9b4-e70a58a52ec9" containerID="2412ba6025f4f27a3515e1b2652864fd9e9442f36c209c168f266b313e10e896" exitCode=60
Oct 30 01:45:05.236032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.235975    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2" event={"ID":"d44cf871-7767-4ad3-a9b4-e70a58a52ec9","Type":"ContainerDied","Data":"2412ba6025f4f27a3515e1b2652864fd9e9442f36c209c168f266b313e10e896"}
Oct 30 01:45:05.236256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.236242    4425 scope.go:122] "RemoveContainer" containerID="2412ba6025f4f27a3515e1b2652864fd9e9442f36c209c168f266b313e10e896"
Oct 30 01:45:05.236482 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:05.236464    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2_odh-ai-gateway-infra(d44cf871-7767-4ad3-a9b4-e70a58a52ec9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2" podUID="d44cf871-7767-4ad3-a9b4-e70a58a52ec9"
Oct 30 01:45:05.237911 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.237890    4425 generic.go:356] "Generic (PLEG): container finished" podID="8cd1b902-d5db-4613-90ea-30a92c1bf2b0" containerID="d8db9937bdd5b1ef3dcd5b27384d157f831f9aeecb9ee6bab4d9c3183982bf93" exitCode=60
Oct 30 01:45:05.238012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.237970    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k" event={"ID":"8cd1b902-d5db-4613-90ea-30a92c1bf2b0","Type":"ContainerDied","Data":"d8db9937bdd5b1ef3dcd5b27384d157f831f9aeecb9ee6bab4d9c3183982bf93"}
Oct 30 01:45:05.238208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.238192    4425 scope.go:122] "RemoveContainer" containerID="d8db9937bdd5b1ef3dcd5b27384d157f831f9aeecb9ee6bab4d9c3183982bf93"
Oct 30 01:45:05.238411 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:05.238391    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k_odh-ai-gateway-infra(8cd1b902-d5db-4613-90ea-30a92c1bf2b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k" podUID="8cd1b902-d5db-4613-90ea-30a92c1bf2b0"
Oct 30 01:45:05.239848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.239825    4425 generic.go:356] "Generic (PLEG): container finished" podID="6c1bab87-9a09-48f6-9302-43b0254810e5" containerID="a86342ec83950b5cb5ab68bf86a1fd9e0e90ce104d91c1865ddce4ce36fdda82" exitCode=60
Oct 30 01:45:05.239920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.239901    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d" event={"ID":"6c1bab87-9a09-48f6-9302-43b0254810e5","Type":"ContainerDied","Data":"a86342ec83950b5cb5ab68bf86a1fd9e0e90ce104d91c1865ddce4ce36fdda82"}
Oct 30 01:45:05.240080 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.240065    4425 scope.go:122] "RemoveContainer" containerID="a86342ec83950b5cb5ab68bf86a1fd9e0e90ce104d91c1865ddce4ce36fdda82"
Oct 30 01:45:05.240293 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:05.240276    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d_odh-ai-gateway-infra(6c1bab87-9a09-48f6-9302-43b0254810e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d" podUID="6c1bab87-9a09-48f6-9302-43b0254810e5"
Oct 30 01:45:05.241333 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.241314    4425 scope.go:122] "RemoveContainer" containerID="99dc5143768b1f0b871327331457f3a9a8d15c9728f49e1112f1702c8cdb5987"
Oct 30 01:45:05.241782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.241759    4425 generic.go:356] "Generic (PLEG): container finished" podID="3cf3962c-f594-478e-89dc-a0ab9671bb9b" containerID="b940451841c081d35304a5f09bd597c9bc5e9fe81fb4a4bb993bf2dd0a46efb8" exitCode=60
Oct 30 01:45:05.241891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.241825    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7" event={"ID":"3cf3962c-f594-478e-89dc-a0ab9671bb9b","Type":"ContainerDied","Data":"b940451841c081d35304a5f09bd597c9bc5e9fe81fb4a4bb993bf2dd0a46efb8"}
Oct 30 01:45:05.242089 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.242068    4425 scope.go:122] "RemoveContainer" containerID="b940451841c081d35304a5f09bd597c9bc5e9fe81fb4a4bb993bf2dd0a46efb8"
Oct 30 01:45:05.242473 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:05.242431    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7_odh-ai-gateway-infra(3cf3962c-f594-478e-89dc-a0ab9671bb9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7" podUID="3cf3962c-f594-478e-89dc-a0ab9671bb9b"
Oct 30 01:45:05.243836 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.243813    4425 generic.go:356] "Generic (PLEG): container finished" podID="af779dae-521f-4f8a-9ad2-e7fe3948dcbc" containerID="eb2ce24a9782de2cfde1f323c69b7d0ef41cd52dac8e4e4de410da3584f5d00d" exitCode=60
Oct 30 01:45:05.243935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.243898    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2" event={"ID":"af779dae-521f-4f8a-9ad2-e7fe3948dcbc","Type":"ContainerDied","Data":"eb2ce24a9782de2cfde1f323c69b7d0ef41cd52dac8e4e4de410da3584f5d00d"}
Oct 30 01:45:05.244112 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.244097    4425 scope.go:122] "RemoveContainer" containerID="eb2ce24a9782de2cfde1f323c69b7d0ef41cd52dac8e4e4de410da3584f5d00d"
Oct 30 01:45:05.244335 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:05.244314    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2_odh-ai-gateway-infra(af779dae-521f-4f8a-9ad2-e7fe3948dcbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2" podUID="af779dae-521f-4f8a-9ad2-e7fe3948dcbc"
Oct 30 01:45:05.245949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.245930    4425 generic.go:356] "Generic (PLEG): container finished" podID="caccf6b1-b7c1-437e-a634-84d62cf8d3b4" containerID="b041af6c4bb1711c0a35a75ca8951e42cea483e7c3cf0fbbea5955e48b75b894" exitCode=60
Oct 30 01:45:05.246052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.246001    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf" event={"ID":"caccf6b1-b7c1-437e-a634-84d62cf8d3b4","Type":"ContainerDied","Data":"b041af6c4bb1711c0a35a75ca8951e42cea483e7c3cf0fbbea5955e48b75b894"}
Oct 30 01:45:05.246245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.246227    4425 scope.go:122] "RemoveContainer" containerID="b041af6c4bb1711c0a35a75ca8951e42cea483e7c3cf0fbbea5955e48b75b894"
Oct 30 01:45:05.246470 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:05.246447    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf_odh-ai-gateway-infra(caccf6b1-b7c1-437e-a634-84d62cf8d3b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf" podUID="caccf6b1-b7c1-437e-a634-84d62cf8d3b4"
Oct 30 01:45:05.247995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.247974    4425 generic.go:356] "Generic (PLEG): container finished" podID="e253e33e-8ac1-4658-aaf1-485bb678f1b5" containerID="06d3f09481afccdd2863a5b5a5d7b03bba010d6313c95623c63f64d4cfccf10d" exitCode=60
Oct 30 01:45:05.248088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.248046    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p" event={"ID":"e253e33e-8ac1-4658-aaf1-485bb678f1b5","Type":"ContainerDied","Data":"06d3f09481afccdd2863a5b5a5d7b03bba010d6313c95623c63f64d4cfccf10d"}
Oct 30 01:45:05.248281 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.248261    4425 scope.go:122] "RemoveContainer" containerID="06d3f09481afccdd2863a5b5a5d7b03bba010d6313c95623c63f64d4cfccf10d"
Oct 30 01:45:05.248513 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:05.248492    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p_odh-ai-gateway-infra(e253e33e-8ac1-4658-aaf1-485bb678f1b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p" podUID="e253e33e-8ac1-4658-aaf1-485bb678f1b5"
Oct 30 01:45:05.250239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.250217    4425 generic.go:356] "Generic (PLEG): container finished" podID="e77b1d4c-dc08-416b-ad79-6e777474940c" containerID="113a15f876239dbc47cf75ebd796c82e22d35650bc0ee5db282a58197759a80a" exitCode=60
Oct 30 01:45:05.250331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.250310    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7" event={"ID":"e77b1d4c-dc08-416b-ad79-6e777474940c","Type":"ContainerDied","Data":"113a15f876239dbc47cf75ebd796c82e22d35650bc0ee5db282a58197759a80a"}
Oct 30 01:45:05.250488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.250470    4425 scope.go:122] "RemoveContainer" containerID="113a15f876239dbc47cf75ebd796c82e22d35650bc0ee5db282a58197759a80a"
Oct 30 01:45:05.250718 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:05.250696    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7_odh-ai-gateway-infra(e77b1d4c-dc08-416b-ad79-6e777474940c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7" podUID="e77b1d4c-dc08-416b-ad79-6e777474940c"
Oct 30 01:45:05.252336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.252304    4425 generic.go:356] "Generic (PLEG): container finished" podID="9a82dd87-6039-44da-b690-297f5449c337" containerID="2d3371a41423945d879100cc9269f5cb6617d8200b5f38dd7481a56dddba988b" exitCode=60
Oct 30 01:45:05.252441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.252347    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9" event={"ID":"9a82dd87-6039-44da-b690-297f5449c337","Type":"ContainerDied","Data":"2d3371a41423945d879100cc9269f5cb6617d8200b5f38dd7481a56dddba988b"}
Oct 30 01:45:05.252595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.252579    4425 scope.go:122] "RemoveContainer" containerID="2d3371a41423945d879100cc9269f5cb6617d8200b5f38dd7481a56dddba988b"
Oct 30 01:45:05.252798 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:05.252778    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9_odh-ai-gateway-infra(9a82dd87-6039-44da-b690-297f5449c337)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9" podUID="9a82dd87-6039-44da-b690-297f5449c337"
Oct 30 01:45:05.256426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.256403    4425 scope.go:122] "RemoveContainer" containerID="aa80fd748767b2abd0c63fedaed4e80d88b8b9739fa75c0b445cac5c791c1c29"
Oct 30 01:45:05.272794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.272773    4425 scope.go:122] "RemoveContainer" containerID="cc5ee4ad3b24f70ca43915e7c76f154d7cbf8ebdf7264d1c94052953a8709503"
Oct 30 01:45:05.287345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.287321    4425 scope.go:122] "RemoveContainer" containerID="c9c6a985c8664cf9fd8fa69531127bce2b1574c1da5e039534f03d62d086d004"
Oct 30 01:45:05.300846 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.300826    4425 scope.go:122] "RemoveContainer" containerID="c1b58d856ac3b29bd3a63e846d882db045b46e839703e9d9627c47408ab96ae4"
Oct 30 01:45:05.314550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.314531    4425 scope.go:122] "RemoveContainer" containerID="533667ebcace24d81929cc921fae13668518d73e459461e199d747958cb71f92"
Oct 30 01:45:05.328057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.328036    4425 scope.go:122] "RemoveContainer" containerID="79b3ccf610de7281e9ae75833fb6b4ef5025ffff1d5cd154a613641e27f89dd4"
Oct 30 01:45:05.342279 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.342247    4425 scope.go:122] "RemoveContainer" containerID="c3ea30508e025481b904db1b726655deda5c1a00668222fd51d2dc17d335fd9c"
Oct 30 01:45:05.360384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.360365    4425 scope.go:122] "RemoveContainer" containerID="824d7a16e1cb753da6dca71c561275804cc2fcebabb5ec5f4539eb419fab750c"
Oct 30 01:45:05.374237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.374218    4425 scope.go:122] "RemoveContainer" containerID="f715b06ae853cb2029c87aeca46d9ffc8b78a12792d2827e895aa5ad04b1f5f4"
Oct 30 01:45:05.387234 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.387219    4425 scope.go:122] "RemoveContainer" containerID="da5845cad102e2663183054f5a2c146b8ce8d67f25c83f7203e98f601e09700d"
Oct 30 01:45:05.401184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:05.401130    4425 scope.go:122] "RemoveContainer" containerID="3fa1cc8fa4d4d315aecfa49f3fb34b67359686de69de68e18602b694cc984bd8"
Oct 30 01:45:06.263201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:06.263166    4425 scope.go:122] "RemoveContainer" containerID="113a15f876239dbc47cf75ebd796c82e22d35650bc0ee5db282a58197759a80a"
Oct 30 01:45:06.263621 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:06.263504    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7_odh-ai-gateway-infra(e77b1d4c-dc08-416b-ad79-6e777474940c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7" podUID="e77b1d4c-dc08-416b-ad79-6e777474940c"
Oct 30 01:45:06.264783 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:06.264760    4425 scope.go:122] "RemoveContainer" containerID="2d3371a41423945d879100cc9269f5cb6617d8200b5f38dd7481a56dddba988b"
Oct 30 01:45:06.265088 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:06.265064    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9_odh-ai-gateway-infra(9a82dd87-6039-44da-b690-297f5449c337)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9" podUID="9a82dd87-6039-44da-b690-297f5449c337"
Oct 30 01:45:06.266367 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:06.266351    4425 scope.go:122] "RemoveContainer" containerID="894605233a89a544380aa7b282437375745b111a887a86c6f04114b057a0fb03"
Oct 30 01:45:06.266614 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:06.266594    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv_odh-ai-gateway-infra(ee57e527-71ff-4a09-8db0-9840a6467fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv" podUID="ee57e527-71ff-4a09-8db0-9840a6467fce"
Oct 30 01:45:06.267952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:06.267935    4425 scope.go:122] "RemoveContainer" containerID="6ed0e2bb1812dda3a8bd9ab01a8862a734143da0786c2956d486148103039138"
Oct 30 01:45:06.268188 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:06.268165    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm_odh-ai-gateway-infra(a2a87024-9da9-4842-afb2-063b447bd15a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm" podUID="a2a87024-9da9-4842-afb2-063b447bd15a"
Oct 30 01:45:06.269516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:06.269494    4425 scope.go:122] "RemoveContainer" containerID="083313e8ae95fc1061277759b63caf4d78509885425cd6e4b3e1882d60ec2e88"
Oct 30 01:45:06.269748 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:06.269728    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj_odh-ai-gateway-infra(ebecef45-aca8-486c-af0c-33bc1380da5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj" podUID="ebecef45-aca8-486c-af0c-33bc1380da5d"
Oct 30 01:45:06.270990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:06.270973    4425 scope.go:122] "RemoveContainer" containerID="fddbc4fe922f699f598f0bab7e53463e80ce373747b0281c8a79b86affcef87a"
Oct 30 01:45:06.271223 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:06.271204    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd_odh-ai-gateway-infra(357853d9-748f-44af-baa0-523e64187284)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd" podUID="357853d9-748f-44af-baa0-523e64187284"
Oct 30 01:45:06.272523 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:06.272505    4425 scope.go:122] "RemoveContainer" containerID="2412ba6025f4f27a3515e1b2652864fd9e9442f36c209c168f266b313e10e896"
Oct 30 01:45:06.272760 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:06.272740    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2_odh-ai-gateway-infra(d44cf871-7767-4ad3-a9b4-e70a58a52ec9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2" podUID="d44cf871-7767-4ad3-a9b4-e70a58a52ec9"
Oct 30 01:45:06.274011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:06.273992    4425 scope.go:122] "RemoveContainer" containerID="d8db9937bdd5b1ef3dcd5b27384d157f831f9aeecb9ee6bab4d9c3183982bf93"
Oct 30 01:45:06.274235 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:06.274213    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k_odh-ai-gateway-infra(8cd1b902-d5db-4613-90ea-30a92c1bf2b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k" podUID="8cd1b902-d5db-4613-90ea-30a92c1bf2b0"
Oct 30 01:45:06.275526 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:06.275507    4425 scope.go:122] "RemoveContainer" containerID="a86342ec83950b5cb5ab68bf86a1fd9e0e90ce104d91c1865ddce4ce36fdda82"
Oct 30 01:45:06.275728 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:06.275705    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d_odh-ai-gateway-infra(6c1bab87-9a09-48f6-9302-43b0254810e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d" podUID="6c1bab87-9a09-48f6-9302-43b0254810e5"
Oct 30 01:45:06.277097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:06.277077    4425 scope.go:122] "RemoveContainer" containerID="b940451841c081d35304a5f09bd597c9bc5e9fe81fb4a4bb993bf2dd0a46efb8"
Oct 30 01:45:06.277299 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:06.277276    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7_odh-ai-gateway-infra(3cf3962c-f594-478e-89dc-a0ab9671bb9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7" podUID="3cf3962c-f594-478e-89dc-a0ab9671bb9b"
Oct 30 01:45:06.278701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:06.278684    4425 scope.go:122] "RemoveContainer" containerID="eb2ce24a9782de2cfde1f323c69b7d0ef41cd52dac8e4e4de410da3584f5d00d"
Oct 30 01:45:06.278904 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:06.278879    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2_odh-ai-gateway-infra(af779dae-521f-4f8a-9ad2-e7fe3948dcbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2" podUID="af779dae-521f-4f8a-9ad2-e7fe3948dcbc"
Oct 30 01:45:06.280315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:06.280294    4425 scope.go:122] "RemoveContainer" containerID="b041af6c4bb1711c0a35a75ca8951e42cea483e7c3cf0fbbea5955e48b75b894"
Oct 30 01:45:06.280512 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:06.280492    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf_odh-ai-gateway-infra(caccf6b1-b7c1-437e-a634-84d62cf8d3b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf" podUID="caccf6b1-b7c1-437e-a634-84d62cf8d3b4"
Oct 30 01:45:06.281880 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:06.281864    4425 scope.go:122] "RemoveContainer" containerID="06d3f09481afccdd2863a5b5a5d7b03bba010d6313c95623c63f64d4cfccf10d"
Oct 30 01:45:06.282065 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:06.282047    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p_odh-ai-gateway-infra(e253e33e-8ac1-4658-aaf1-485bb678f1b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p" podUID="e253e33e-8ac1-4658-aaf1-485bb678f1b5"
Oct 30 01:45:07.556033 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:07.555988    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:45:09.080232 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:09.080190    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9" podUID="2efb0b19-a05d-46bd-bf1d-622e6410cafe" containerName="payload-processing" containerID="cri-o://95e685605b3853240b34d92e8629923d777c4ca5f1f344e92b022d2217e79309" gracePeriod=25
Oct 30 01:45:09.315260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:09.315221    4425 generic.go:356] "Generic (PLEG): container finished" podID="2efb0b19-a05d-46bd-bf1d-622e6410cafe" containerID="95e685605b3853240b34d92e8629923d777c4ca5f1f344e92b022d2217e79309" exitCode=0
Oct 30 01:45:09.315431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:09.315272    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9" event={"ID":"2efb0b19-a05d-46bd-bf1d-622e6410cafe","Type":"ContainerDied","Data":"95e685605b3853240b34d92e8629923d777c4ca5f1f344e92b022d2217e79309"}
Oct 30 01:45:09.391820 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:09.391794    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"
Oct 30 01:45:09.530612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:09.530582    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2efb0b19-a05d-46bd-bf1d-622e6410cafe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2efb0b19-a05d-46bd-bf1d-622e6410cafe-plugins-config-volume\") pod \"2efb0b19-a05d-46bd-bf1d-622e6410cafe\" (UID: \"2efb0b19-a05d-46bd-bf1d-622e6410cafe\") "
Oct 30 01:45:09.530835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:09.530822    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2efb0b19-a05d-46bd-bf1d-622e6410cafe-kube-api-access-zq97j\" (UniqueName: \"kubernetes.io/projected/2efb0b19-a05d-46bd-bf1d-622e6410cafe-kube-api-access-zq97j\") pod \"2efb0b19-a05d-46bd-bf1d-622e6410cafe\" (UID: \"2efb0b19-a05d-46bd-bf1d-622e6410cafe\") "
Oct 30 01:45:09.531015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:09.530991    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2efb0b19-a05d-46bd-bf1d-622e6410cafe-plugins-config-volume" pod "2efb0b19-a05d-46bd-bf1d-622e6410cafe" (UID: "2efb0b19-a05d-46bd-bf1d-622e6410cafe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:09.531275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:09.531259    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2efb0b19-a05d-46bd-bf1d-622e6410cafe-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:09.532919 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:09.532898    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2efb0b19-a05d-46bd-bf1d-622e6410cafe-kube-api-access-zq97j" pod "2efb0b19-a05d-46bd-bf1d-622e6410cafe" (UID: "2efb0b19-a05d-46bd-bf1d-622e6410cafe"). InnerVolumeSpecName "kube-api-access-zq97j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:09.632929 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:09.632870    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq97j\" (UniqueName: \"kubernetes.io/projected/2efb0b19-a05d-46bd-bf1d-622e6410cafe-kube-api-access-zq97j\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:10.325678 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:10.325645    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"
Oct 30 01:45:10.326142 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:10.325683    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9" event={"ID":"2efb0b19-a05d-46bd-bf1d-622e6410cafe","Type":"ContainerDied","Data":"eb41ca5afb5a6f643f84fa0b9478f897b2305c19696fc87ebe682d508a55abe8"}
Oct 30 01:45:10.326142 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:10.325760    4425 scope.go:122] "RemoveContainer" containerID="95e685605b3853240b34d92e8629923d777c4ca5f1f344e92b022d2217e79309"
Oct 30 01:45:10.373765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:10.373740    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"]
Oct 30 01:45:10.383254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:10.383230    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5457f2-55dbbb876-77ng9"]
Oct 30 01:45:11.424961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:11.424929    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2efb0b19-a05d-46bd-bf1d-622e6410cafe" path="/var/lib/kubelet/pods/2efb0b19-a05d-46bd-bf1d-622e6410cafe/volumes"
Oct 30 01:45:12.248573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:12.248540    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:45:15.415999 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:15.415916    4425 scope.go:122] "RemoveContainer" containerID="06d3f09481afccdd2863a5b5a5d7b03bba010d6313c95623c63f64d4cfccf10d"
Oct 30 01:45:15.416465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:15.416083    4425 scope.go:122] "RemoveContainer" containerID="4e9b0ad353abaf08a8b651dada3d9321cb9eca5e74972a0a4e5e7430e03e97c8"
Oct 30 01:45:15.416465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:15.416285    4425 scope.go:122] "RemoveContainer" containerID="113a15f876239dbc47cf75ebd796c82e22d35650bc0ee5db282a58197759a80a"
Oct 30 01:45:15.417045 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:15.416629    4425 scope.go:122] "RemoveContainer" containerID="eb2ce24a9782de2cfde1f323c69b7d0ef41cd52dac8e4e4de410da3584f5d00d"
Oct 30 01:45:16.382055 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:16.381136    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p" event={"ID":"e253e33e-8ac1-4658-aaf1-485bb678f1b5","Type":"ContainerStarted","Data":"95913f39df5753f8b63cef364ca37a863b69cf437f648784530fcfad8b5af172"}
Oct 30 01:45:16.383309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:16.383276    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7" event={"ID":"e77b1d4c-dc08-416b-ad79-6e777474940c","Type":"ContainerStarted","Data":"68a4fd6e11ccf422089db8cb5008c20a18ef2b0c212ac284218d0e145e34a02e"}
Oct 30 01:45:16.385211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:16.385178    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs" event={"ID":"336e3514-5287-4632-84f7-2673bc71a606","Type":"ContainerStarted","Data":"47759e0c22ef1b9df1bee2ebac4a9637a610d63ed508952e3a401fc7a65bf562"}
Oct 30 01:45:16.387060 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:16.387039    4425 generic.go:356] "Generic (PLEG): container finished" podID="af779dae-521f-4f8a-9ad2-e7fe3948dcbc" containerID="f0be75bf2ecc550b6d0e01f2aaaf66cd8fd5b3fe8224fb2aca782b55d1791f89" exitCode=60
Oct 30 01:45:16.387141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:16.387095    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2" event={"ID":"af779dae-521f-4f8a-9ad2-e7fe3948dcbc","Type":"ContainerDied","Data":"f0be75bf2ecc550b6d0e01f2aaaf66cd8fd5b3fe8224fb2aca782b55d1791f89"}
Oct 30 01:45:16.387141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:16.387134    4425 scope.go:122] "RemoveContainer" containerID="eb2ce24a9782de2cfde1f323c69b7d0ef41cd52dac8e4e4de410da3584f5d00d"
Oct 30 01:45:16.387446 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:16.387428    4425 scope.go:122] "RemoveContainer" containerID="f0be75bf2ecc550b6d0e01f2aaaf66cd8fd5b3fe8224fb2aca782b55d1791f89"
Oct 30 01:45:16.387724 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:16.387702    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2_odh-ai-gateway-infra(af779dae-521f-4f8a-9ad2-e7fe3948dcbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2" podUID="af779dae-521f-4f8a-9ad2-e7fe3948dcbc"
Oct 30 01:45:16.415340 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:16.415321    4425 scope.go:122] "RemoveContainer" containerID="2412ba6025f4f27a3515e1b2652864fd9e9442f36c209c168f266b313e10e896"
Oct 30 01:45:17.401887 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.401857    4425 generic.go:356] "Generic (PLEG): container finished" podID="e253e33e-8ac1-4658-aaf1-485bb678f1b5" containerID="95913f39df5753f8b63cef364ca37a863b69cf437f648784530fcfad8b5af172" exitCode=60
Oct 30 01:45:17.402291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.401888    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p" event={"ID":"e253e33e-8ac1-4658-aaf1-485bb678f1b5","Type":"ContainerDied","Data":"95913f39df5753f8b63cef364ca37a863b69cf437f648784530fcfad8b5af172"}
Oct 30 01:45:17.402291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.401966    4425 scope.go:122] "RemoveContainer" containerID="06d3f09481afccdd2863a5b5a5d7b03bba010d6313c95623c63f64d4cfccf10d"
Oct 30 01:45:17.402390 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.402346    4425 scope.go:122] "RemoveContainer" containerID="95913f39df5753f8b63cef364ca37a863b69cf437f648784530fcfad8b5af172"
Oct 30 01:45:17.402755 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:17.402725    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p_odh-ai-gateway-infra(e253e33e-8ac1-4658-aaf1-485bb678f1b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p" podUID="e253e33e-8ac1-4658-aaf1-485bb678f1b5"
Oct 30 01:45:17.404428 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.404402    4425 generic.go:356] "Generic (PLEG): container finished" podID="e77b1d4c-dc08-416b-ad79-6e777474940c" containerID="68a4fd6e11ccf422089db8cb5008c20a18ef2b0c212ac284218d0e145e34a02e" exitCode=60
Oct 30 01:45:17.404526 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.404477    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7" event={"ID":"e77b1d4c-dc08-416b-ad79-6e777474940c","Type":"ContainerDied","Data":"68a4fd6e11ccf422089db8cb5008c20a18ef2b0c212ac284218d0e145e34a02e"}
Oct 30 01:45:17.404930 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.404908    4425 scope.go:122] "RemoveContainer" containerID="68a4fd6e11ccf422089db8cb5008c20a18ef2b0c212ac284218d0e145e34a02e"
Oct 30 01:45:17.405350 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:17.405324    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7_odh-ai-gateway-infra(e77b1d4c-dc08-416b-ad79-6e777474940c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7" podUID="e77b1d4c-dc08-416b-ad79-6e777474940c"
Oct 30 01:45:17.406715 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.406694    4425 generic.go:356] "Generic (PLEG): container finished" podID="336e3514-5287-4632-84f7-2673bc71a606" containerID="47759e0c22ef1b9df1bee2ebac4a9637a610d63ed508952e3a401fc7a65bf562" exitCode=60
Oct 30 01:45:17.406795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.406775    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs" event={"ID":"336e3514-5287-4632-84f7-2673bc71a606","Type":"ContainerDied","Data":"47759e0c22ef1b9df1bee2ebac4a9637a610d63ed508952e3a401fc7a65bf562"}
Oct 30 01:45:17.407226 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.407206    4425 scope.go:122] "RemoveContainer" containerID="47759e0c22ef1b9df1bee2ebac4a9637a610d63ed508952e3a401fc7a65bf562"
Oct 30 01:45:17.407580 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:17.407559    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs_odh-ai-gateway-infra(336e3514-5287-4632-84f7-2673bc71a606)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs" podUID="336e3514-5287-4632-84f7-2673bc71a606"
Oct 30 01:45:17.409110 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.409087    4425 generic.go:356] "Generic (PLEG): container finished" podID="d44cf871-7767-4ad3-a9b4-e70a58a52ec9" containerID="f91f1ac6fbaa5d9655cdffe24ff80470b41ccdd66f7389a7cd32b6915927985d" exitCode=60
Oct 30 01:45:17.409227 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.409145    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2" event={"ID":"d44cf871-7767-4ad3-a9b4-e70a58a52ec9","Type":"ContainerDied","Data":"f91f1ac6fbaa5d9655cdffe24ff80470b41ccdd66f7389a7cd32b6915927985d"}
Oct 30 01:45:17.409555 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.409539    4425 scope.go:122] "RemoveContainer" containerID="f91f1ac6fbaa5d9655cdffe24ff80470b41ccdd66f7389a7cd32b6915927985d"
Oct 30 01:45:17.409882 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:17.409859    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2_odh-ai-gateway-infra(d44cf871-7767-4ad3-a9b4-e70a58a52ec9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2" podUID="d44cf871-7767-4ad3-a9b4-e70a58a52ec9"
Oct 30 01:45:17.415065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.415045    4425 scope.go:122] "RemoveContainer" containerID="b041af6c4bb1711c0a35a75ca8951e42cea483e7c3cf0fbbea5955e48b75b894"
Oct 30 01:45:17.415219 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.415198    4425 scope.go:122] "RemoveContainer" containerID="6ed0e2bb1812dda3a8bd9ab01a8862a734143da0786c2956d486148103039138"
Oct 30 01:45:17.415813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.415796    4425 scope.go:122] "RemoveContainer" containerID="d8db9937bdd5b1ef3dcd5b27384d157f831f9aeecb9ee6bab4d9c3183982bf93"
Oct 30 01:45:17.416101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.416075    4425 scope.go:122] "RemoveContainer" containerID="b940451841c081d35304a5f09bd597c9bc5e9fe81fb4a4bb993bf2dd0a46efb8"
Oct 30 01:45:17.416261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.416225    4425 scope.go:122] "RemoveContainer" containerID="fddbc4fe922f699f598f0bab7e53463e80ce373747b0281c8a79b86affcef87a"
Oct 30 01:45:17.419671 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.419233    4425 scope.go:122] "RemoveContainer" containerID="113a15f876239dbc47cf75ebd796c82e22d35650bc0ee5db282a58197759a80a"
Oct 30 01:45:17.448796 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.448772    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p"]
Oct 30 01:45:17.467365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.467334    4425 scope.go:122] "RemoveContainer" containerID="4e9b0ad353abaf08a8b651dada3d9321cb9eca5e74972a0a4e5e7430e03e97c8"
Oct 30 01:45:17.474509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.474480    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs"]
Oct 30 01:45:17.517284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.517256    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7"]
Oct 30 01:45:17.539297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.539147    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2"]
Oct 30 01:45:17.583303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.582956    4425 scope.go:122] "RemoveContainer" containerID="2412ba6025f4f27a3515e1b2652864fd9e9442f36c209c168f266b313e10e896"
Oct 30 01:45:17.795168 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.795137    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2"
Oct 30 01:45:17.949380 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.949346    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af779dae-521f-4f8a-9ad2-e7fe3948dcbc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af779dae-521f-4f8a-9ad2-e7fe3948dcbc-maas-api-service-ca\") pod \"af779dae-521f-4f8a-9ad2-e7fe3948dcbc\" (UID: \"af779dae-521f-4f8a-9ad2-e7fe3948dcbc\") "
Oct 30 01:45:17.949586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.949545    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af779dae-521f-4f8a-9ad2-e7fe3948dcbc-kube-api-access-kbhzd\" (UniqueName: \"kubernetes.io/projected/af779dae-521f-4f8a-9ad2-e7fe3948dcbc-kube-api-access-kbhzd\") pod \"af779dae-521f-4f8a-9ad2-e7fe3948dcbc\" (UID: \"af779dae-521f-4f8a-9ad2-e7fe3948dcbc\") "
Oct 30 01:45:17.949788 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.949761    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af779dae-521f-4f8a-9ad2-e7fe3948dcbc-maas-api-service-ca" pod "af779dae-521f-4f8a-9ad2-e7fe3948dcbc" (UID: "af779dae-521f-4f8a-9ad2-e7fe3948dcbc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:17.950286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.950130    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af779dae-521f-4f8a-9ad2-e7fe3948dcbc-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:17.952704 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:17.952678    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af779dae-521f-4f8a-9ad2-e7fe3948dcbc-kube-api-access-kbhzd" pod "af779dae-521f-4f8a-9ad2-e7fe3948dcbc" (UID: "af779dae-521f-4f8a-9ad2-e7fe3948dcbc"). InnerVolumeSpecName "kube-api-access-kbhzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:18.052706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:18.052174    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbhzd\" (UniqueName: \"kubernetes.io/projected/af779dae-521f-4f8a-9ad2-e7fe3948dcbc-kube-api-access-kbhzd\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:18.415782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:18.415693    4425 scope.go:122] "RemoveContainer" containerID="894605233a89a544380aa7b282437375745b111a887a86c6f04114b057a0fb03"
Oct 30 01:45:18.416256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:18.415875    4425 scope.go:122] "RemoveContainer" containerID="a86342ec83950b5cb5ab68bf86a1fd9e0e90ce104d91c1865ddce4ce36fdda82"
Oct 30 01:45:18.416256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:18.415959    4425 scope.go:122] "RemoveContainer" containerID="2d3371a41423945d879100cc9269f5cb6617d8200b5f38dd7481a56dddba988b"
Oct 30 01:45:18.423623 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:18.423595    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm" event={"ID":"a2a87024-9da9-4842-afb2-063b447bd15a","Type":"ContainerStarted","Data":"917e61ce493809421d51d7d673081a5a3c675bf5c91f81ec6dbb8bec92bdacfc"}
Oct 30 01:45:18.428359 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:18.428335    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd" event={"ID":"357853d9-748f-44af-baa0-523e64187284","Type":"ContainerStarted","Data":"97dff42541264adfa7919e701242ea349a5bb9865c2786ffd545ec4b5088b962"}
Oct 30 01:45:18.433757 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:18.433279    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k" event={"ID":"8cd1b902-d5db-4613-90ea-30a92c1bf2b0","Type":"ContainerStarted","Data":"2fe38aa77ad44c39ceba25a348b06220be0fc2af7d295fc3462d565948ac2071"}
Oct 30 01:45:18.441212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:18.436319    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7" event={"ID":"3cf3962c-f594-478e-89dc-a0ab9671bb9b","Type":"ContainerStarted","Data":"ac1b37cef21831cf3cf1e681def6aa9e16cda7feebcedca75157d41da8fc556e"}
Oct 30 01:45:18.441212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:18.438704    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2"
Oct 30 01:45:18.441212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:18.438698    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2" event={"ID":"af779dae-521f-4f8a-9ad2-e7fe3948dcbc","Type":"ContainerDied","Data":"400225d45a029ec129127cff30b3cf3804ba37e1e0c0c87f6c480a724e515f34"}
Oct 30 01:45:18.441212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:18.438810    4425 scope.go:122] "RemoveContainer" containerID="f0be75bf2ecc550b6d0e01f2aaaf66cd8fd5b3fe8224fb2aca782b55d1791f89"
Oct 30 01:45:18.441606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:18.441275    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf" event={"ID":"caccf6b1-b7c1-437e-a634-84d62cf8d3b4","Type":"ContainerStarted","Data":"aa59d1d5f2d0c8142fe7c6d93e88e615b35e2775471c8424f49f5b1dd1a4d630"}
Oct 30 01:45:18.594936 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:18.593755    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2"]
Oct 30 01:45:18.619937 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:18.619905    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2"]
Oct 30 01:45:18.625830 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:18.625794    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c7bba9-29888745-m77x2"]
Oct 30 01:45:18.766542 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:18.766496    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd" podStartSLOduration=18.766483064 podStartE2EDuration="18.766483064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:18.761633972 +0000 UTC m=+3827.511289281" watchObservedRunningTime="2026-10-30 01:45:18.766483064 +0000 UTC m=+3827.516138412"
Oct 30 01:45:19.148082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.148058    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs"
Oct 30 01:45:19.150168 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.150136    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p"
Oct 30 01:45:19.155502 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.155481    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7"
Oct 30 01:45:19.158351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.158334    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2"
Oct 30 01:45:19.179930 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.178651    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e253e33e-8ac1-4658-aaf1-485bb678f1b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e253e33e-8ac1-4658-aaf1-485bb678f1b5-maas-api-service-ca\") pod \"e253e33e-8ac1-4658-aaf1-485bb678f1b5\" (UID: \"e253e33e-8ac1-4658-aaf1-485bb678f1b5\") "
Oct 30 01:45:19.179930 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.178768    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336e3514-5287-4632-84f7-2673bc71a606-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/336e3514-5287-4632-84f7-2673bc71a606-maas-api-service-ca\") pod \"336e3514-5287-4632-84f7-2673bc71a606\" (UID: \"336e3514-5287-4632-84f7-2673bc71a606\") "
Oct 30 01:45:19.179930 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.178804    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d44cf871-7767-4ad3-a9b4-e70a58a52ec9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44cf871-7767-4ad3-a9b4-e70a58a52ec9-maas-api-service-ca\") pod \"d44cf871-7767-4ad3-a9b4-e70a58a52ec9\" (UID: \"d44cf871-7767-4ad3-a9b4-e70a58a52ec9\") "
Oct 30 01:45:19.179930 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.178836    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e253e33e-8ac1-4658-aaf1-485bb678f1b5-kube-api-access-9d7ps\" (UniqueName: \"kubernetes.io/projected/e253e33e-8ac1-4658-aaf1-485bb678f1b5-kube-api-access-9d7ps\") pod \"e253e33e-8ac1-4658-aaf1-485bb678f1b5\" (UID: \"e253e33e-8ac1-4658-aaf1-485bb678f1b5\") "
Oct 30 01:45:19.179930 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.178903    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77b1d4c-dc08-416b-ad79-6e777474940c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b1d4c-dc08-416b-ad79-6e777474940c-maas-api-service-ca\") pod \"e77b1d4c-dc08-416b-ad79-6e777474940c\" (UID: \"e77b1d4c-dc08-416b-ad79-6e777474940c\") "
Oct 30 01:45:19.179930 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.178943    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336e3514-5287-4632-84f7-2673bc71a606-kube-api-access-g954k\" (UniqueName: \"kubernetes.io/projected/336e3514-5287-4632-84f7-2673bc71a606-kube-api-access-g954k\") pod \"336e3514-5287-4632-84f7-2673bc71a606\" (UID: \"336e3514-5287-4632-84f7-2673bc71a606\") "
Oct 30 01:45:19.179930 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.178975    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77b1d4c-dc08-416b-ad79-6e777474940c-kube-api-access-n2fk5\" (UniqueName: \"kubernetes.io/projected/e77b1d4c-dc08-416b-ad79-6e777474940c-kube-api-access-n2fk5\") pod \"e77b1d4c-dc08-416b-ad79-6e777474940c\" (UID: \"e77b1d4c-dc08-416b-ad79-6e777474940c\") "
Oct 30 01:45:19.179930 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.179176    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d44cf871-7767-4ad3-a9b4-e70a58a52ec9-kube-api-access-77mgx\" (UniqueName: \"kubernetes.io/projected/d44cf871-7767-4ad3-a9b4-e70a58a52ec9-kube-api-access-77mgx\") pod \"d44cf871-7767-4ad3-a9b4-e70a58a52ec9\" (UID: \"d44cf871-7767-4ad3-a9b4-e70a58a52ec9\") "
Oct 30 01:45:19.180486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.180257    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e253e33e-8ac1-4658-aaf1-485bb678f1b5-maas-api-service-ca" pod "e253e33e-8ac1-4658-aaf1-485bb678f1b5" (UID: "e253e33e-8ac1-4658-aaf1-485bb678f1b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:19.180858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.180656    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336e3514-5287-4632-84f7-2673bc71a606-maas-api-service-ca" pod "336e3514-5287-4632-84f7-2673bc71a606" (UID: "336e3514-5287-4632-84f7-2673bc71a606"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:19.180858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.180787    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44cf871-7767-4ad3-a9b4-e70a58a52ec9-maas-api-service-ca" pod "d44cf871-7767-4ad3-a9b4-e70a58a52ec9" (UID: "d44cf871-7767-4ad3-a9b4-e70a58a52ec9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:19.181404 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.181382    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77b1d4c-dc08-416b-ad79-6e777474940c-maas-api-service-ca" pod "e77b1d4c-dc08-416b-ad79-6e777474940c" (UID: "e77b1d4c-dc08-416b-ad79-6e777474940c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:19.181851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.181813    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336e3514-5287-4632-84f7-2673bc71a606-kube-api-access-g954k" pod "336e3514-5287-4632-84f7-2673bc71a606" (UID: "336e3514-5287-4632-84f7-2673bc71a606"). InnerVolumeSpecName "kube-api-access-g954k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:19.182729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.182705    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77b1d4c-dc08-416b-ad79-6e777474940c-kube-api-access-n2fk5" pod "e77b1d4c-dc08-416b-ad79-6e777474940c" (UID: "e77b1d4c-dc08-416b-ad79-6e777474940c"). InnerVolumeSpecName "kube-api-access-n2fk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:19.251710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.251680    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44cf871-7767-4ad3-a9b4-e70a58a52ec9-kube-api-access-77mgx" pod "d44cf871-7767-4ad3-a9b4-e70a58a52ec9" (UID: "d44cf871-7767-4ad3-a9b4-e70a58a52ec9"). InnerVolumeSpecName "kube-api-access-77mgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:19.254753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.254725    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e253e33e-8ac1-4658-aaf1-485bb678f1b5-kube-api-access-9d7ps" pod "e253e33e-8ac1-4658-aaf1-485bb678f1b5" (UID: "e253e33e-8ac1-4658-aaf1-485bb678f1b5"). InnerVolumeSpecName "kube-api-access-9d7ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:19.281104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.280968    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77mgx\" (UniqueName: \"kubernetes.io/projected/d44cf871-7767-4ad3-a9b4-e70a58a52ec9-kube-api-access-77mgx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:19.281104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.281000    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e253e33e-8ac1-4658-aaf1-485bb678f1b5-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:19.281104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.281017    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/336e3514-5287-4632-84f7-2673bc71a606-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:19.281104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.281030    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44cf871-7767-4ad3-a9b4-e70a58a52ec9-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:19.281104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.281044    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d7ps\" (UniqueName: \"kubernetes.io/projected/e253e33e-8ac1-4658-aaf1-485bb678f1b5-kube-api-access-9d7ps\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:19.281104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.281057    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b1d4c-dc08-416b-ad79-6e777474940c-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:19.281104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.281071    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g954k\" (UniqueName: \"kubernetes.io/projected/336e3514-5287-4632-84f7-2673bc71a606-kube-api-access-g954k\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:19.281104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.281084    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2fk5\" (UniqueName: \"kubernetes.io/projected/e77b1d4c-dc08-416b-ad79-6e777474940c-kube-api-access-n2fk5\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:19.416109 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.416029    4425 scope.go:122] "RemoveContainer" containerID="083313e8ae95fc1061277759b63caf4d78509885425cd6e4b3e1882d60ec2e88"
Oct 30 01:45:19.427335 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.427305    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af779dae-521f-4f8a-9ad2-e7fe3948dcbc" path="/var/lib/kubelet/pods/af779dae-521f-4f8a-9ad2-e7fe3948dcbc/volumes"
Oct 30 01:45:19.457607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.457581    4425 generic.go:356] "Generic (PLEG): container finished" podID="3cf3962c-f594-478e-89dc-a0ab9671bb9b" containerID="ac1b37cef21831cf3cf1e681def6aa9e16cda7feebcedca75157d41da8fc556e" exitCode=60
Oct 30 01:45:19.457716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.457649    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7" event={"ID":"3cf3962c-f594-478e-89dc-a0ab9671bb9b","Type":"ContainerDied","Data":"ac1b37cef21831cf3cf1e681def6aa9e16cda7feebcedca75157d41da8fc556e"}
Oct 30 01:45:19.457716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.457707    4425 scope.go:122] "RemoveContainer" containerID="b940451841c081d35304a5f09bd597c9bc5e9fe81fb4a4bb993bf2dd0a46efb8"
Oct 30 01:45:19.458240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.458221    4425 scope.go:122] "RemoveContainer" containerID="ac1b37cef21831cf3cf1e681def6aa9e16cda7feebcedca75157d41da8fc556e"
Oct 30 01:45:19.458510 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:19.458490    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7_odh-ai-gateway-infra(3cf3962c-f594-478e-89dc-a0ab9671bb9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7" podUID="3cf3962c-f594-478e-89dc-a0ab9671bb9b"
Oct 30 01:45:19.461364 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.461344    4425 generic.go:356] "Generic (PLEG): container finished" podID="caccf6b1-b7c1-437e-a634-84d62cf8d3b4" containerID="aa59d1d5f2d0c8142fe7c6d93e88e615b35e2775471c8424f49f5b1dd1a4d630" exitCode=60
Oct 30 01:45:19.461482 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.461417    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf" event={"ID":"caccf6b1-b7c1-437e-a634-84d62cf8d3b4","Type":"ContainerDied","Data":"aa59d1d5f2d0c8142fe7c6d93e88e615b35e2775471c8424f49f5b1dd1a4d630"}
Oct 30 01:45:19.461834 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.461816    4425 scope.go:122] "RemoveContainer" containerID="aa59d1d5f2d0c8142fe7c6d93e88e615b35e2775471c8424f49f5b1dd1a4d630"
Oct 30 01:45:19.462103 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:19.462082    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf_odh-ai-gateway-infra(caccf6b1-b7c1-437e-a634-84d62cf8d3b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf" podUID="caccf6b1-b7c1-437e-a634-84d62cf8d3b4"
Oct 30 01:45:19.463515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.463499    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p"
Oct 30 01:45:19.463596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.463520    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p" event={"ID":"e253e33e-8ac1-4658-aaf1-485bb678f1b5","Type":"ContainerDied","Data":"802a05980d501b17b8d559dc383ef8eacf099101273b45766fcd49fc05d296cd"}
Oct 30 01:45:19.464987 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.464970    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7"
Oct 30 01:45:19.465087 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.465003    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7" event={"ID":"e77b1d4c-dc08-416b-ad79-6e777474940c","Type":"ContainerDied","Data":"025622ab2bf6ec03484a5d3af21a3672c62886daffb45358fc8822287eed4fc7"}
Oct 30 01:45:19.467563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.467537    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9" event={"ID":"9a82dd87-6039-44da-b690-297f5449c337","Type":"ContainerStarted","Data":"3b922a20408f25d74429e0748eab28a25a4e1db4834a100d6c2d43ec3cc719a7"}
Oct 30 01:45:19.470838 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.470813    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv" event={"ID":"ee57e527-71ff-4a09-8db0-9840a6467fce","Type":"ContainerStarted","Data":"185f46dfd067fd899ada86f50b1b3e3cac84ec0867ec9c167457356ac0e8381d"}
Oct 30 01:45:19.473179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.473139    4425 generic.go:356] "Generic (PLEG): container finished" podID="a2a87024-9da9-4842-afb2-063b447bd15a" containerID="917e61ce493809421d51d7d673081a5a3c675bf5c91f81ec6dbb8bec92bdacfc" exitCode=60
Oct 30 01:45:19.473294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.473198    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm" event={"ID":"a2a87024-9da9-4842-afb2-063b447bd15a","Type":"ContainerDied","Data":"917e61ce493809421d51d7d673081a5a3c675bf5c91f81ec6dbb8bec92bdacfc"}
Oct 30 01:45:19.473583 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.473562    4425 scope.go:122] "RemoveContainer" containerID="917e61ce493809421d51d7d673081a5a3c675bf5c91f81ec6dbb8bec92bdacfc"
Oct 30 01:45:19.473815 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:19.473793    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm_odh-ai-gateway-infra(a2a87024-9da9-4842-afb2-063b447bd15a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm" podUID="a2a87024-9da9-4842-afb2-063b447bd15a"
Oct 30 01:45:19.474527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.474509    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm"]
Oct 30 01:45:19.475252 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.475143    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs"
Oct 30 01:45:19.475252 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.475187    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs" event={"ID":"336e3514-5287-4632-84f7-2673bc71a606","Type":"ContainerDied","Data":"97c07333a34443fff685fdbc45af861ab0f58b0f77d3b48c6f02ddbe4722dc87"}
Oct 30 01:45:19.476368 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.476335    4425 scope.go:122] "RemoveContainer" containerID="b041af6c4bb1711c0a35a75ca8951e42cea483e7c3cf0fbbea5955e48b75b894"
Oct 30 01:45:19.477582 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.477564    4425 generic.go:356] "Generic (PLEG): container finished" podID="357853d9-748f-44af-baa0-523e64187284" containerID="97dff42541264adfa7919e701242ea349a5bb9865c2786ffd545ec4b5088b962" exitCode=60
Oct 30 01:45:19.477670 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.477632    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd" event={"ID":"357853d9-748f-44af-baa0-523e64187284","Type":"ContainerDied","Data":"97dff42541264adfa7919e701242ea349a5bb9865c2786ffd545ec4b5088b962"}
Oct 30 01:45:19.477969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.477925    4425 scope.go:122] "RemoveContainer" containerID="97dff42541264adfa7919e701242ea349a5bb9865c2786ffd545ec4b5088b962"
Oct 30 01:45:19.478200 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:19.478143    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd_odh-ai-gateway-infra(357853d9-748f-44af-baa0-523e64187284)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd" podUID="357853d9-748f-44af-baa0-523e64187284"
Oct 30 01:45:19.479130 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.479106    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2"
Oct 30 01:45:19.479130 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.479112    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2" event={"ID":"d44cf871-7767-4ad3-a9b4-e70a58a52ec9","Type":"ContainerDied","Data":"9153eb76c14b25521a881686865c7c5a59e9bbb6443d16195be681102a825158"}
Oct 30 01:45:19.481151 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.481115    4425 generic.go:356] "Generic (PLEG): container finished" podID="8cd1b902-d5db-4613-90ea-30a92c1bf2b0" containerID="2fe38aa77ad44c39ceba25a348b06220be0fc2af7d295fc3462d565948ac2071" exitCode=60
Oct 30 01:45:19.481296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.481201    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k" event={"ID":"8cd1b902-d5db-4613-90ea-30a92c1bf2b0","Type":"ContainerDied","Data":"2fe38aa77ad44c39ceba25a348b06220be0fc2af7d295fc3462d565948ac2071"}
Oct 30 01:45:19.481705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.481629    4425 scope.go:122] "RemoveContainer" containerID="2fe38aa77ad44c39ceba25a348b06220be0fc2af7d295fc3462d565948ac2071"
Oct 30 01:45:19.481870 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:19.481848    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k_odh-ai-gateway-infra(8cd1b902-d5db-4613-90ea-30a92c1bf2b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k" podUID="8cd1b902-d5db-4613-90ea-30a92c1bf2b0"
Oct 30 01:45:19.483663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.483643    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d" event={"ID":"6c1bab87-9a09-48f6-9302-43b0254810e5","Type":"ContainerStarted","Data":"8178d8f1469dbf49729fe626496d835a2b759a2aff9ce1cf2bcf7451b743d1f2"}
Oct 30 01:45:19.519046 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.519024    4425 scope.go:122] "RemoveContainer" containerID="95913f39df5753f8b63cef364ca37a863b69cf437f648784530fcfad8b5af172"
Oct 30 01:45:19.529985 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.529965    4425 scope.go:122] "RemoveContainer" containerID="68a4fd6e11ccf422089db8cb5008c20a18ef2b0c212ac284218d0e145e34a02e"
Oct 30 01:45:19.530769 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.530746    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p"]
Oct 30 01:45:19.535858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.535834    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a47b9a-29888745-fj48p"]
Oct 30 01:45:19.541419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.541395    4425 scope.go:122] "RemoveContainer" containerID="6ed0e2bb1812dda3a8bd9ab01a8862a734143da0786c2956d486148103039138"
Oct 30 01:45:19.554128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.554101    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7"]
Oct 30 01:45:19.561581 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.561562    4425 scope.go:122] "RemoveContainer" containerID="47759e0c22ef1b9df1bee2ebac4a9637a610d63ed508952e3a401fc7a65bf562"
Oct 30 01:45:19.571936 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.571912    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs"]
Oct 30 01:45:19.579465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.579442    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6aa9aa-29888745-d7zhs"]
Oct 30 01:45:19.590533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.590509    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf"]
Oct 30 01:45:19.590677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.590654    4425 scope.go:122] "RemoveContainer" containerID="fddbc4fe922f699f598f0bab7e53463e80ce373747b0281c8a79b86affcef87a"
Oct 30 01:45:19.604969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.604950    4425 scope.go:122] "RemoveContainer" containerID="f91f1ac6fbaa5d9655cdffe24ff80470b41ccdd66f7389a7cd32b6915927985d"
Oct 30 01:45:19.614973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.614954    4425 scope.go:122] "RemoveContainer" containerID="d8db9937bdd5b1ef3dcd5b27384d157f831f9aeecb9ee6bab4d9c3183982bf93"
Oct 30 01:45:19.645575 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.645549    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7"]
Oct 30 01:45:19.653060 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.653035    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c666c2-29888745-9fbm7"]
Oct 30 01:45:19.667005 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.666945    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k"]
Oct 30 01:45:19.775486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.775457    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd"]
Oct 30 01:45:19.909324 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.909281    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2"]
Oct 30 01:45:19.913871 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:19.913852    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1afb3e-29888745-8qtl2"]
Oct 30 01:45:20.499137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.499103    4425 generic.go:356] "Generic (PLEG): container finished" podID="9a82dd87-6039-44da-b690-297f5449c337" containerID="3b922a20408f25d74429e0748eab28a25a4e1db4834a100d6c2d43ec3cc719a7" exitCode=60
Oct 30 01:45:20.499599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.499183    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9" event={"ID":"9a82dd87-6039-44da-b690-297f5449c337","Type":"ContainerDied","Data":"3b922a20408f25d74429e0748eab28a25a4e1db4834a100d6c2d43ec3cc719a7"}
Oct 30 01:45:20.499599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.499248    4425 scope.go:122] "RemoveContainer" containerID="2d3371a41423945d879100cc9269f5cb6617d8200b5f38dd7481a56dddba988b"
Oct 30 01:45:20.499715 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.499644    4425 scope.go:122] "RemoveContainer" containerID="3b922a20408f25d74429e0748eab28a25a4e1db4834a100d6c2d43ec3cc719a7"
Oct 30 01:45:20.500050 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:20.500019    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9_odh-ai-gateway-infra(9a82dd87-6039-44da-b690-297f5449c337)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9" podUID="9a82dd87-6039-44da-b690-297f5449c337"
Oct 30 01:45:20.502145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.502121    4425 generic.go:356] "Generic (PLEG): container finished" podID="ee57e527-71ff-4a09-8db0-9840a6467fce" containerID="185f46dfd067fd899ada86f50b1b3e3cac84ec0867ec9c167457356ac0e8381d" exitCode=60
Oct 30 01:45:20.502260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.502185    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv" event={"ID":"ee57e527-71ff-4a09-8db0-9840a6467fce","Type":"ContainerDied","Data":"185f46dfd067fd899ada86f50b1b3e3cac84ec0867ec9c167457356ac0e8381d"}
Oct 30 01:45:20.502580 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.502556    4425 scope.go:122] "RemoveContainer" containerID="185f46dfd067fd899ada86f50b1b3e3cac84ec0867ec9c167457356ac0e8381d"
Oct 30 01:45:20.502929 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:20.502903    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv_odh-ai-gateway-infra(ee57e527-71ff-4a09-8db0-9840a6467fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv" podUID="ee57e527-71ff-4a09-8db0-9840a6467fce"
Oct 30 01:45:20.507803 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.507776    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj" event={"ID":"ebecef45-aca8-486c-af0c-33bc1380da5d","Type":"ContainerStarted","Data":"2c13fc928b1638ecb641007af59504394c024d505436a04281b41b8424db17eb"}
Oct 30 01:45:20.517948 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.517929    4425 scope.go:122] "RemoveContainer" containerID="894605233a89a544380aa7b282437375745b111a887a86c6f04114b057a0fb03"
Oct 30 01:45:20.522766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.522742    4425 generic.go:356] "Generic (PLEG): container finished" podID="6c1bab87-9a09-48f6-9302-43b0254810e5" containerID="8178d8f1469dbf49729fe626496d835a2b759a2aff9ce1cf2bcf7451b743d1f2" exitCode=60
Oct 30 01:45:20.522861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.522822    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d" event={"ID":"6c1bab87-9a09-48f6-9302-43b0254810e5","Type":"ContainerDied","Data":"8178d8f1469dbf49729fe626496d835a2b759a2aff9ce1cf2bcf7451b743d1f2"}
Oct 30 01:45:20.523317 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.523165    4425 scope.go:122] "RemoveContainer" containerID="8178d8f1469dbf49729fe626496d835a2b759a2aff9ce1cf2bcf7451b743d1f2"
Oct 30 01:45:20.523576 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:20.523498    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d_odh-ai-gateway-infra(6c1bab87-9a09-48f6-9302-43b0254810e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d" podUID="6c1bab87-9a09-48f6-9302-43b0254810e5"
Oct 30 01:45:20.544345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.544315    4425 scope.go:122] "RemoveContainer" containerID="a86342ec83950b5cb5ab68bf86a1fd9e0e90ce104d91c1865ddce4ce36fdda82"
Oct 30 01:45:20.772237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.766327    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv"]
Oct 30 01:45:20.800574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.800543    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm"
Oct 30 01:45:20.840807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.840780    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2a87024-9da9-4842-afb2-063b447bd15a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a87024-9da9-4842-afb2-063b447bd15a-maas-api-service-ca\") pod \"a2a87024-9da9-4842-afb2-063b447bd15a\" (UID: \"a2a87024-9da9-4842-afb2-063b447bd15a\") "
Oct 30 01:45:20.840885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.840866    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a87024-9da9-4842-afb2-063b447bd15a-kube-api-access-7prb5\" (UniqueName: \"kubernetes.io/projected/a2a87024-9da9-4842-afb2-063b447bd15a-kube-api-access-7prb5\") pod \"a2a87024-9da9-4842-afb2-063b447bd15a\" (UID: \"a2a87024-9da9-4842-afb2-063b447bd15a\") "
Oct 30 01:45:20.841584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.841460    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2a87024-9da9-4842-afb2-063b447bd15a-maas-api-service-ca" pod "a2a87024-9da9-4842-afb2-063b447bd15a" (UID: "a2a87024-9da9-4842-afb2-063b447bd15a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:20.844110 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.844085    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9"]
Oct 30 01:45:20.844679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.844658    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a87024-9da9-4842-afb2-063b447bd15a-kube-api-access-7prb5" pod "a2a87024-9da9-4842-afb2-063b447bd15a" (UID: "a2a87024-9da9-4842-afb2-063b447bd15a"). InnerVolumeSpecName "kube-api-access-7prb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:20.844835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.844801    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a87024-9da9-4842-afb2-063b447bd15a-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:20.874273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.874249    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf"
Oct 30 01:45:20.877115 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.877099    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd"
Oct 30 01:45:20.881723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.881705    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k"
Oct 30 01:45:20.885254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.885236    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7"
Oct 30 01:45:20.888183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.888149    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d"]
Oct 30 01:45:20.945506 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.945478    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cd1b902-d5db-4613-90ea-30a92c1bf2b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd1b902-d5db-4613-90ea-30a92c1bf2b0-maas-api-service-ca\") pod \"8cd1b902-d5db-4613-90ea-30a92c1bf2b0\" (UID: \"8cd1b902-d5db-4613-90ea-30a92c1bf2b0\") "
Oct 30 01:45:20.945628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.945532    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf3962c-f594-478e-89dc-a0ab9671bb9b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf3962c-f594-478e-89dc-a0ab9671bb9b-maas-api-service-ca\") pod \"3cf3962c-f594-478e-89dc-a0ab9671bb9b\" (UID: \"3cf3962c-f594-478e-89dc-a0ab9671bb9b\") "
Oct 30 01:45:20.945628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.945555    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357853d9-748f-44af-baa0-523e64187284-kube-api-access-zls8w\" (UniqueName: \"kubernetes.io/projected/357853d9-748f-44af-baa0-523e64187284-kube-api-access-zls8w\") pod \"357853d9-748f-44af-baa0-523e64187284\" (UID: \"357853d9-748f-44af-baa0-523e64187284\") "
Oct 30 01:45:20.945628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.945619    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caccf6b1-b7c1-437e-a634-84d62cf8d3b4-kube-api-access-6bfmc\" (UniqueName: \"kubernetes.io/projected/caccf6b1-b7c1-437e-a634-84d62cf8d3b4-kube-api-access-6bfmc\") pod \"caccf6b1-b7c1-437e-a634-84d62cf8d3b4\" (UID: \"caccf6b1-b7c1-437e-a634-84d62cf8d3b4\") "
Oct 30 01:45:20.945793 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.945646    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf3962c-f594-478e-89dc-a0ab9671bb9b-kube-api-access-wtkdk\" (UniqueName: \"kubernetes.io/projected/3cf3962c-f594-478e-89dc-a0ab9671bb9b-kube-api-access-wtkdk\") pod \"3cf3962c-f594-478e-89dc-a0ab9671bb9b\" (UID: \"3cf3962c-f594-478e-89dc-a0ab9671bb9b\") "
Oct 30 01:45:20.945793 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.945731    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caccf6b1-b7c1-437e-a634-84d62cf8d3b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caccf6b1-b7c1-437e-a634-84d62cf8d3b4-maas-api-service-ca\") pod \"caccf6b1-b7c1-437e-a634-84d62cf8d3b4\" (UID: \"caccf6b1-b7c1-437e-a634-84d62cf8d3b4\") "
Oct 30 01:45:20.945927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.945902    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/357853d9-748f-44af-baa0-523e64187284-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357853d9-748f-44af-baa0-523e64187284-maas-api-service-ca\") pod \"357853d9-748f-44af-baa0-523e64187284\" (UID: \"357853d9-748f-44af-baa0-523e64187284\") "
Oct 30 01:45:20.945988 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.945939    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cd1b902-d5db-4613-90ea-30a92c1bf2b0-kube-api-access-khdh7\" (UniqueName: \"kubernetes.io/projected/8cd1b902-d5db-4613-90ea-30a92c1bf2b0-kube-api-access-khdh7\") pod \"8cd1b902-d5db-4613-90ea-30a92c1bf2b0\" (UID: \"8cd1b902-d5db-4613-90ea-30a92c1bf2b0\") "
Oct 30 01:45:20.946046 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.946002    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf3962c-f594-478e-89dc-a0ab9671bb9b-maas-api-service-ca" pod "3cf3962c-f594-478e-89dc-a0ab9671bb9b" (UID: "3cf3962c-f594-478e-89dc-a0ab9671bb9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:20.946110 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.946091    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caccf6b1-b7c1-437e-a634-84d62cf8d3b4-maas-api-service-ca" pod "caccf6b1-b7c1-437e-a634-84d62cf8d3b4" (UID: "caccf6b1-b7c1-437e-a634-84d62cf8d3b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:20.946391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.946351    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357853d9-748f-44af-baa0-523e64187284-maas-api-service-ca" pod "357853d9-748f-44af-baa0-523e64187284" (UID: "357853d9-748f-44af-baa0-523e64187284"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:20.946509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.946405    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd1b902-d5db-4613-90ea-30a92c1bf2b0-maas-api-service-ca" pod "8cd1b902-d5db-4613-90ea-30a92c1bf2b0" (UID: "8cd1b902-d5db-4613-90ea-30a92c1bf2b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:20.946669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.946646    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357853d9-748f-44af-baa0-523e64187284-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:20.946741 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.946674    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd1b902-d5db-4613-90ea-30a92c1bf2b0-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:20.946741 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.946689    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cf3962c-f594-478e-89dc-a0ab9671bb9b-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:20.946741 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.946703    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7prb5\" (UniqueName: \"kubernetes.io/projected/a2a87024-9da9-4842-afb2-063b447bd15a-kube-api-access-7prb5\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:20.946741 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.946717    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caccf6b1-b7c1-437e-a634-84d62cf8d3b4-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:20.948023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.947995    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357853d9-748f-44af-baa0-523e64187284-kube-api-access-zls8w" pod "357853d9-748f-44af-baa0-523e64187284" (UID: "357853d9-748f-44af-baa0-523e64187284"). InnerVolumeSpecName "kube-api-access-zls8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:20.948115 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.948031    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf3962c-f594-478e-89dc-a0ab9671bb9b-kube-api-access-wtkdk" pod "3cf3962c-f594-478e-89dc-a0ab9671bb9b" (UID: "3cf3962c-f594-478e-89dc-a0ab9671bb9b"). InnerVolumeSpecName "kube-api-access-wtkdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:20.948253 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.948188    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd1b902-d5db-4613-90ea-30a92c1bf2b0-kube-api-access-khdh7" pod "8cd1b902-d5db-4613-90ea-30a92c1bf2b0" (UID: "8cd1b902-d5db-4613-90ea-30a92c1bf2b0"). InnerVolumeSpecName "kube-api-access-khdh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:20.948308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:20.948261    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caccf6b1-b7c1-437e-a634-84d62cf8d3b4-kube-api-access-6bfmc" pod "caccf6b1-b7c1-437e-a634-84d62cf8d3b4" (UID: "caccf6b1-b7c1-437e-a634-84d62cf8d3b4"). InnerVolumeSpecName "kube-api-access-6bfmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:21.048206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.048113    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zls8w\" (UniqueName: \"kubernetes.io/projected/357853d9-748f-44af-baa0-523e64187284-kube-api-access-zls8w\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:21.048206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.048148    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bfmc\" (UniqueName: \"kubernetes.io/projected/caccf6b1-b7c1-437e-a634-84d62cf8d3b4-kube-api-access-6bfmc\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:21.048206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.048189    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtkdk\" (UniqueName: \"kubernetes.io/projected/3cf3962c-f594-478e-89dc-a0ab9671bb9b-kube-api-access-wtkdk\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:21.048206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.048203    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khdh7\" (UniqueName: \"kubernetes.io/projected/8cd1b902-d5db-4613-90ea-30a92c1bf2b0-kube-api-access-khdh7\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:21.425282 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.425208    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336e3514-5287-4632-84f7-2673bc71a606" path="/var/lib/kubelet/pods/336e3514-5287-4632-84f7-2673bc71a606/volumes"
Oct 30 01:45:21.425689 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.425674    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d44cf871-7767-4ad3-a9b4-e70a58a52ec9" path="/var/lib/kubelet/pods/d44cf871-7767-4ad3-a9b4-e70a58a52ec9/volumes"
Oct 30 01:45:21.426069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.426057    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e253e33e-8ac1-4658-aaf1-485bb678f1b5" path="/var/lib/kubelet/pods/e253e33e-8ac1-4658-aaf1-485bb678f1b5/volumes"
Oct 30 01:45:21.426766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.426751    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77b1d4c-dc08-416b-ad79-6e777474940c" path="/var/lib/kubelet/pods/e77b1d4c-dc08-416b-ad79-6e777474940c/volumes"
Oct 30 01:45:21.543137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.543097    4425 generic.go:356] "Generic (PLEG): container finished" podID="ebecef45-aca8-486c-af0c-33bc1380da5d" containerID="2c13fc928b1638ecb641007af59504394c024d505436a04281b41b8424db17eb" exitCode=60
Oct 30 01:45:21.543609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.543191    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj" event={"ID":"ebecef45-aca8-486c-af0c-33bc1380da5d","Type":"ContainerDied","Data":"2c13fc928b1638ecb641007af59504394c024d505436a04281b41b8424db17eb"}
Oct 30 01:45:21.543609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.543261    4425 scope.go:122] "RemoveContainer" containerID="083313e8ae95fc1061277759b63caf4d78509885425cd6e4b3e1882d60ec2e88"
Oct 30 01:45:21.543743 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.543627    4425 scope.go:122] "RemoveContainer" containerID="2c13fc928b1638ecb641007af59504394c024d505436a04281b41b8424db17eb"
Oct 30 01:45:21.544001 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:21.543967    4425 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj_odh-ai-gateway-infra(ebecef45-aca8-486c-af0c-33bc1380da5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj" podUID="ebecef45-aca8-486c-af0c-33bc1380da5d"
Oct 30 01:45:21.545510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.545485    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd" event={"ID":"357853d9-748f-44af-baa0-523e64187284","Type":"ContainerDied","Data":"d6c61079e7fb299acefbf9d4495b6562d282070788bff1f1db27c1c242b06618"}
Oct 30 01:45:21.545938 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.545916    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd"
Oct 30 01:45:21.548405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.547997    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k"
Oct 30 01:45:21.548405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.547997    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k" event={"ID":"8cd1b902-d5db-4613-90ea-30a92c1bf2b0","Type":"ContainerDied","Data":"39ff05363bd13feded1da7a67169b89492f8b040019fe3115301d8118c05ea07"}
Oct 30 01:45:21.553590 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.553562    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7" event={"ID":"3cf3962c-f594-478e-89dc-a0ab9671bb9b","Type":"ContainerDied","Data":"12e5bcb1d4482cf0a4ec1e2e25ffea7bf1e53034c8a22b124f4f1fc37819e842"}
Oct 30 01:45:21.553684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.553649    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7"
Oct 30 01:45:21.555814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.555785    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf"
Oct 30 01:45:21.555904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.555885    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf" event={"ID":"caccf6b1-b7c1-437e-a634-84d62cf8d3b4","Type":"ContainerDied","Data":"a0881d748e2c12ba9f914870a8ddfa7b3aa278e4f2f76ac93aca64ee955f6a12"}
Oct 30 01:45:21.574389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.574362    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm" event={"ID":"a2a87024-9da9-4842-afb2-063b447bd15a","Type":"ContainerDied","Data":"39e01a46a38f3d74aa76282b834052050550a006105ac2cd2e2b0f4f7129d177"}
Oct 30 01:45:21.574504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.574458    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm"
Oct 30 01:45:21.605030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.605008    4425 scope.go:122] "RemoveContainer" containerID="97dff42541264adfa7919e701242ea349a5bb9865c2786ffd545ec4b5088b962"
Oct 30 01:45:21.632144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.631976    4425 scope.go:122] "RemoveContainer" containerID="2fe38aa77ad44c39ceba25a348b06220be0fc2af7d295fc3462d565948ac2071"
Oct 30 01:45:21.658347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.657118    4425 scope.go:122] "RemoveContainer" containerID="ac1b37cef21831cf3cf1e681def6aa9e16cda7feebcedca75157d41da8fc556e"
Oct 30 01:45:21.675931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.675901    4425 scope.go:122] "RemoveContainer" containerID="aa59d1d5f2d0c8142fe7c6d93e88e615b35e2775471c8424f49f5b1dd1a4d630"
Oct 30 01:45:21.686481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.686445    4425 scope.go:122] "RemoveContainer" containerID="917e61ce493809421d51d7d673081a5a3c675bf5c91f81ec6dbb8bec92bdacfc"
Oct 30 01:45:21.703533 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.699395    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k"]
Oct 30 01:45:21.706328 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.706302    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-664c2b-29888745-zdm2k"]
Oct 30 01:45:21.727121 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.727067    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj"]
Oct 30 01:45:21.768933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.768882    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d"
Oct 30 01:45:21.814668 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.814643    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd"]
Oct 30 01:45:21.821413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.821389    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2f820e-29888745-wdfbd"]
Oct 30 01:45:21.836975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.836953    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9"
Oct 30 01:45:21.840741 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.840726    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv"
Oct 30 01:45:21.852105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.852079    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7"]
Oct 30 01:45:21.862268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.862232    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c1bab87-9a09-48f6-9302-43b0254810e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1bab87-9a09-48f6-9302-43b0254810e5-maas-api-service-ca\") pod \"6c1bab87-9a09-48f6-9302-43b0254810e5\" (UID: \"6c1bab87-9a09-48f6-9302-43b0254810e5\") "
Oct 30 01:45:21.862432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.862321    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c1bab87-9a09-48f6-9302-43b0254810e5-kube-api-access-lskpp\" (UniqueName: \"kubernetes.io/projected/6c1bab87-9a09-48f6-9302-43b0254810e5-kube-api-access-lskpp\") pod \"6c1bab87-9a09-48f6-9302-43b0254810e5\" (UID: \"6c1bab87-9a09-48f6-9302-43b0254810e5\") "
Oct 30 01:45:21.862686 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.862658    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1bab87-9a09-48f6-9302-43b0254810e5-maas-api-service-ca" pod "6c1bab87-9a09-48f6-9302-43b0254810e5" (UID: "6c1bab87-9a09-48f6-9302-43b0254810e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:21.863145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.863104    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1bab87-9a09-48f6-9302-43b0254810e5-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:21.863905 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.863884    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5457f2-29888745-6t9x7"]
Oct 30 01:45:21.865402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.865336    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1bab87-9a09-48f6-9302-43b0254810e5-kube-api-access-lskpp" pod "6c1bab87-9a09-48f6-9302-43b0254810e5" (UID: "6c1bab87-9a09-48f6-9302-43b0254810e5"). InnerVolumeSpecName "kube-api-access-lskpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:21.896201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.896176    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf"]
Oct 30 01:45:21.901583 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.901560    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1cb829-29888745-rkkdf"]
Oct 30 01:45:21.931291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.931233    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm"]
Oct 30 01:45:21.942641 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.942613    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d5b2b8-29888745-rv9fm"]
Oct 30 01:45:21.964108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.964084    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee57e527-71ff-4a09-8db0-9840a6467fce-kube-api-access-kdvpv\" (UniqueName: \"kubernetes.io/projected/ee57e527-71ff-4a09-8db0-9840a6467fce-kube-api-access-kdvpv\") pod \"ee57e527-71ff-4a09-8db0-9840a6467fce\" (UID: \"ee57e527-71ff-4a09-8db0-9840a6467fce\") "
Oct 30 01:45:21.964210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.964150    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee57e527-71ff-4a09-8db0-9840a6467fce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee57e527-71ff-4a09-8db0-9840a6467fce-maas-api-service-ca\") pod \"ee57e527-71ff-4a09-8db0-9840a6467fce\" (UID: \"ee57e527-71ff-4a09-8db0-9840a6467fce\") "
Oct 30 01:45:21.964341 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.964317    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a82dd87-6039-44da-b690-297f5449c337-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a82dd87-6039-44da-b690-297f5449c337-maas-api-service-ca\") pod \"9a82dd87-6039-44da-b690-297f5449c337\" (UID: \"9a82dd87-6039-44da-b690-297f5449c337\") "
Oct 30 01:45:21.964578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.964556    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a82dd87-6039-44da-b690-297f5449c337-kube-api-access-t6hf7\" (UniqueName: \"kubernetes.io/projected/9a82dd87-6039-44da-b690-297f5449c337-kube-api-access-t6hf7\") pod \"9a82dd87-6039-44da-b690-297f5449c337\" (UID: \"9a82dd87-6039-44da-b690-297f5449c337\") "
Oct 30 01:45:21.964675 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.964658    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a82dd87-6039-44da-b690-297f5449c337-maas-api-service-ca" pod "9a82dd87-6039-44da-b690-297f5449c337" (UID: "9a82dd87-6039-44da-b690-297f5449c337"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:21.964732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.964665    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee57e527-71ff-4a09-8db0-9840a6467fce-maas-api-service-ca" pod "ee57e527-71ff-4a09-8db0-9840a6467fce" (UID: "ee57e527-71ff-4a09-8db0-9840a6467fce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:21.965078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.965051    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a82dd87-6039-44da-b690-297f5449c337-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:21.965078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.965073    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lskpp\" (UniqueName: \"kubernetes.io/projected/6c1bab87-9a09-48f6-9302-43b0254810e5-kube-api-access-lskpp\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:21.965214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.965087    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee57e527-71ff-4a09-8db0-9840a6467fce-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:21.966104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.966085    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee57e527-71ff-4a09-8db0-9840a6467fce-kube-api-access-kdvpv" pod "ee57e527-71ff-4a09-8db0-9840a6467fce" (UID: "ee57e527-71ff-4a09-8db0-9840a6467fce"). InnerVolumeSpecName "kube-api-access-kdvpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:21.966557 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:21.966538    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a82dd87-6039-44da-b690-297f5449c337-kube-api-access-t6hf7" pod "9a82dd87-6039-44da-b690-297f5449c337" (UID: "9a82dd87-6039-44da-b690-297f5449c337"). InnerVolumeSpecName "kube-api-access-t6hf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:22.066044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.066008    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdvpv\" (UniqueName: \"kubernetes.io/projected/ee57e527-71ff-4a09-8db0-9840a6467fce-kube-api-access-kdvpv\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:22.066044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.066048    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6hf7\" (UniqueName: \"kubernetes.io/projected/9a82dd87-6039-44da-b690-297f5449c337-kube-api-access-t6hf7\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:22.583720 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.583692    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9"
Oct 30 01:45:22.583720 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.583698    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9" event={"ID":"9a82dd87-6039-44da-b690-297f5449c337","Type":"ContainerDied","Data":"28d891bd860e8e947c8ea249602f387cfc5e6e30f950c4230cee346f52ee1dd1"}
Oct 30 01:45:22.584215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.583767    4425 scope.go:122] "RemoveContainer" containerID="3b922a20408f25d74429e0748eab28a25a4e1db4834a100d6c2d43ec3cc719a7"
Oct 30 01:45:22.585347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.585328    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv"
Oct 30 01:45:22.585441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.585340    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv" event={"ID":"ee57e527-71ff-4a09-8db0-9840a6467fce","Type":"ContainerDied","Data":"fdd69bf54b8a8b3e86ec910052e5582103e50bdde1bd6890d5ebc2890571229f"}
Oct 30 01:45:22.593354 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.593334    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d"
Oct 30 01:45:22.594392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.593586    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d" event={"ID":"6c1bab87-9a09-48f6-9302-43b0254810e5","Type":"ContainerDied","Data":"118d333264a6503706b0c5080d49a55ed80776bd50d0d7eee8fbaa9ccd4d3845"}
Oct 30 01:45:22.598235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.597986    4425 scope.go:122] "RemoveContainer" containerID="185f46dfd067fd899ada86f50b1b3e3cac84ec0867ec9c167457356ac0e8381d"
Oct 30 01:45:22.607840 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.607819    4425 scope.go:122] "RemoveContainer" containerID="8178d8f1469dbf49729fe626496d835a2b759a2aff9ce1cf2bcf7451b743d1f2"
Oct 30 01:45:22.697806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.697768    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv"]
Oct 30 01:45:22.701931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.701901    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-73423b-29888745-97pjv"]
Oct 30 01:45:22.729030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.729008    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9"]
Oct 30 01:45:22.735766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.735744    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-dd80cb-29888745-q25s9"]
Oct 30 01:45:22.756404 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.756381    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj"
Oct 30 01:45:22.764345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.764322    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d"]
Oct 30 01:45:22.770553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.770531    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9bfe36-29888745-b6k5d"]
Oct 30 01:45:22.875829 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.875753    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebecef45-aca8-486c-af0c-33bc1380da5d-kube-api-access-ql2cz\" (UniqueName: \"kubernetes.io/projected/ebecef45-aca8-486c-af0c-33bc1380da5d-kube-api-access-ql2cz\") pod \"ebecef45-aca8-486c-af0c-33bc1380da5d\" (UID: \"ebecef45-aca8-486c-af0c-33bc1380da5d\") "
Oct 30 01:45:22.875829 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.875814    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebecef45-aca8-486c-af0c-33bc1380da5d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebecef45-aca8-486c-af0c-33bc1380da5d-maas-api-service-ca\") pod \"ebecef45-aca8-486c-af0c-33bc1380da5d\" (UID: \"ebecef45-aca8-486c-af0c-33bc1380da5d\") "
Oct 30 01:45:22.876206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.876147    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebecef45-aca8-486c-af0c-33bc1380da5d-maas-api-service-ca" pod "ebecef45-aca8-486c-af0c-33bc1380da5d" (UID: "ebecef45-aca8-486c-af0c-33bc1380da5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:22.876310 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.876297    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebecef45-aca8-486c-af0c-33bc1380da5d-maas-api-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:22.877779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.877756    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebecef45-aca8-486c-af0c-33bc1380da5d-kube-api-access-ql2cz" pod "ebecef45-aca8-486c-af0c-33bc1380da5d" (UID: "ebecef45-aca8-486c-af0c-33bc1380da5d"). InnerVolumeSpecName "kube-api-access-ql2cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:22.977260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:22.977236    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql2cz\" (UniqueName: \"kubernetes.io/projected/ebecef45-aca8-486c-af0c-33bc1380da5d-kube-api-access-ql2cz\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:23.425025 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:23.424994    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="357853d9-748f-44af-baa0-523e64187284" path="/var/lib/kubelet/pods/357853d9-748f-44af-baa0-523e64187284/volumes"
Oct 30 01:45:23.425464 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:23.425447    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf3962c-f594-478e-89dc-a0ab9671bb9b" path="/var/lib/kubelet/pods/3cf3962c-f594-478e-89dc-a0ab9671bb9b/volumes"
Oct 30 01:45:23.425842 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:23.425829    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c1bab87-9a09-48f6-9302-43b0254810e5" path="/var/lib/kubelet/pods/6c1bab87-9a09-48f6-9302-43b0254810e5/volumes"
Oct 30 01:45:23.426512 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:23.426497    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cd1b902-d5db-4613-90ea-30a92c1bf2b0" path="/var/lib/kubelet/pods/8cd1b902-d5db-4613-90ea-30a92c1bf2b0/volumes"
Oct 30 01:45:23.426917 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:23.426902    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a82dd87-6039-44da-b690-297f5449c337" path="/var/lib/kubelet/pods/9a82dd87-6039-44da-b690-297f5449c337/volumes"
Oct 30 01:45:23.427384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:23.427367    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2a87024-9da9-4842-afb2-063b447bd15a" path="/var/lib/kubelet/pods/a2a87024-9da9-4842-afb2-063b447bd15a/volumes"
Oct 30 01:45:23.428049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:23.428034    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caccf6b1-b7c1-437e-a634-84d62cf8d3b4" path="/var/lib/kubelet/pods/caccf6b1-b7c1-437e-a634-84d62cf8d3b4/volumes"
Oct 30 01:45:23.428453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:23.428439    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee57e527-71ff-4a09-8db0-9840a6467fce" path="/var/lib/kubelet/pods/ee57e527-71ff-4a09-8db0-9840a6467fce/volumes"
Oct 30 01:45:23.608351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:23.608325    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj"
Oct 30 01:45:23.608766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:23.608352    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj" event={"ID":"ebecef45-aca8-486c-af0c-33bc1380da5d","Type":"ContainerDied","Data":"9132d1cee819fbeae251ee75985f5a7adf1cc50c07d2b334a747ab34215d41f7"}
Oct 30 01:45:23.608766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:23.608412    4425 scope.go:122] "RemoveContainer" containerID="2c13fc928b1638ecb641007af59504394c024d505436a04281b41b8424db17eb"
Oct 30 01:45:23.655117 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:23.655095    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj"]
Oct 30 01:45:23.659196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:23.659174    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1a4b17-29888745-qwtlj"]
Oct 30 01:45:25.429843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:25.429802    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebecef45-aca8-486c-af0c-33bc1380da5d" path="/var/lib/kubelet/pods/ebecef45-aca8-486c-af0c-33bc1380da5d/volumes"
Oct 30 01:45:37.556334 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:37.556295    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:45:38.097242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.097203    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"]
Oct 30 01:45:38.098000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.097977    4425 cpu_manager.go:414] "containerMap: removing container" podUID="336e3514-5287-4632-84f7-2673bc71a606" containerName="cleanup"
Oct 30 01:45:38.098000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098000    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336e3514-5287-4632-84f7-2673bc71a606" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098014    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ee57e527-71ff-4a09-8db0-9840a6467fce" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098021    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee57e527-71ff-4a09-8db0-9840a6467fce" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098033    4425 cpu_manager.go:414] "containerMap: removing container" podUID="3cf3962c-f594-478e-89dc-a0ab9671bb9b" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098040    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf3962c-f594-478e-89dc-a0ab9671bb9b" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098053    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9a82dd87-6039-44da-b690-297f5449c337" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098060    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a82dd87-6039-44da-b690-297f5449c337" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098070    4425 cpu_manager.go:414] "containerMap: removing container" podUID="3cf3962c-f594-478e-89dc-a0ab9671bb9b" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098077    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf3962c-f594-478e-89dc-a0ab9671bb9b" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098087    4425 cpu_manager.go:414] "containerMap: removing container" podUID="af779dae-521f-4f8a-9ad2-e7fe3948dcbc" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098094    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af779dae-521f-4f8a-9ad2-e7fe3948dcbc" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098102    4425 cpu_manager.go:414] "containerMap: removing container" podUID="357853d9-748f-44af-baa0-523e64187284" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098108    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357853d9-748f-44af-baa0-523e64187284" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098119    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8cd1b902-d5db-4613-90ea-30a92c1bf2b0" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098126    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cd1b902-d5db-4613-90ea-30a92c1bf2b0" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098138    4425 cpu_manager.go:414] "containerMap: removing container" podUID="caccf6b1-b7c1-437e-a634-84d62cf8d3b4" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098145    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caccf6b1-b7c1-437e-a634-84d62cf8d3b4" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098170    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a2a87024-9da9-4842-afb2-063b447bd15a" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098177    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a87024-9da9-4842-afb2-063b447bd15a" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098192    4425 cpu_manager.go:414] "containerMap: removing container" podUID="357853d9-748f-44af-baa0-523e64187284" containerName="cleanup"
Oct 30 01:45:38.098192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098200    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357853d9-748f-44af-baa0-523e64187284" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098216    4425 cpu_manager.go:414] "containerMap: removing container" podUID="e253e33e-8ac1-4658-aaf1-485bb678f1b5" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098223    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e253e33e-8ac1-4658-aaf1-485bb678f1b5" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098233    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d44cf871-7767-4ad3-a9b4-e70a58a52ec9" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098240    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44cf871-7767-4ad3-a9b4-e70a58a52ec9" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098253    4425 cpu_manager.go:414] "containerMap: removing container" podUID="3cf3962c-f594-478e-89dc-a0ab9671bb9b" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098260    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf3962c-f594-478e-89dc-a0ab9671bb9b" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098271    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ee57e527-71ff-4a09-8db0-9840a6467fce" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098277    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee57e527-71ff-4a09-8db0-9840a6467fce" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098292    4425 cpu_manager.go:414] "containerMap: removing container" podUID="357853d9-748f-44af-baa0-523e64187284" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098298    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357853d9-748f-44af-baa0-523e64187284" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098320    4425 cpu_manager.go:414] "containerMap: removing container" podUID="caccf6b1-b7c1-437e-a634-84d62cf8d3b4" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098327    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caccf6b1-b7c1-437e-a634-84d62cf8d3b4" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098340    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9a82dd87-6039-44da-b690-297f5449c337" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098347    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a82dd87-6039-44da-b690-297f5449c337" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098355    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d44cf871-7767-4ad3-a9b4-e70a58a52ec9" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098361    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44cf871-7767-4ad3-a9b4-e70a58a52ec9" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098379    4425 cpu_manager.go:414] "containerMap: removing container" podUID="6c1bab87-9a09-48f6-9302-43b0254810e5" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098386    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c1bab87-9a09-48f6-9302-43b0254810e5" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098398    4425 cpu_manager.go:414] "containerMap: removing container" podUID="6c1bab87-9a09-48f6-9302-43b0254810e5" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098405    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c1bab87-9a09-48f6-9302-43b0254810e5" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098417    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ebecef45-aca8-486c-af0c-33bc1380da5d" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098424    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebecef45-aca8-486c-af0c-33bc1380da5d" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098438    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a2a87024-9da9-4842-afb2-063b447bd15a" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098445    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a87024-9da9-4842-afb2-063b447bd15a" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098456    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8cd1b902-d5db-4613-90ea-30a92c1bf2b0" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098464    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cd1b902-d5db-4613-90ea-30a92c1bf2b0" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098478    4425 cpu_manager.go:414] "containerMap: removing container" podUID="e77b1d4c-dc08-416b-ad79-6e777474940c" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098486    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77b1d4c-dc08-416b-ad79-6e777474940c" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098506    4425 cpu_manager.go:414] "containerMap: removing container" podUID="e77b1d4c-dc08-416b-ad79-6e777474940c" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098514    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77b1d4c-dc08-416b-ad79-6e777474940c" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098525    4425 cpu_manager.go:414] "containerMap: removing container" podUID="336e3514-5287-4632-84f7-2673bc71a606" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098532    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336e3514-5287-4632-84f7-2673bc71a606" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098546    4425 cpu_manager.go:414] "containerMap: removing container" podUID="336e3514-5287-4632-84f7-2673bc71a606" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098553    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336e3514-5287-4632-84f7-2673bc71a606" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098568    4425 cpu_manager.go:414] "containerMap: removing container" podUID="e253e33e-8ac1-4658-aaf1-485bb678f1b5" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098575    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e253e33e-8ac1-4658-aaf1-485bb678f1b5" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098590    4425 cpu_manager.go:414] "containerMap: removing container" podUID="e253e33e-8ac1-4658-aaf1-485bb678f1b5" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098596    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e253e33e-8ac1-4658-aaf1-485bb678f1b5" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098611    4425 cpu_manager.go:414] "containerMap: removing container" podUID="af779dae-521f-4f8a-9ad2-e7fe3948dcbc" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098618    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af779dae-521f-4f8a-9ad2-e7fe3948dcbc" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098627    4425 cpu_manager.go:414] "containerMap: removing container" podUID="2efb0b19-a05d-46bd-bf1d-622e6410cafe" containerName="payload-processing"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098634    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2efb0b19-a05d-46bd-bf1d-622e6410cafe" containerName="payload-processing"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098645    4425 cpu_manager.go:414] "containerMap: removing container" podUID="e77b1d4c-dc08-416b-ad79-6e777474940c" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098652    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77b1d4c-dc08-416b-ad79-6e777474940c" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098663    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ebecef45-aca8-486c-af0c-33bc1380da5d" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098670    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebecef45-aca8-486c-af0c-33bc1380da5d" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098679    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ebecef45-aca8-486c-af0c-33bc1380da5d" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098686    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebecef45-aca8-486c-af0c-33bc1380da5d" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098886    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="336e3514-5287-4632-84f7-2673bc71a606" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098900    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee57e527-71ff-4a09-8db0-9840a6467fce" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098909    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf3962c-f594-478e-89dc-a0ab9671bb9b" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098919    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a82dd87-6039-44da-b690-297f5449c337" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098927    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf3962c-f594-478e-89dc-a0ab9671bb9b" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098936    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="af779dae-521f-4f8a-9ad2-e7fe3948dcbc" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098943    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="357853d9-748f-44af-baa0-523e64187284" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098950    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cd1b902-d5db-4613-90ea-30a92c1bf2b0" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098959    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="caccf6b1-b7c1-437e-a634-84d62cf8d3b4" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098969    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a87024-9da9-4842-afb2-063b447bd15a" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098977    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e253e33e-8ac1-4658-aaf1-485bb678f1b5" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098985    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44cf871-7767-4ad3-a9b4-e70a58a52ec9" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.098996    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee57e527-71ff-4a09-8db0-9840a6467fce" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099006    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="357853d9-748f-44af-baa0-523e64187284" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099015    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44cf871-7767-4ad3-a9b4-e70a58a52ec9" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099023    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cd1b902-d5db-4613-90ea-30a92c1bf2b0" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099033    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a82dd87-6039-44da-b690-297f5449c337" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099041    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44cf871-7767-4ad3-a9b4-e70a58a52ec9" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099053    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c1bab87-9a09-48f6-9302-43b0254810e5" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099062    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c1bab87-9a09-48f6-9302-43b0254810e5" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099071    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebecef45-aca8-486c-af0c-33bc1380da5d" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099079    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cd1b902-d5db-4613-90ea-30a92c1bf2b0" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099089    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77b1d4c-dc08-416b-ad79-6e777474940c" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099096    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee57e527-71ff-4a09-8db0-9840a6467fce" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099106    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a87024-9da9-4842-afb2-063b447bd15a" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099116    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="336e3514-5287-4632-84f7-2673bc71a606" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099125    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e253e33e-8ac1-4658-aaf1-485bb678f1b5" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099132    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="caccf6b1-b7c1-437e-a634-84d62cf8d3b4" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099140    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e253e33e-8ac1-4658-aaf1-485bb678f1b5" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099147    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="af779dae-521f-4f8a-9ad2-e7fe3948dcbc" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099170    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="2efb0b19-a05d-46bd-bf1d-622e6410cafe" containerName="payload-processing"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099181    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77b1d4c-dc08-416b-ad79-6e777474940c" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099189    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebecef45-aca8-486c-af0c-33bc1380da5d" containerName="cleanup"
Oct 30 01:45:38.099275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.099198    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebecef45-aca8-486c-af0c-33bc1380da5d" containerName="cleanup"
Oct 30 01:45:38.105993 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.105972    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"
Oct 30 01:45:38.121319 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.121294    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"]
Oct 30 01:45:38.125846 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.125824    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:45:38.125969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.125946    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:45:38.150680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.150653    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47cfb795-a866-454b-813c-b81eb4ca8271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d36c4a3-dvsrv\" (UID: \"47cfb795-a866-454b-813c-b81eb4ca8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"
Oct 30 01:45:38.150808 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.150695    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47cfb795-a866-454b-813c-b81eb4ca8271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d36c4a3-dvsrv\" (UID: \"47cfb795-a866-454b-813c-b81eb4ca8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"
Oct 30 01:45:38.150899 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.150881    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwppk\" (UniqueName: \"kubernetes.io/projected/47cfb795-a866-454b-813c-b81eb4ca8271-kube-api-access-hwppk\") pod \"maas-api-revoke-sub-5d36c4a3-dvsrv\" (UID: \"47cfb795-a866-454b-813c-b81eb4ca8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"
Oct 30 01:45:38.252289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.252258    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwppk\" (UniqueName: \"kubernetes.io/projected/47cfb795-a866-454b-813c-b81eb4ca8271-kube-api-access-hwppk\") pod \"maas-api-revoke-sub-5d36c4a3-dvsrv\" (UID: \"47cfb795-a866-454b-813c-b81eb4ca8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"
Oct 30 01:45:38.252452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.252373    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47cfb795-a866-454b-813c-b81eb4ca8271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d36c4a3-dvsrv\" (UID: \"47cfb795-a866-454b-813c-b81eb4ca8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"
Oct 30 01:45:38.252452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.252395    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47cfb795-a866-454b-813c-b81eb4ca8271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d36c4a3-dvsrv\" (UID: \"47cfb795-a866-454b-813c-b81eb4ca8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"
Oct 30 01:45:38.253032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.253011    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47cfb795-a866-454b-813c-b81eb4ca8271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d36c4a3-dvsrv\" (UID: \"47cfb795-a866-454b-813c-b81eb4ca8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"
Oct 30 01:45:38.254795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.254772    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47cfb795-a866-454b-813c-b81eb4ca8271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d36c4a3-dvsrv\" (UID: \"47cfb795-a866-454b-813c-b81eb4ca8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"
Oct 30 01:45:38.273467 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.273438    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwppk\" (UniqueName: \"kubernetes.io/projected/47cfb795-a866-454b-813c-b81eb4ca8271-kube-api-access-hwppk\") pod \"maas-api-revoke-sub-5d36c4a3-dvsrv\" (UID: \"47cfb795-a866-454b-813c-b81eb4ca8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"
Oct 30 01:45:38.421590 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.421511    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"
Oct 30 01:45:38.725793 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:45:38.725760    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47cfb795_a866_454b_813c_b81eb4ca8271.slice/crio-9372caf7f65a17af9ab6770efd908692a462556bf7183ea322d0c18299154d22 WatchSource:0}: Error finding container 9372caf7f65a17af9ab6770efd908692a462556bf7183ea322d0c18299154d22: Status 404 returned error can't find the container with id 9372caf7f65a17af9ab6770efd908692a462556bf7183ea322d0c18299154d22
Oct 30 01:45:38.726233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.725981    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"]
Oct 30 01:45:38.744288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:38.744262    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv" event={"ID":"47cfb795-a866-454b-813c-b81eb4ca8271","Type":"ContainerStarted","Data":"9372caf7f65a17af9ab6770efd908692a462556bf7183ea322d0c18299154d22"}
Oct 30 01:45:39.756015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:39.755973    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv" event={"ID":"47cfb795-a866-454b-813c-b81eb4ca8271","Type":"ContainerStarted","Data":"7f33d774e4f19c3416c2be3f4fc69e3714111e6c603209b86883129b9bda237d"}
Oct 30 01:45:39.794878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:39.794824    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv" podStartSLOduration=1.794809683 podStartE2EDuration="1.794809683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:39.791778668 +0000 UTC m=+3848.541433987" watchObservedRunningTime="2026-10-30 01:45:39.794809683 +0000 UTC m=+3848.544464990"
Oct 30 01:45:40.765447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:40.765412    4425 generic.go:356] "Generic (PLEG): container finished" podID="47cfb795-a866-454b-813c-b81eb4ca8271" containerID="7f33d774e4f19c3416c2be3f4fc69e3714111e6c603209b86883129b9bda237d" exitCode=0
Oct 30 01:45:40.765861 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:40.765498    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv" event={"ID":"47cfb795-a866-454b-813c-b81eb4ca8271","Type":"ContainerDied","Data":"7f33d774e4f19c3416c2be3f4fc69e3714111e6c603209b86883129b9bda237d"}
Oct 30 01:45:41.939975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:41.939952    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"
Oct 30 01:45:41.995109 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:41.995077    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47cfb795-a866-454b-813c-b81eb4ca8271-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47cfb795-a866-454b-813c-b81eb4ca8271-maas-api-openshift-ca\") pod \"47cfb795-a866-454b-813c-b81eb4ca8271\" (UID: \"47cfb795-a866-454b-813c-b81eb4ca8271\") "
Oct 30 01:45:41.995275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:41.995214    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47cfb795-a866-454b-813c-b81eb4ca8271-kube-api-access-hwppk\" (UniqueName: \"kubernetes.io/projected/47cfb795-a866-454b-813c-b81eb4ca8271-kube-api-access-hwppk\") pod \"47cfb795-a866-454b-813c-b81eb4ca8271\" (UID: \"47cfb795-a866-454b-813c-b81eb4ca8271\") "
Oct 30 01:45:41.995402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:41.995384    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47cfb795-a866-454b-813c-b81eb4ca8271-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47cfb795-a866-454b-813c-b81eb4ca8271-maas-api-xks-ca\") pod \"47cfb795-a866-454b-813c-b81eb4ca8271\" (UID: \"47cfb795-a866-454b-813c-b81eb4ca8271\") "
Oct 30 01:45:41.995548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:41.995520    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47cfb795-a866-454b-813c-b81eb4ca8271-maas-api-openshift-ca" pod "47cfb795-a866-454b-813c-b81eb4ca8271" (UID: "47cfb795-a866-454b-813c-b81eb4ca8271"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:41.995970 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:41.995943    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47cfb795-a866-454b-813c-b81eb4ca8271-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:41.997370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:41.997343    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47cfb795-a866-454b-813c-b81eb4ca8271-kube-api-access-hwppk" pod "47cfb795-a866-454b-813c-b81eb4ca8271" (UID: "47cfb795-a866-454b-813c-b81eb4ca8271"). InnerVolumeSpecName "kube-api-access-hwppk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:41.997723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:41.997703    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47cfb795-a866-454b-813c-b81eb4ca8271-maas-api-xks-ca" pod "47cfb795-a866-454b-813c-b81eb4ca8271" (UID: "47cfb795-a866-454b-813c-b81eb4ca8271"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:45:42.097267 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:42.097206    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwppk\" (UniqueName: \"kubernetes.io/projected/47cfb795-a866-454b-813c-b81eb4ca8271-kube-api-access-hwppk\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:42.097267 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:42.097229    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47cfb795-a866-454b-813c-b81eb4ca8271-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:42.782048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:42.782018    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"
Oct 30 01:45:42.782222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:42.782059    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv" event={"ID":"47cfb795-a866-454b-813c-b81eb4ca8271","Type":"ContainerDied","Data":"9372caf7f65a17af9ab6770efd908692a462556bf7183ea322d0c18299154d22"}
Oct 30 01:45:42.782222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:42.782113    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9372caf7f65a17af9ab6770efd908692a462556bf7183ea322d0c18299154d22"
Oct 30 01:45:46.230647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.230613    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"]
Oct 30 01:45:46.231135 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.230797    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v" podUID="1e2b9d25-c7ed-451c-8b66-ec294e97b3e2" containerName="maas-api" containerID="cri-o://46ae6bc5a5c19f2d8e3a86f84a8a32c1b7af4b20522bc29b51937c4a3e47431c" gracePeriod=30
Oct 30 01:45:46.547426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.547393    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:45:46.658022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.657986    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-metrics-certs\") pod \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\" (UID: \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\") "
Oct 30 01:45:46.658223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.658057    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-openshift-service-ca\") pod \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\" (UID: \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\") "
Oct 30 01:45:46.658223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.658098    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-kube-api-access-kdbhl\" (UniqueName: \"kubernetes.io/projected/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-kube-api-access-kdbhl\") pod \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\" (UID: \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\") "
Oct 30 01:45:46.658223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.658137    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-maas-api-tls\") pod \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\" (UID: \"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2\") "
Oct 30 01:45:46.658511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.658477    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-openshift-service-ca" pod "1e2b9d25-c7ed-451c-8b66-ec294e97b3e2" (UID: "1e2b9d25-c7ed-451c-8b66-ec294e97b3e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:46.658997 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.658973    4425 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-openshift-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:46.660286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.660254    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-maas-api-tls" pod "1e2b9d25-c7ed-451c-8b66-ec294e97b3e2" (UID: "1e2b9d25-c7ed-451c-8b66-ec294e97b3e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:45:46.660403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.660303    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-metrics-certs" pod "1e2b9d25-c7ed-451c-8b66-ec294e97b3e2" (UID: "1e2b9d25-c7ed-451c-8b66-ec294e97b3e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:45:46.660403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.660369    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-kube-api-access-kdbhl" pod "1e2b9d25-c7ed-451c-8b66-ec294e97b3e2" (UID: "1e2b9d25-c7ed-451c-8b66-ec294e97b3e2"). InnerVolumeSpecName "kube-api-access-kdbhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:46.760670 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.760633    4425 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-metrics-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:46.760670 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.760672    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdbhl\" (UniqueName: \"kubernetes.io/projected/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-kube-api-access-kdbhl\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:46.760963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.760687    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2-maas-api-tls\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:46.815457 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.815380    4425 generic.go:356] "Generic (PLEG): container finished" podID="1e2b9d25-c7ed-451c-8b66-ec294e97b3e2" containerID="46ae6bc5a5c19f2d8e3a86f84a8a32c1b7af4b20522bc29b51937c4a3e47431c" exitCode=0
Oct 30 01:45:46.815457 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.815422    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"
Oct 30 01:45:46.815672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.815460    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v" event={"ID":"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2","Type":"ContainerDied","Data":"46ae6bc5a5c19f2d8e3a86f84a8a32c1b7af4b20522bc29b51937c4a3e47431c"}
Oct 30 01:45:46.815672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.815518    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v" event={"ID":"1e2b9d25-c7ed-451c-8b66-ec294e97b3e2","Type":"ContainerDied","Data":"6aeec947b78cd74b3fbfa3460eed816bd3cac114734e4470c8207b23886baa7a"}
Oct 30 01:45:46.815672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.815547    4425 scope.go:122] "RemoveContainer" containerID="46ae6bc5a5c19f2d8e3a86f84a8a32c1b7af4b20522bc29b51937c4a3e47431c"
Oct 30 01:45:46.827403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.827384    4425 scope.go:122] "RemoveContainer" containerID="46ae6bc5a5c19f2d8e3a86f84a8a32c1b7af4b20522bc29b51937c4a3e47431c"
Oct 30 01:45:46.827679 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:45:46.827660    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ae6bc5a5c19f2d8e3a86f84a8a32c1b7af4b20522bc29b51937c4a3e47431c\": container with ID starting with 46ae6bc5a5c19f2d8e3a86f84a8a32c1b7af4b20522bc29b51937c4a3e47431c not found: ID does not exist" containerID="46ae6bc5a5c19f2d8e3a86f84a8a32c1b7af4b20522bc29b51937c4a3e47431c"
Oct 30 01:45:46.827723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.827689    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ae6bc5a5c19f2d8e3a86f84a8a32c1b7af4b20522bc29b51937c4a3e47431c"} err="failed to get container status \"46ae6bc5a5c19f2d8e3a86f84a8a32c1b7af4b20522bc29b51937c4a3e47431c\": rpc error: code = NotFound desc = could not find container \"46ae6bc5a5c19f2d8e3a86f84a8a32c1b7af4b20522bc29b51937c4a3e47431c\": container with ID starting with 46ae6bc5a5c19f2d8e3a86f84a8a32c1b7af4b20522bc29b51937c4a3e47431c not found: ID does not exist"
Oct 30 01:45:46.870284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.870263    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"]
Oct 30 01:45:46.875062 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.875041    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-73423b-5dbf498786-ddm6v"]
Oct 30 01:45:46.944361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.944326    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"]
Oct 30 01:45:46.948201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:46.948175    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73423b-9gvdv"]
Oct 30 01:45:47.424047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:47.424010    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e2b9d25-c7ed-451c-8b66-ec294e97b3e2" path="/var/lib/kubelet/pods/1e2b9d25-c7ed-451c-8b66-ec294e97b3e2/volumes"
Oct 30 01:45:47.424565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:47.424549    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c70768-1d75-4208-a8b4-a745ef199520" path="/var/lib/kubelet/pods/41c70768-1d75-4208-a8b4-a745ef199520/volumes"
Oct 30 01:45:48.597587 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:48.597545    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"]
Oct 30 01:45:48.598079 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:48.597833    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b" podUID="45aca910-a167-41a3-8792-43502dd22283" containerName="maas-api" containerID="cri-o://0fcf5d10b55f07f02f684969a3430fd02602345202ba1f6110ac959f32cf9ce1" gracePeriod=30
Oct 30 01:45:48.833860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:48.833828    4425 generic.go:356] "Generic (PLEG): container finished" podID="45aca910-a167-41a3-8792-43502dd22283" containerID="0fcf5d10b55f07f02f684969a3430fd02602345202ba1f6110ac959f32cf9ce1" exitCode=0
Oct 30 01:45:48.834029 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:48.833895    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b" event={"ID":"45aca910-a167-41a3-8792-43502dd22283","Type":"ContainerDied","Data":"0fcf5d10b55f07f02f684969a3430fd02602345202ba1f6110ac959f32cf9ce1"}
Oct 30 01:45:48.927800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:48.927391    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:45:49.103178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.102407    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45aca910-a167-41a3-8792-43502dd22283-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45aca910-a167-41a3-8792-43502dd22283-openshift-service-ca\") pod \"45aca910-a167-41a3-8792-43502dd22283\" (UID: \"45aca910-a167-41a3-8792-43502dd22283\") "
Oct 30 01:45:49.103178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.102479    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45aca910-a167-41a3-8792-43502dd22283-kube-api-access-gxl66\" (UniqueName: \"kubernetes.io/projected/45aca910-a167-41a3-8792-43502dd22283-kube-api-access-gxl66\") pod \"45aca910-a167-41a3-8792-43502dd22283\" (UID: \"45aca910-a167-41a3-8792-43502dd22283\") "
Oct 30 01:45:49.103178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.102542    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45aca910-a167-41a3-8792-43502dd22283-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45aca910-a167-41a3-8792-43502dd22283-maas-api-tls\") pod \"45aca910-a167-41a3-8792-43502dd22283\" (UID: \"45aca910-a167-41a3-8792-43502dd22283\") "
Oct 30 01:45:49.103178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.102616    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45aca910-a167-41a3-8792-43502dd22283-metrics-certs\" (UniqueName: \"kubernetes.io/secret/45aca910-a167-41a3-8792-43502dd22283-metrics-certs\") pod \"45aca910-a167-41a3-8792-43502dd22283\" (UID: \"45aca910-a167-41a3-8792-43502dd22283\") "
Oct 30 01:45:49.106037 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.104729    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45aca910-a167-41a3-8792-43502dd22283-openshift-service-ca" pod "45aca910-a167-41a3-8792-43502dd22283" (UID: "45aca910-a167-41a3-8792-43502dd22283"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:49.106037 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.105946    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45aca910-a167-41a3-8792-43502dd22283-metrics-certs" pod "45aca910-a167-41a3-8792-43502dd22283" (UID: "45aca910-a167-41a3-8792-43502dd22283"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:45:49.122453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.122379    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45aca910-a167-41a3-8792-43502dd22283-kube-api-access-gxl66" pod "45aca910-a167-41a3-8792-43502dd22283" (UID: "45aca910-a167-41a3-8792-43502dd22283"). InnerVolumeSpecName "kube-api-access-gxl66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:49.123044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.123018    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45aca910-a167-41a3-8792-43502dd22283-maas-api-tls" pod "45aca910-a167-41a3-8792-43502dd22283" (UID: "45aca910-a167-41a3-8792-43502dd22283"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:45:49.208176 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.207214    4425 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45aca910-a167-41a3-8792-43502dd22283-openshift-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:49.208176 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.207249    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxl66\" (UniqueName: \"kubernetes.io/projected/45aca910-a167-41a3-8792-43502dd22283-kube-api-access-gxl66\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:49.208176 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.207281    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45aca910-a167-41a3-8792-43502dd22283-maas-api-tls\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:49.208176 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.207295    4425 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45aca910-a167-41a3-8792-43502dd22283-metrics-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:49.494263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.494078    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"]
Oct 30 01:45:49.536245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.536211    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"]
Oct 30 01:45:49.573177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.560610    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5457f2-v5tb4"]
Oct 30 01:45:49.844631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.844602    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"
Oct 30 01:45:49.845196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.844601    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b" event={"ID":"45aca910-a167-41a3-8792-43502dd22283","Type":"ContainerDied","Data":"28be4bf89009b942056aa6926f13ba904985ced0d6c7ccde3af0029555cf24d0"}
Oct 30 01:45:49.845196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.844747    4425 scope.go:122] "RemoveContainer" containerID="0fcf5d10b55f07f02f684969a3430fd02602345202ba1f6110ac959f32cf9ce1"
Oct 30 01:45:49.904244 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.904209    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"]
Oct 30 01:45:49.919915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:49.919880    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5457f2-5cf4f6466b-qls5b"]
Oct 30 01:45:50.071088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.071053    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"]
Oct 30 01:45:50.072304 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.072277    4425 cpu_manager.go:414] "containerMap: removing container" podUID="af779dae-521f-4f8a-9ad2-e7fe3948dcbc" containerName="cleanup"
Oct 30 01:45:50.072535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.072516    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af779dae-521f-4f8a-9ad2-e7fe3948dcbc" containerName="cleanup"
Oct 30 01:45:50.072687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.072674    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ee57e527-71ff-4a09-8db0-9840a6467fce" containerName="cleanup"
Oct 30 01:45:50.072785 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.072772    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee57e527-71ff-4a09-8db0-9840a6467fce" containerName="cleanup"
Oct 30 01:45:50.072878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.072866    4425 cpu_manager.go:414] "containerMap: removing container" podUID="45aca910-a167-41a3-8792-43502dd22283" containerName="maas-api"
Oct 30 01:45:50.072958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.072948    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45aca910-a167-41a3-8792-43502dd22283" containerName="maas-api"
Oct 30 01:45:50.073069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.073059    4425 cpu_manager.go:414] "containerMap: removing container" podUID="caccf6b1-b7c1-437e-a634-84d62cf8d3b4" containerName="cleanup"
Oct 30 01:45:50.073171 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.073144    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caccf6b1-b7c1-437e-a634-84d62cf8d3b4" containerName="cleanup"
Oct 30 01:45:50.073275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.073262    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9a82dd87-6039-44da-b690-297f5449c337" containerName="cleanup"
Oct 30 01:45:50.073376 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.073363    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a82dd87-6039-44da-b690-297f5449c337" containerName="cleanup"
Oct 30 01:45:50.073470 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.073457    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d44cf871-7767-4ad3-a9b4-e70a58a52ec9" containerName="cleanup"
Oct 30 01:45:50.073557 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.073545    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44cf871-7767-4ad3-a9b4-e70a58a52ec9" containerName="cleanup"
Oct 30 01:45:50.073664 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.073653    4425 cpu_manager.go:414] "containerMap: removing container" podUID="1e2b9d25-c7ed-451c-8b66-ec294e97b3e2" containerName="maas-api"
Oct 30 01:45:50.073749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.073738    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e2b9d25-c7ed-451c-8b66-ec294e97b3e2" containerName="maas-api"
Oct 30 01:45:50.073836 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.073825    4425 cpu_manager.go:414] "containerMap: removing container" podUID="6c1bab87-9a09-48f6-9302-43b0254810e5" containerName="cleanup"
Oct 30 01:45:50.073920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.073909    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c1bab87-9a09-48f6-9302-43b0254810e5" containerName="cleanup"
Oct 30 01:45:50.074014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.074004    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a2a87024-9da9-4842-afb2-063b447bd15a" containerName="cleanup"
Oct 30 01:45:50.074124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.074109    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a87024-9da9-4842-afb2-063b447bd15a" containerName="cleanup"
Oct 30 01:45:50.074258 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.074245    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8cd1b902-d5db-4613-90ea-30a92c1bf2b0" containerName="cleanup"
Oct 30 01:45:50.074349 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.074337    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cd1b902-d5db-4613-90ea-30a92c1bf2b0" containerName="cleanup"
Oct 30 01:45:50.074466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.074454    4425 cpu_manager.go:414] "containerMap: removing container" podUID="47cfb795-a866-454b-813c-b81eb4ca8271" containerName="revoke-keys"
Oct 30 01:45:50.074548 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.074536    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47cfb795-a866-454b-813c-b81eb4ca8271" containerName="revoke-keys"
Oct 30 01:45:50.074872 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.074855    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="45aca910-a167-41a3-8792-43502dd22283" containerName="maas-api"
Oct 30 01:45:50.074991 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.074979    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e2b9d25-c7ed-451c-8b66-ec294e97b3e2" containerName="maas-api"
Oct 30 01:45:50.075085 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.075074    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c1bab87-9a09-48f6-9302-43b0254810e5" containerName="cleanup"
Oct 30 01:45:50.075196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.075184    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a87024-9da9-4842-afb2-063b447bd15a" containerName="cleanup"
Oct 30 01:45:50.075302 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.075290    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="47cfb795-a866-454b-813c-b81eb4ca8271" containerName="revoke-keys"
Oct 30 01:45:50.075404 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.075384    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="336e3514-5287-4632-84f7-2673bc71a606" containerName="cleanup"
Oct 30 01:45:50.075504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.075493    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77b1d4c-dc08-416b-ad79-6e777474940c" containerName="cleanup"
Oct 30 01:45:50.075606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.075595    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a82dd87-6039-44da-b690-297f5449c337" containerName="cleanup"
Oct 30 01:45:50.075689 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.075679    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf3962c-f594-478e-89dc-a0ab9671bb9b" containerName="cleanup"
Oct 30 01:45:50.075773 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.075763    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="af779dae-521f-4f8a-9ad2-e7fe3948dcbc" containerName="cleanup"
Oct 30 01:45:50.075864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.075854    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="357853d9-748f-44af-baa0-523e64187284" containerName="cleanup"
Oct 30 01:45:50.075952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.075943    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="caccf6b1-b7c1-437e-a634-84d62cf8d3b4" containerName="cleanup"
Oct 30 01:45:50.082240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.082210    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"
Oct 30 01:45:50.093889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.093851    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b63fe40d-7k6sm\" (UID: \"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"
Oct 30 01:45:50.094197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.094176    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b63fe40d-7k6sm\" (UID: \"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"
Oct 30 01:45:50.094373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.094358    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6wtj\" (UniqueName: \"kubernetes.io/projected/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-kube-api-access-v6wtj\") pod \"maas-api-revoke-sub-b63fe40d-7k6sm\" (UID: \"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"
Oct 30 01:45:50.097594 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.097520    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:45:50.098523 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.098500    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:45:50.122588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.116141    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"]
Oct 30 01:45:50.228756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.221344    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b63fe40d-7k6sm\" (UID: \"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"
Oct 30 01:45:50.228756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.221677    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b63fe40d-7k6sm\" (UID: \"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"
Oct 30 01:45:50.232823 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.221773    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6wtj\" (UniqueName: \"kubernetes.io/projected/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-kube-api-access-v6wtj\") pod \"maas-api-revoke-sub-b63fe40d-7k6sm\" (UID: \"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"
Oct 30 01:45:50.248912 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.248850    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b63fe40d-7k6sm\" (UID: \"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"
Oct 30 01:45:50.415225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.407557    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b63fe40d-7k6sm\" (UID: \"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"
Oct 30 01:45:50.438178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.437644    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6wtj\" (UniqueName: \"kubernetes.io/projected/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-kube-api-access-v6wtj\") pod \"maas-api-revoke-sub-b63fe40d-7k6sm\" (UID: \"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"
Oct 30 01:45:50.545203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:50.542385    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"
Oct 30 01:45:51.018300 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:45:51.018238    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a4f68a_68f3_4bbe_a3fa_58eb92d9a294.slice/crio-1aa6c7b7e0c449f2b1183f2328576e7df9a2cd66b76e9d79a9212df339d455bc WatchSource:0}: Error finding container 1aa6c7b7e0c449f2b1183f2328576e7df9a2cd66b76e9d79a9212df339d455bc: Status 404 returned error can't find the container with id 1aa6c7b7e0c449f2b1183f2328576e7df9a2cd66b76e9d79a9212df339d455bc
Oct 30 01:45:51.021057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:51.020824    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"]
Oct 30 01:45:51.427315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:51.427282    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45aca910-a167-41a3-8792-43502dd22283" path="/var/lib/kubelet/pods/45aca910-a167-41a3-8792-43502dd22283/volumes"
Oct 30 01:45:51.427762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:51.427747    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492dec49-fde4-4b48-aa34-afccd3610dda" path="/var/lib/kubelet/pods/492dec49-fde4-4b48-aa34-afccd3610dda/volumes"
Oct 30 01:45:51.876827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:51.876736    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm" event={"ID":"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294","Type":"ContainerStarted","Data":"986e335a83caac437e4d0377cc1f9bb1a8ef2b628995aab5559748b07f7d16fd"}
Oct 30 01:45:51.877084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:51.877050    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm" event={"ID":"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294","Type":"ContainerStarted","Data":"1aa6c7b7e0c449f2b1183f2328576e7df9a2cd66b76e9d79a9212df339d455bc"}
Oct 30 01:45:51.926600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:51.926546    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm" podStartSLOduration=1.926529855 podStartE2EDuration="1.926529855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:51.925748692 +0000 UTC m=+3860.675404001" watchObservedRunningTime="2026-10-30 01:45:51.926529855 +0000 UTC m=+3860.676185546"
Oct 30 01:45:52.886364 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:52.886327    4425 generic.go:356] "Generic (PLEG): container finished" podID="c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294" containerID="986e335a83caac437e4d0377cc1f9bb1a8ef2b628995aab5559748b07f7d16fd" exitCode=0
Oct 30 01:45:52.886772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:52.886402    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm" event={"ID":"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294","Type":"ContainerDied","Data":"986e335a83caac437e4d0377cc1f9bb1a8ef2b628995aab5559748b07f7d16fd"}
Oct 30 01:45:54.058503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:54.058477    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"
Oct 30 01:45:54.157481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:54.157397    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-kube-api-access-v6wtj\" (UniqueName: \"kubernetes.io/projected/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-kube-api-access-v6wtj\") pod \"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294\" (UID: \"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294\") "
Oct 30 01:45:54.157668 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:54.157650    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-maas-api-openshift-ca\") pod \"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294\" (UID: \"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294\") "
Oct 30 01:45:54.157741 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:54.157715    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-maas-api-xks-ca\") pod \"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294\" (UID: \"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294\") "
Oct 30 01:45:54.158064 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:54.158037    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-maas-api-openshift-ca" pod "c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294" (UID: "c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:45:54.158317 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:54.158297    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:54.159514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:54.159494    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-kube-api-access-v6wtj" pod "c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294" (UID: "c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294"). InnerVolumeSpecName "kube-api-access-v6wtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:45:54.159688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:54.159666    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-maas-api-xks-ca" pod "c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294" (UID: "c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:45:54.259313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:54.259290    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:54.259313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:54.259312    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6wtj\" (UniqueName: \"kubernetes.io/projected/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294-kube-api-access-v6wtj\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:45:54.905751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:54.905709    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm" event={"ID":"c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294","Type":"ContainerDied","Data":"1aa6c7b7e0c449f2b1183f2328576e7df9a2cd66b76e9d79a9212df339d455bc"}
Oct 30 01:45:54.905751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:54.905747    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"
Oct 30 01:45:54.905973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:45:54.905762    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aa6c7b7e0c449f2b1183f2328576e7df9a2cd66b76e9d79a9212df339d455bc"
Oct 30 01:46:01.267133 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.267097    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"]
Oct 30 01:46:01.267812 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.267797    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294" containerName="revoke-keys"
Oct 30 01:46:01.267857 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.267813    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294" containerName="revoke-keys"
Oct 30 01:46:01.267958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.267949    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294" containerName="revoke-keys"
Oct 30 01:46:01.271778 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.271755    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"
Oct 30 01:46:01.286916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.286893    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:46:01.296774 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.296752    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:46:01.303246 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.303223    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"]
Oct 30 01:46:01.350076 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.350044    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36dd6581-a435-4715-a354-94959d69e3a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e48f727c-njsm2\" (UID: \"36dd6581-a435-4715-a354-94959d69e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"
Oct 30 01:46:01.350242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.350204    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36dd6581-a435-4715-a354-94959d69e3a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e48f727c-njsm2\" (UID: \"36dd6581-a435-4715-a354-94959d69e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"
Oct 30 01:46:01.350320 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.350307    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vcd2\" (UniqueName: \"kubernetes.io/projected/36dd6581-a435-4715-a354-94959d69e3a6-kube-api-access-9vcd2\") pod \"maas-api-revoke-sub-e48f727c-njsm2\" (UID: \"36dd6581-a435-4715-a354-94959d69e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"
Oct 30 01:46:01.451702 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.451672    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36dd6581-a435-4715-a354-94959d69e3a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e48f727c-njsm2\" (UID: \"36dd6581-a435-4715-a354-94959d69e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"
Oct 30 01:46:01.451883 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.451745    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vcd2\" (UniqueName: \"kubernetes.io/projected/36dd6581-a435-4715-a354-94959d69e3a6-kube-api-access-9vcd2\") pod \"maas-api-revoke-sub-e48f727c-njsm2\" (UID: \"36dd6581-a435-4715-a354-94959d69e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"
Oct 30 01:46:01.451883 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.451796    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36dd6581-a435-4715-a354-94959d69e3a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e48f727c-njsm2\" (UID: \"36dd6581-a435-4715-a354-94959d69e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"
Oct 30 01:46:01.452415 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.452389    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36dd6581-a435-4715-a354-94959d69e3a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e48f727c-njsm2\" (UID: \"36dd6581-a435-4715-a354-94959d69e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"
Oct 30 01:46:01.454093 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.454076    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36dd6581-a435-4715-a354-94959d69e3a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e48f727c-njsm2\" (UID: \"36dd6581-a435-4715-a354-94959d69e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"
Oct 30 01:46:01.472658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.472632    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vcd2\" (UniqueName: \"kubernetes.io/projected/36dd6581-a435-4715-a354-94959d69e3a6-kube-api-access-9vcd2\") pod \"maas-api-revoke-sub-e48f727c-njsm2\" (UID: \"36dd6581-a435-4715-a354-94959d69e3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"
Oct 30 01:46:01.586626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.586562    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"
Oct 30 01:46:01.894320 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:46:01.894283    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36dd6581_a435_4715_a354_94959d69e3a6.slice/crio-29ea20b86cb369e26fbcea27f1c3aa4f6b6f98fa12f8de1b18d0c091a11be3f7 WatchSource:0}: Error finding container 29ea20b86cb369e26fbcea27f1c3aa4f6b6f98fa12f8de1b18d0c091a11be3f7: Status 404 returned error can't find the container with id 29ea20b86cb369e26fbcea27f1c3aa4f6b6f98fa12f8de1b18d0c091a11be3f7
Oct 30 01:46:01.894483 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.894328    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"]
Oct 30 01:46:01.959018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:01.958984    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2" event={"ID":"36dd6581-a435-4715-a354-94959d69e3a6","Type":"ContainerStarted","Data":"29ea20b86cb369e26fbcea27f1c3aa4f6b6f98fa12f8de1b18d0c091a11be3f7"}
Oct 30 01:46:02.970521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:02.970488    4425 generic.go:356] "Generic (PLEG): container finished" podID="36dd6581-a435-4715-a354-94959d69e3a6" containerID="bdb67249427b4ded225c6defe86b76ee70702296882f40360f5d29af99a74316" exitCode=0
Oct 30 01:46:02.971122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:02.970639    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2" event={"ID":"36dd6581-a435-4715-a354-94959d69e3a6","Type":"ContainerDied","Data":"bdb67249427b4ded225c6defe86b76ee70702296882f40360f5d29af99a74316"}
Oct 30 01:46:04.172115 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.172088    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"
Oct 30 01:46:04.309417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.309386    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36dd6581-a435-4715-a354-94959d69e3a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36dd6581-a435-4715-a354-94959d69e3a6-maas-api-xks-ca\") pod \"36dd6581-a435-4715-a354-94959d69e3a6\" (UID: \"36dd6581-a435-4715-a354-94959d69e3a6\") "
Oct 30 01:46:04.309603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.309524    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36dd6581-a435-4715-a354-94959d69e3a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36dd6581-a435-4715-a354-94959d69e3a6-maas-api-openshift-ca\") pod \"36dd6581-a435-4715-a354-94959d69e3a6\" (UID: \"36dd6581-a435-4715-a354-94959d69e3a6\") "
Oct 30 01:46:04.309603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.309586    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36dd6581-a435-4715-a354-94959d69e3a6-kube-api-access-9vcd2\" (UniqueName: \"kubernetes.io/projected/36dd6581-a435-4715-a354-94959d69e3a6-kube-api-access-9vcd2\") pod \"36dd6581-a435-4715-a354-94959d69e3a6\" (UID: \"36dd6581-a435-4715-a354-94959d69e3a6\") "
Oct 30 01:46:04.309895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.309871    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36dd6581-a435-4715-a354-94959d69e3a6-maas-api-openshift-ca" pod "36dd6581-a435-4715-a354-94959d69e3a6" (UID: "36dd6581-a435-4715-a354-94959d69e3a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:46:04.310023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.310006    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36dd6581-a435-4715-a354-94959d69e3a6-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:04.311521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.311495    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36dd6581-a435-4715-a354-94959d69e3a6-maas-api-xks-ca" pod "36dd6581-a435-4715-a354-94959d69e3a6" (UID: "36dd6581-a435-4715-a354-94959d69e3a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:46:04.311626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.311554    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36dd6581-a435-4715-a354-94959d69e3a6-kube-api-access-9vcd2" pod "36dd6581-a435-4715-a354-94959d69e3a6" (UID: "36dd6581-a435-4715-a354-94959d69e3a6"). InnerVolumeSpecName "kube-api-access-9vcd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:46:04.411172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.411122    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vcd2\" (UniqueName: \"kubernetes.io/projected/36dd6581-a435-4715-a354-94959d69e3a6-kube-api-access-9vcd2\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:04.411172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.411167    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36dd6581-a435-4715-a354-94959d69e3a6-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:04.601418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.601325    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v" podUID="fc7d715c-4856-4653-bc23-e7479bb24f9a" containerName="main" containerID="cri-o://800d78103f5c9d8d09c245d509f5078948edb4632caaa9896007f8f0b6036f01" gracePeriod=45
Oct 30 01:46:04.710432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.710396    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v" podUID="fc7d715c-4856-4653-bc23-e7479bb24f9a" containerName="main" probeResult="failure" output="Get \"https://10.133.1.92:8000/ready\": dial tcp 10.133.1.92:8000: connect: connection refused"
Oct 30 01:46:04.858651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.858580    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:46:04.987966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.987935    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"
Oct 30 01:46:04.988146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.987969    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2" event={"ID":"36dd6581-a435-4715-a354-94959d69e3a6","Type":"ContainerDied","Data":"29ea20b86cb369e26fbcea27f1c3aa4f6b6f98fa12f8de1b18d0c091a11be3f7"}
Oct 30 01:46:04.988146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.988014    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29ea20b86cb369e26fbcea27f1c3aa4f6b6f98fa12f8de1b18d0c091a11be3f7"
Oct 30 01:46:04.989565 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.989533    4425 generic.go:356] "Generic (PLEG): container finished" podID="fc7d715c-4856-4653-bc23-e7479bb24f9a" containerID="800d78103f5c9d8d09c245d509f5078948edb4632caaa9896007f8f0b6036f01" exitCode=0
Oct 30 01:46:04.989704 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.989586    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"
Oct 30 01:46:04.989704 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.989608    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v" event={"ID":"fc7d715c-4856-4653-bc23-e7479bb24f9a","Type":"ContainerDied","Data":"800d78103f5c9d8d09c245d509f5078948edb4632caaa9896007f8f0b6036f01"}
Oct 30 01:46:04.989704 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.989637    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v" event={"ID":"fc7d715c-4856-4653-bc23-e7479bb24f9a","Type":"ContainerDied","Data":"d0c3821547bce83165a4723dcb5e642b76d87e55ed161fd4f7e45aeef3db8c7d"}
Oct 30 01:46:04.989704 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:04.989659    4425 scope.go:122] "RemoveContainer" containerID="800d78103f5c9d8d09c245d509f5078948edb4632caaa9896007f8f0b6036f01"
Oct 30 01:46:05.000282 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.000266    4425 scope.go:122] "RemoveContainer" containerID="800d78103f5c9d8d09c245d509f5078948edb4632caaa9896007f8f0b6036f01"
Oct 30 01:46:05.000555 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:46:05.000537    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800d78103f5c9d8d09c245d509f5078948edb4632caaa9896007f8f0b6036f01\": container with ID starting with 800d78103f5c9d8d09c245d509f5078948edb4632caaa9896007f8f0b6036f01 not found: ID does not exist" containerID="800d78103f5c9d8d09c245d509f5078948edb4632caaa9896007f8f0b6036f01"
Oct 30 01:46:05.000628 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.000561    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800d78103f5c9d8d09c245d509f5078948edb4632caaa9896007f8f0b6036f01"} err="failed to get container status \"800d78103f5c9d8d09c245d509f5078948edb4632caaa9896007f8f0b6036f01\": rpc error: code = NotFound desc = could not find container \"800d78103f5c9d8d09c245d509f5078948edb4632caaa9896007f8f0b6036f01\": container with ID starting with 800d78103f5c9d8d09c245d509f5078948edb4632caaa9896007f8f0b6036f01 not found: ID does not exist"
Oct 30 01:46:05.020665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.020635    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-model-cache\") pod \"fc7d715c-4856-4653-bc23-e7479bb24f9a\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") "
Oct 30 01:46:05.020810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.020794    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc7d715c-4856-4653-bc23-e7479bb24f9a-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc7d715c-4856-4653-bc23-e7479bb24f9a-tls-certs\") pod \"fc7d715c-4856-4653-bc23-e7479bb24f9a\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") "
Oct 30 01:46:05.020864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.020827    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-tmp-dir\") pod \"fc7d715c-4856-4653-bc23-e7479bb24f9a\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") "
Oct 30 01:46:05.020918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.020904    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-model-cache" pod "fc7d715c-4856-4653-bc23-e7479bb24f9a" (UID: "fc7d715c-4856-4653-bc23-e7479bb24f9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:05.020960 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.020922    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-home\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-home\") pod \"fc7d715c-4856-4653-bc23-e7479bb24f9a\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") "
Oct 30 01:46:05.021005 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.020971    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-dshm\") pod \"fc7d715c-4856-4653-bc23-e7479bb24f9a\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") "
Oct 30 01:46:05.021069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.021042    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc7d715c-4856-4653-bc23-e7479bb24f9a-kube-api-access-7wv4s\" (UniqueName: \"kubernetes.io/projected/fc7d715c-4856-4653-bc23-e7479bb24f9a-kube-api-access-7wv4s\") pod \"fc7d715c-4856-4653-bc23-e7479bb24f9a\" (UID: \"fc7d715c-4856-4653-bc23-e7479bb24f9a\") "
Oct 30 01:46:05.021150 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.021067    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-tmp-dir" pod "fc7d715c-4856-4653-bc23-e7479bb24f9a" (UID: "fc7d715c-4856-4653-bc23-e7479bb24f9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:05.021265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.021209    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-home" pod "fc7d715c-4856-4653-bc23-e7479bb24f9a" (UID: "fc7d715c-4856-4653-bc23-e7479bb24f9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:05.021672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.021652    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:05.021770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.021676    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:05.021770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.021691    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:05.023309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.023283    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc7d715c-4856-4653-bc23-e7479bb24f9a-tls-certs" pod "fc7d715c-4856-4653-bc23-e7479bb24f9a" (UID: "fc7d715c-4856-4653-bc23-e7479bb24f9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:46:05.023389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.023287    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-dshm" pod "fc7d715c-4856-4653-bc23-e7479bb24f9a" (UID: "fc7d715c-4856-4653-bc23-e7479bb24f9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:05.023389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.023381    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7d715c-4856-4653-bc23-e7479bb24f9a-kube-api-access-7wv4s" pod "fc7d715c-4856-4653-bc23-e7479bb24f9a" (UID: "fc7d715c-4856-4653-bc23-e7479bb24f9a"). InnerVolumeSpecName "kube-api-access-7wv4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:46:05.123029 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.123008    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc7d715c-4856-4653-bc23-e7479bb24f9a-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:05.123139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.123032    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wv4s\" (UniqueName: \"kubernetes.io/projected/fc7d715c-4856-4653-bc23-e7479bb24f9a-kube-api-access-7wv4s\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:05.123139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.123042    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc7d715c-4856-4653-bc23-e7479bb24f9a-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:05.334043 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.334009    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"]
Oct 30 01:46:05.337782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.337758    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6aa9aa/test-model-6aa9aa-kserve-5b7c4485d9-q4g2v"]
Oct 30 01:46:05.424769 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:05.424706    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc7d715c-4856-4653-bc23-e7479bb24f9a" path="/var/lib/kubelet/pods/fc7d715c-4856-4653-bc23-e7479bb24f9a/volumes"
Oct 30 01:46:07.555260 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:46:07.555228    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:46:11.289719 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.289661    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"]
Oct 30 01:46:11.545967 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.545822    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"]
Oct 30 01:46:11.549822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.546571    4425 cpu_manager.go:414] "containerMap: removing container" podUID="36dd6581-a435-4715-a354-94959d69e3a6" containerName="revoke-keys"
Oct 30 01:46:11.549822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.546586    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36dd6581-a435-4715-a354-94959d69e3a6" containerName="revoke-keys"
Oct 30 01:46:11.549822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.546601    4425 cpu_manager.go:414] "containerMap: removing container" podUID="fc7d715c-4856-4653-bc23-e7479bb24f9a" containerName="main"
Oct 30 01:46:11.549822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.546606    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc7d715c-4856-4653-bc23-e7479bb24f9a" containerName="main"
Oct 30 01:46:11.549822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.546713    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="36dd6581-a435-4715-a354-94959d69e3a6" containerName="revoke-keys"
Oct 30 01:46:11.549822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.546722    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc7d715c-4856-4653-bc23-e7479bb24f9a" containerName="main"
Oct 30 01:46:11.550779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.550763    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"
Oct 30 01:46:11.570688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.570662    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:46:11.570688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.570685    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:46:11.577137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.577113    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"]
Oct 30 01:46:11.599280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.599244    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8995d47f-a3b1-4f6a-a759-d4abceffcb09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth\" (UID: \"8995d47f-a3b1-4f6a-a759-d4abceffcb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"
Oct 30 01:46:11.599411 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.599293    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8995d47f-a3b1-4f6a-a759-d4abceffcb09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth\" (UID: \"8995d47f-a3b1-4f6a-a759-d4abceffcb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"
Oct 30 01:46:11.599459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.599400    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b6d4\" (UniqueName: \"kubernetes.io/projected/8995d47f-a3b1-4f6a-a759-d4abceffcb09-kube-api-access-8b6d4\") pod \"maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth\" (UID: \"8995d47f-a3b1-4f6a-a759-d4abceffcb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"
Oct 30 01:46:11.700884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.700846    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b6d4\" (UniqueName: \"kubernetes.io/projected/8995d47f-a3b1-4f6a-a759-d4abceffcb09-kube-api-access-8b6d4\") pod \"maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth\" (UID: \"8995d47f-a3b1-4f6a-a759-d4abceffcb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"
Oct 30 01:46:11.701138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.701117    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8995d47f-a3b1-4f6a-a759-d4abceffcb09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth\" (UID: \"8995d47f-a3b1-4f6a-a759-d4abceffcb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"
Oct 30 01:46:11.701248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.701169    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8995d47f-a3b1-4f6a-a759-d4abceffcb09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth\" (UID: \"8995d47f-a3b1-4f6a-a759-d4abceffcb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"
Oct 30 01:46:11.702049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.702018    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8995d47f-a3b1-4f6a-a759-d4abceffcb09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth\" (UID: \"8995d47f-a3b1-4f6a-a759-d4abceffcb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"
Oct 30 01:46:11.703630 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.703609    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8995d47f-a3b1-4f6a-a759-d4abceffcb09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth\" (UID: \"8995d47f-a3b1-4f6a-a759-d4abceffcb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"
Oct 30 01:46:11.722555 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.722529    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b6d4\" (UniqueName: \"kubernetes.io/projected/8995d47f-a3b1-4f6a-a759-d4abceffcb09-kube-api-access-8b6d4\") pod \"maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth\" (UID: \"8995d47f-a3b1-4f6a-a759-d4abceffcb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"
Oct 30 01:46:11.866509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:11.866424    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"
Oct 30 01:46:12.191821 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:12.191793    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"]
Oct 30 01:46:12.192282 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:46:12.192251    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8995d47f_a3b1_4f6a_a759_d4abceffcb09.slice/crio-a20ea0a3be6c72e7e51004b783bc33a1ecc291985810d20e8208b9cd8b8e1d76 WatchSource:0}: Error finding container a20ea0a3be6c72e7e51004b783bc33a1ecc291985810d20e8208b9cd8b8e1d76: Status 404 returned error can't find the container with id a20ea0a3be6c72e7e51004b783bc33a1ecc291985810d20e8208b9cd8b8e1d76
Oct 30 01:46:13.057816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:13.057780    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth" event={"ID":"8995d47f-a3b1-4f6a-a759-d4abceffcb09","Type":"ContainerStarted","Data":"0bafa7f2ccf0330d40868e8c0fbd1becb609eea6ffd4f0d5e0c3c50767df4c42"}
Oct 30 01:46:13.058280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:13.057840    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth" event={"ID":"8995d47f-a3b1-4f6a-a759-d4abceffcb09","Type":"ContainerStarted","Data":"a20ea0a3be6c72e7e51004b783bc33a1ecc291985810d20e8208b9cd8b8e1d76"}
Oct 30 01:46:13.113935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:13.113891    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth" podStartSLOduration=2.113877932 podStartE2EDuration="2.113877932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:13.109858939 +0000 UTC m=+3881.859514251" watchObservedRunningTime="2026-10-30 01:46:13.113877932 +0000 UTC m=+3881.863533240"
Oct 30 01:46:13.677799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:13.677762    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:46:14.077526 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.077489    4425 generic.go:356] "Generic (PLEG): container finished" podID="8995d47f-a3b1-4f6a-a759-d4abceffcb09" containerID="0bafa7f2ccf0330d40868e8c0fbd1becb609eea6ffd4f0d5e0c3c50767df4c42" exitCode=0
Oct 30 01:46:14.078003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.077521    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth" event={"ID":"8995d47f-a3b1-4f6a-a759-d4abceffcb09","Type":"ContainerDied","Data":"0bafa7f2ccf0330d40868e8c0fbd1becb609eea6ffd4f0d5e0c3c50767df4c42"}
Oct 30 01:46:14.477776 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.477691    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"]
Oct 30 01:46:14.482544 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.482526    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"
Oct 30 01:46:14.504837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.504818    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e6fc524e\""
Oct 30 01:46:14.517541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.517517    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"]
Oct 30 01:46:14.536668 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.536643    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbrx2\" (UniqueName: \"kubernetes.io/projected/30d24304-fb77-479a-9a11-06cc577c4382-kube-api-access-pbrx2\") pod \"payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l\" (UID: \"30d24304-fb77-479a-9a11-06cc577c4382\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"
Oct 30 01:46:14.536791 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.536753    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d24304-fb77-479a-9a11-06cc577c4382-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l\" (UID: \"30d24304-fb77-479a-9a11-06cc577c4382\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"
Oct 30 01:46:14.637745 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.637708    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d24304-fb77-479a-9a11-06cc577c4382-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l\" (UID: \"30d24304-fb77-479a-9a11-06cc577c4382\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"
Oct 30 01:46:14.637999 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.637980    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbrx2\" (UniqueName: \"kubernetes.io/projected/30d24304-fb77-479a-9a11-06cc577c4382-kube-api-access-pbrx2\") pod \"payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l\" (UID: \"30d24304-fb77-479a-9a11-06cc577c4382\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"
Oct 30 01:46:14.638486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.638461    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d24304-fb77-479a-9a11-06cc577c4382-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l\" (UID: \"30d24304-fb77-479a-9a11-06cc577c4382\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"
Oct 30 01:46:14.662737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.662704    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbrx2\" (UniqueName: \"kubernetes.io/projected/30d24304-fb77-479a-9a11-06cc577c4382-kube-api-access-pbrx2\") pod \"payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l\" (UID: \"30d24304-fb77-479a-9a11-06cc577c4382\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"
Oct 30 01:46:14.797796 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.797764    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"
Oct 30 01:46:14.810338 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.810305    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"]
Oct 30 01:46:14.816001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.815981    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"
Oct 30 01:46:14.837779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.837755    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e6fc524e-dockercfg-5l2z6\""
Oct 30 01:46:14.840504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.840472    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb4td\" (UniqueName: \"kubernetes.io/projected/19f75c87-9928-4901-8e16-18086f0760e2-kube-api-access-kb4td\") pod \"payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d\" (UID: \"19f75c87-9928-4901-8e16-18086f0760e2\") " pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"
Oct 30 01:46:14.840596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.840579    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19f75c87-9928-4901-8e16-18086f0760e2-plugins-config-volume\") pod \"payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d\" (UID: \"19f75c87-9928-4901-8e16-18086f0760e2\") " pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"
Oct 30 01:46:14.848194 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.847956    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"]
Oct 30 01:46:14.941906 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.941871    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb4td\" (UniqueName: \"kubernetes.io/projected/19f75c87-9928-4901-8e16-18086f0760e2-kube-api-access-kb4td\") pod \"payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d\" (UID: \"19f75c87-9928-4901-8e16-18086f0760e2\") " pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"
Oct 30 01:46:14.942070 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.941982    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19f75c87-9928-4901-8e16-18086f0760e2-plugins-config-volume\") pod \"payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d\" (UID: \"19f75c87-9928-4901-8e16-18086f0760e2\") " pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"
Oct 30 01:46:14.942676 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.942656    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19f75c87-9928-4901-8e16-18086f0760e2-plugins-config-volume\") pod \"payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d\" (UID: \"19f75c87-9928-4901-8e16-18086f0760e2\") " pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"
Oct 30 01:46:14.964895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:14.964872    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb4td\" (UniqueName: \"kubernetes.io/projected/19f75c87-9928-4901-8e16-18086f0760e2-kube-api-access-kb4td\") pod \"payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d\" (UID: \"19f75c87-9928-4901-8e16-18086f0760e2\") " pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"
Oct 30 01:46:15.041080 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:15.041050    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"]
Oct 30 01:46:15.048482 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:15.048421    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4611660-fh9hc"]
Oct 30 01:46:15.121987 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:15.121956    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"]
Oct 30 01:46:15.149493 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:15.149469    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"
Oct 30 01:46:15.234145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:15.234111    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"
Oct 30 01:46:15.353172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:15.347934    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8995d47f-a3b1-4f6a-a759-d4abceffcb09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8995d47f-a3b1-4f6a-a759-d4abceffcb09-maas-api-openshift-ca\") pod \"8995d47f-a3b1-4f6a-a759-d4abceffcb09\" (UID: \"8995d47f-a3b1-4f6a-a759-d4abceffcb09\") "
Oct 30 01:46:15.353172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:15.348020    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8995d47f-a3b1-4f6a-a759-d4abceffcb09-kube-api-access-8b6d4\" (UniqueName: \"kubernetes.io/projected/8995d47f-a3b1-4f6a-a759-d4abceffcb09-kube-api-access-8b6d4\") pod \"8995d47f-a3b1-4f6a-a759-d4abceffcb09\" (UID: \"8995d47f-a3b1-4f6a-a759-d4abceffcb09\") "
Oct 30 01:46:15.353172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:15.348072    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8995d47f-a3b1-4f6a-a759-d4abceffcb09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8995d47f-a3b1-4f6a-a759-d4abceffcb09-maas-api-xks-ca\") pod \"8995d47f-a3b1-4f6a-a759-d4abceffcb09\" (UID: \"8995d47f-a3b1-4f6a-a759-d4abceffcb09\") "
Oct 30 01:46:15.354069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:15.354039    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8995d47f-a3b1-4f6a-a759-d4abceffcb09-maas-api-openshift-ca" pod "8995d47f-a3b1-4f6a-a759-d4abceffcb09" (UID: "8995d47f-a3b1-4f6a-a759-d4abceffcb09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:46:15.361660 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:15.361451    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8995d47f-a3b1-4f6a-a759-d4abceffcb09-maas-api-xks-ca" pod "8995d47f-a3b1-4f6a-a759-d4abceffcb09" (UID: "8995d47f-a3b1-4f6a-a759-d4abceffcb09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:46:15.367285 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:15.367253    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8995d47f-a3b1-4f6a-a759-d4abceffcb09-kube-api-access-8b6d4" pod "8995d47f-a3b1-4f6a-a759-d4abceffcb09" (UID: "8995d47f-a3b1-4f6a-a759-d4abceffcb09"). InnerVolumeSpecName "kube-api-access-8b6d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:46:15.431175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:15.430798    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f6bd158-e648-47bb-867d-4435ba723c72" path="/var/lib/kubelet/pods/3f6bd158-e648-47bb-867d-4435ba723c72/volumes"
Oct 30 01:46:15.450103 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:15.450074    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8995d47f-a3b1-4f6a-a759-d4abceffcb09-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:15.450317 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:15.450212    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b6d4\" (UniqueName: \"kubernetes.io/projected/8995d47f-a3b1-4f6a-a759-d4abceffcb09-kube-api-access-8b6d4\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:15.450317 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:15.450236    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8995d47f-a3b1-4f6a-a759-d4abceffcb09-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:15.496002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:15.495974    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"]
Oct 30 01:46:15.497205 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:46:15.497137    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f75c87_9928_4901_8e16_18086f0760e2.slice/crio-60dc4e68b6466e65e3860d84382d096c5d96a54adbba387f604bfca5ede2b1e3 WatchSource:0}: Error finding container 60dc4e68b6466e65e3860d84382d096c5d96a54adbba387f604bfca5ede2b1e3: Status 404 returned error can't find the container with id 60dc4e68b6466e65e3860d84382d096c5d96a54adbba387f604bfca5ede2b1e3
Oct 30 01:46:16.097568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:16.097528    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l" event={"ID":"30d24304-fb77-479a-9a11-06cc577c4382","Type":"ContainerStarted","Data":"d00723cc413edeecb7b1e297e3119b2e03acc056dedbebda17df325c8ba3cda4"}
Oct 30 01:46:16.097782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:16.097588    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l" event={"ID":"30d24304-fb77-479a-9a11-06cc577c4382","Type":"ContainerStarted","Data":"b93d51f90a4a3a7cbb3f70e74c6cfa5b480d25a21356cfdef1b22a67c43a3f9f"}
Oct 30 01:46:16.097937 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:16.097821    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"
Oct 30 01:46:16.100556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:16.100525    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth" event={"ID":"8995d47f-a3b1-4f6a-a759-d4abceffcb09","Type":"ContainerDied","Data":"a20ea0a3be6c72e7e51004b783bc33a1ecc291985810d20e8208b9cd8b8e1d76"}
Oct 30 01:46:16.100701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:16.100576    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a20ea0a3be6c72e7e51004b783bc33a1ecc291985810d20e8208b9cd8b8e1d76"
Oct 30 01:46:16.100701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:16.100642    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"
Oct 30 01:46:16.104623 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:16.104599    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d" event={"ID":"19f75c87-9928-4901-8e16-18086f0760e2","Type":"ContainerStarted","Data":"7b5cda28518cfd15095810a6295650bf8aeb836d27d353396fecfefc35becd6c"}
Oct 30 01:46:16.104746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:16.104644    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d" event={"ID":"19f75c87-9928-4901-8e16-18086f0760e2","Type":"ContainerStarted","Data":"60dc4e68b6466e65e3860d84382d096c5d96a54adbba387f604bfca5ede2b1e3"}
Oct 30 01:46:16.104746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:16.104676    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"
Oct 30 01:46:16.155048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:16.154988    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l" podStartSLOduration=2.154970756 podStartE2EDuration="2.154970756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:16.152813254 +0000 UTC m=+3884.902468567" watchObservedRunningTime="2026-10-30 01:46:16.154970756 +0000 UTC m=+3884.904626065"
Oct 30 01:46:16.209918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:16.208907    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d" podStartSLOduration=2.208886998 podStartE2EDuration="2.208886998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:16.205588494 +0000 UTC m=+3884.955243805" watchObservedRunningTime="2026-10-30 01:46:16.208886998 +0000 UTC m=+3884.958542308"
Oct 30 01:46:17.892410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:17.891853    4425 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cxntb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:46:17.892410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:17.891934    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cxntb" podUID="50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:46:26.314781 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.314738    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c" podUID="e560b5a5-44db-4f11-ba85-30220486eb66" containerName="main" containerID="cri-o://56da17c33394d79c73935b6e543c53e2ca4b69623be2b0b7c6b9963379866845" gracePeriod=45
Oct 30 01:46:26.589041 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.589018    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:46:26.751220 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.751189    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-model-cache\") pod \"e560b5a5-44db-4f11-ba85-30220486eb66\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") "
Oct 30 01:46:26.751420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.751227    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-home\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-home\") pod \"e560b5a5-44db-4f11-ba85-30220486eb66\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") "
Oct 30 01:46:26.751499 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.751415    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-dshm\") pod \"e560b5a5-44db-4f11-ba85-30220486eb66\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") "
Oct 30 01:46:26.751499 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.751463    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-model-cache" pod "e560b5a5-44db-4f11-ba85-30220486eb66" (UID: "e560b5a5-44db-4f11-ba85-30220486eb66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:26.751611 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.751519    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e560b5a5-44db-4f11-ba85-30220486eb66-kube-api-access-g7gcx\" (UniqueName: \"kubernetes.io/projected/e560b5a5-44db-4f11-ba85-30220486eb66-kube-api-access-g7gcx\") pod \"e560b5a5-44db-4f11-ba85-30220486eb66\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") "
Oct 30 01:46:26.751611 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.751537    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-home" pod "e560b5a5-44db-4f11-ba85-30220486eb66" (UID: "e560b5a5-44db-4f11-ba85-30220486eb66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:26.751611 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.751568    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-tmp-dir\") pod \"e560b5a5-44db-4f11-ba85-30220486eb66\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") "
Oct 30 01:46:26.751758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.751625    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e560b5a5-44db-4f11-ba85-30220486eb66-tls-certs\" (UniqueName: \"kubernetes.io/secret/e560b5a5-44db-4f11-ba85-30220486eb66-tls-certs\") pod \"e560b5a5-44db-4f11-ba85-30220486eb66\" (UID: \"e560b5a5-44db-4f11-ba85-30220486eb66\") "
Oct 30 01:46:26.751832 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.751806    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-tmp-dir" pod "e560b5a5-44db-4f11-ba85-30220486eb66" (UID: "e560b5a5-44db-4f11-ba85-30220486eb66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:26.752114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.752093    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:26.752114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.752113    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:26.752374 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.752121    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:26.753642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.753614    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-dshm" pod "e560b5a5-44db-4f11-ba85-30220486eb66" (UID: "e560b5a5-44db-4f11-ba85-30220486eb66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:46:26.753741 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.753659    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e560b5a5-44db-4f11-ba85-30220486eb66-tls-certs" pod "e560b5a5-44db-4f11-ba85-30220486eb66" (UID: "e560b5a5-44db-4f11-ba85-30220486eb66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:46:26.753787 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.753741    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e560b5a5-44db-4f11-ba85-30220486eb66-kube-api-access-g7gcx" pod "e560b5a5-44db-4f11-ba85-30220486eb66" (UID: "e560b5a5-44db-4f11-ba85-30220486eb66"). InnerVolumeSpecName "kube-api-access-g7gcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:46:26.853422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.853362    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e560b5a5-44db-4f11-ba85-30220486eb66-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:26.853422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.853385    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7gcx\" (UniqueName: \"kubernetes.io/projected/e560b5a5-44db-4f11-ba85-30220486eb66-kube-api-access-g7gcx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:26.853422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:26.853405    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e560b5a5-44db-4f11-ba85-30220486eb66-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:27.173878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:27.173806    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"
Oct 30 01:46:27.173878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:27.173879    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"
Oct 30 01:46:27.241851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:27.241818    4425 generic.go:356] "Generic (PLEG): container finished" podID="e560b5a5-44db-4f11-ba85-30220486eb66" containerID="56da17c33394d79c73935b6e543c53e2ca4b69623be2b0b7c6b9963379866845" exitCode=0
Oct 30 01:46:27.242039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:27.241865    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"
Oct 30 01:46:27.242039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:27.241902    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c" event={"ID":"e560b5a5-44db-4f11-ba85-30220486eb66","Type":"ContainerDied","Data":"56da17c33394d79c73935b6e543c53e2ca4b69623be2b0b7c6b9963379866845"}
Oct 30 01:46:27.242039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:27.241948    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c" event={"ID":"e560b5a5-44db-4f11-ba85-30220486eb66","Type":"ContainerDied","Data":"39c486ec34b2a24ed6e2a5fbb6bf35274111fb5bf2ae639bf713986a53799ce1"}
Oct 30 01:46:27.242039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:27.241975    4425 scope.go:122] "RemoveContainer" containerID="56da17c33394d79c73935b6e543c53e2ca4b69623be2b0b7c6b9963379866845"
Oct 30 01:46:27.260452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:27.260423    4425 scope.go:122] "RemoveContainer" containerID="56da17c33394d79c73935b6e543c53e2ca4b69623be2b0b7c6b9963379866845"
Oct 30 01:46:27.265114 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:46:27.265080    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56da17c33394d79c73935b6e543c53e2ca4b69623be2b0b7c6b9963379866845\": container with ID starting with 56da17c33394d79c73935b6e543c53e2ca4b69623be2b0b7c6b9963379866845 not found: ID does not exist" containerID="56da17c33394d79c73935b6e543c53e2ca4b69623be2b0b7c6b9963379866845"
Oct 30 01:46:27.265695 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:27.265124    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56da17c33394d79c73935b6e543c53e2ca4b69623be2b0b7c6b9963379866845"} err="failed to get container status \"56da17c33394d79c73935b6e543c53e2ca4b69623be2b0b7c6b9963379866845\": rpc error: code = NotFound desc = could not find container \"56da17c33394d79c73935b6e543c53e2ca4b69623be2b0b7c6b9963379866845\": container with ID starting with 56da17c33394d79c73935b6e543c53e2ca4b69623be2b0b7c6b9963379866845 not found: ID does not exist"
Oct 30 01:46:27.302374 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:27.302338    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"]
Oct 30 01:46:27.307356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:27.307322    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1a4b17/test-model-1a4b17-kserve-54d78d64c-2jx7c"]
Oct 30 01:46:27.424510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:27.424433    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e560b5a5-44db-4f11-ba85-30220486eb66" path="/var/lib/kubelet/pods/e560b5a5-44db-4f11-ba85-30220486eb66/volumes"
Oct 30 01:46:28.073041 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:28.073009    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"]
Oct 30 01:46:28.083579 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:28.083550    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44621952-xgh4t"]
Oct 30 01:46:29.424862 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:29.424831    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2113693f-bc34-42c1-8fea-1dc510220472" path="/var/lib/kubelet/pods/2113693f-bc34-42c1-8fea-1dc510220472/volumes"
Oct 30 01:46:31.230422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:31.230386    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"]
Oct 30 01:46:31.230921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:31.230651    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9" podUID="24025a00-4226-4ea7-9c91-7d5805cec35c" containerName="maas-api" containerID="cri-o://ce8d34b7a9eb4d45f17e8de1cbcfff8bcc9169408eb58d52b08465ffe58ff600" gracePeriod=30
Oct 30 01:46:31.737954 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:31.737916    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:46:31.915597 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:31.915568    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24025a00-4226-4ea7-9c91-7d5805cec35c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/24025a00-4226-4ea7-9c91-7d5805cec35c-metrics-certs\") pod \"24025a00-4226-4ea7-9c91-7d5805cec35c\" (UID: \"24025a00-4226-4ea7-9c91-7d5805cec35c\") "
Oct 30 01:46:31.915772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:31.915640    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24025a00-4226-4ea7-9c91-7d5805cec35c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24025a00-4226-4ea7-9c91-7d5805cec35c-openshift-service-ca\") pod \"24025a00-4226-4ea7-9c91-7d5805cec35c\" (UID: \"24025a00-4226-4ea7-9c91-7d5805cec35c\") "
Oct 30 01:46:31.915772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:31.915672    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24025a00-4226-4ea7-9c91-7d5805cec35c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24025a00-4226-4ea7-9c91-7d5805cec35c-maas-api-tls\") pod \"24025a00-4226-4ea7-9c91-7d5805cec35c\" (UID: \"24025a00-4226-4ea7-9c91-7d5805cec35c\") "
Oct 30 01:46:31.915772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:31.915709    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24025a00-4226-4ea7-9c91-7d5805cec35c-kube-api-access-xdlwd\" (UniqueName: \"kubernetes.io/projected/24025a00-4226-4ea7-9c91-7d5805cec35c-kube-api-access-xdlwd\") pod \"24025a00-4226-4ea7-9c91-7d5805cec35c\" (UID: \"24025a00-4226-4ea7-9c91-7d5805cec35c\") "
Oct 30 01:46:31.916041 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:31.916012    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24025a00-4226-4ea7-9c91-7d5805cec35c-openshift-service-ca" pod "24025a00-4226-4ea7-9c91-7d5805cec35c" (UID: "24025a00-4226-4ea7-9c91-7d5805cec35c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:46:31.916367 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:31.916346    4425 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24025a00-4226-4ea7-9c91-7d5805cec35c-openshift-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:31.917680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:31.917655    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24025a00-4226-4ea7-9c91-7d5805cec35c-metrics-certs" pod "24025a00-4226-4ea7-9c91-7d5805cec35c" (UID: "24025a00-4226-4ea7-9c91-7d5805cec35c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:46:31.918100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:31.918075    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24025a00-4226-4ea7-9c91-7d5805cec35c-maas-api-tls" pod "24025a00-4226-4ea7-9c91-7d5805cec35c" (UID: "24025a00-4226-4ea7-9c91-7d5805cec35c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:46:31.918184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:31.918082    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24025a00-4226-4ea7-9c91-7d5805cec35c-kube-api-access-xdlwd" pod "24025a00-4226-4ea7-9c91-7d5805cec35c" (UID: "24025a00-4226-4ea7-9c91-7d5805cec35c"). InnerVolumeSpecName "kube-api-access-xdlwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:46:32.017850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:32.017826    4425 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24025a00-4226-4ea7-9c91-7d5805cec35c-metrics-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:32.017850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:32.017851    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24025a00-4226-4ea7-9c91-7d5805cec35c-maas-api-tls\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:32.017996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:32.017862    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdlwd\" (UniqueName: \"kubernetes.io/projected/24025a00-4226-4ea7-9c91-7d5805cec35c-kube-api-access-xdlwd\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:32.133247 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:32.133219    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"]
Oct 30 01:46:32.139309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:32.139285    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6aa9aa-bdvth"]
Oct 30 01:46:32.288671 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:32.288637    4425 generic.go:356] "Generic (PLEG): container finished" podID="24025a00-4226-4ea7-9c91-7d5805cec35c" containerID="ce8d34b7a9eb4d45f17e8de1cbcfff8bcc9169408eb58d52b08465ffe58ff600" exitCode=0
Oct 30 01:46:32.289068 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:32.288682    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"
Oct 30 01:46:32.289068 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:32.288686    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9" event={"ID":"24025a00-4226-4ea7-9c91-7d5805cec35c","Type":"ContainerDied","Data":"ce8d34b7a9eb4d45f17e8de1cbcfff8bcc9169408eb58d52b08465ffe58ff600"}
Oct 30 01:46:32.289068 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:32.288826    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9" event={"ID":"24025a00-4226-4ea7-9c91-7d5805cec35c","Type":"ContainerDied","Data":"05f8b28c36209273225e804c2443e65f42f06b7c437800b23f0e8f4d97010b36"}
Oct 30 01:46:32.289068 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:32.288854    4425 scope.go:122] "RemoveContainer" containerID="ce8d34b7a9eb4d45f17e8de1cbcfff8bcc9169408eb58d52b08465ffe58ff600"
Oct 30 01:46:32.299828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:32.299817    4425 scope.go:122] "RemoveContainer" containerID="ce8d34b7a9eb4d45f17e8de1cbcfff8bcc9169408eb58d52b08465ffe58ff600"
Oct 30 01:46:32.300118 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:46:32.300098    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8d34b7a9eb4d45f17e8de1cbcfff8bcc9169408eb58d52b08465ffe58ff600\": container with ID starting with ce8d34b7a9eb4d45f17e8de1cbcfff8bcc9169408eb58d52b08465ffe58ff600 not found: ID does not exist" containerID="ce8d34b7a9eb4d45f17e8de1cbcfff8bcc9169408eb58d52b08465ffe58ff600"
Oct 30 01:46:32.300218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:32.300125    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8d34b7a9eb4d45f17e8de1cbcfff8bcc9169408eb58d52b08465ffe58ff600"} err="failed to get container status \"ce8d34b7a9eb4d45f17e8de1cbcfff8bcc9169408eb58d52b08465ffe58ff600\": rpc error: code = NotFound desc = could not find container \"ce8d34b7a9eb4d45f17e8de1cbcfff8bcc9169408eb58d52b08465ffe58ff600\": container with ID starting with ce8d34b7a9eb4d45f17e8de1cbcfff8bcc9169408eb58d52b08465ffe58ff600 not found: ID does not exist"
Oct 30 01:46:32.343422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:32.343392    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"]
Oct 30 01:46:32.355506 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:32.355478    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6aa9aa-8db65ff85-z55h9"]
Oct 30 01:46:33.423679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:33.423645    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24025a00-4226-4ea7-9c91-7d5805cec35c" path="/var/lib/kubelet/pods/24025a00-4226-4ea7-9c91-7d5805cec35c/volumes"
Oct 30 01:46:33.424340 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:33.424290    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8995d47f-a3b1-4f6a-a759-d4abceffcb09" path="/var/lib/kubelet/pods/8995d47f-a3b1-4f6a-a759-d4abceffcb09/volumes"
Oct 30 01:46:37.555285 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:46:37.555251    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:46:42.072463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.072433    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"]
Oct 30 01:46:42.083432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.083403    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52dd0bed-84qvv"]
Oct 30 01:46:42.709184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.709135    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"]
Oct 30 01:46:42.709941 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.709918    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8995d47f-a3b1-4f6a-a759-d4abceffcb09" containerName="revoke-keys"
Oct 30 01:46:42.709987 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.709943    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8995d47f-a3b1-4f6a-a759-d4abceffcb09" containerName="revoke-keys"
Oct 30 01:46:42.709987 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.709957    4425 cpu_manager.go:414] "containerMap: removing container" podUID="e560b5a5-44db-4f11-ba85-30220486eb66" containerName="main"
Oct 30 01:46:42.709987 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.709963    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e560b5a5-44db-4f11-ba85-30220486eb66" containerName="main"
Oct 30 01:46:42.710082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.709989    4425 cpu_manager.go:414] "containerMap: removing container" podUID="24025a00-4226-4ea7-9c91-7d5805cec35c" containerName="maas-api"
Oct 30 01:46:42.710082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.709994    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24025a00-4226-4ea7-9c91-7d5805cec35c" containerName="maas-api"
Oct 30 01:46:42.710150 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.710108    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8995d47f-a3b1-4f6a-a759-d4abceffcb09" containerName="revoke-keys"
Oct 30 01:46:42.710150 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.710117    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e560b5a5-44db-4f11-ba85-30220486eb66" containerName="main"
Oct 30 01:46:42.710150 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.710129    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="24025a00-4226-4ea7-9c91-7d5805cec35c" containerName="maas-api"
Oct 30 01:46:42.713935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.713918    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"
Oct 30 01:46:42.742611 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.742590    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:46:42.744676 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.744660    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:46:42.775267 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.775242    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"]
Oct 30 01:46:42.845016 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.844980    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8g8v\" (UniqueName: \"kubernetes.io/projected/1ec7a537-538e-42de-9124-201a0f77088e-kube-api-access-f8g8v\") pod \"maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl\" (UID: \"1ec7a537-538e-42de-9124-201a0f77088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"
Oct 30 01:46:42.845193 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.845020    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec7a537-538e-42de-9124-201a0f77088e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl\" (UID: \"1ec7a537-538e-42de-9124-201a0f77088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"
Oct 30 01:46:42.845257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.845241    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec7a537-538e-42de-9124-201a0f77088e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl\" (UID: \"1ec7a537-538e-42de-9124-201a0f77088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"
Oct 30 01:46:42.947090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.947064    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8g8v\" (UniqueName: \"kubernetes.io/projected/1ec7a537-538e-42de-9124-201a0f77088e-kube-api-access-f8g8v\") pod \"maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl\" (UID: \"1ec7a537-538e-42de-9124-201a0f77088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"
Oct 30 01:46:42.947273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.947096    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec7a537-538e-42de-9124-201a0f77088e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl\" (UID: \"1ec7a537-538e-42de-9124-201a0f77088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"
Oct 30 01:46:42.947273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.947199    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec7a537-538e-42de-9124-201a0f77088e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl\" (UID: \"1ec7a537-538e-42de-9124-201a0f77088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"
Oct 30 01:46:42.947800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.947774    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec7a537-538e-42de-9124-201a0f77088e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl\" (UID: \"1ec7a537-538e-42de-9124-201a0f77088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"
Oct 30 01:46:42.949343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.949324    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec7a537-538e-42de-9124-201a0f77088e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl\" (UID: \"1ec7a537-538e-42de-9124-201a0f77088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"
Oct 30 01:46:42.969121 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:42.969064    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8g8v\" (UniqueName: \"kubernetes.io/projected/1ec7a537-538e-42de-9124-201a0f77088e-kube-api-access-f8g8v\") pod \"maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl\" (UID: \"1ec7a537-538e-42de-9124-201a0f77088e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"
Oct 30 01:46:43.027413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:43.027385    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"
Oct 30 01:46:43.338091 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:43.338062    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"]
Oct 30 01:46:43.339427 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:46:43.339400    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec7a537_538e_42de_9124_201a0f77088e.slice/crio-a085db55693b6b9205bdad278b4d2c8b67e5723f9863acbbb7e90691d30a884a WatchSource:0}: Error finding container a085db55693b6b9205bdad278b4d2c8b67e5723f9863acbbb7e90691d30a884a: Status 404 returned error can't find the container with id a085db55693b6b9205bdad278b4d2c8b67e5723f9863acbbb7e90691d30a884a
Oct 30 01:46:43.383081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:43.383048    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl" event={"ID":"1ec7a537-538e-42de-9124-201a0f77088e","Type":"ContainerStarted","Data":"a085db55693b6b9205bdad278b4d2c8b67e5723f9863acbbb7e90691d30a884a"}
Oct 30 01:46:43.423720 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:43.423699    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a361f3-187a-4583-bbb2-6e8064cc948b" path="/var/lib/kubelet/pods/84a361f3-187a-4583-bbb2-6e8064cc948b/volumes"
Oct 30 01:46:44.393035 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:44.392949    4425 generic.go:356] "Generic (PLEG): container finished" podID="1ec7a537-538e-42de-9124-201a0f77088e" containerID="fbcd456fe2f7ac850db046f8a1fb507dd249521bcd2d34a7a3c4a5cc3e50a6f2" exitCode=0
Oct 30 01:46:44.393445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:44.393040    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl" event={"ID":"1ec7a537-538e-42de-9124-201a0f77088e","Type":"ContainerDied","Data":"fbcd456fe2f7ac850db046f8a1fb507dd249521bcd2d34a7a3c4a5cc3e50a6f2"}
Oct 30 01:46:45.551582 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:45.551556    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"
Oct 30 01:46:45.678339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:45.678265    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec7a537-538e-42de-9124-201a0f77088e-kube-api-access-f8g8v\" (UniqueName: \"kubernetes.io/projected/1ec7a537-538e-42de-9124-201a0f77088e-kube-api-access-f8g8v\") pod \"1ec7a537-538e-42de-9124-201a0f77088e\" (UID: \"1ec7a537-538e-42de-9124-201a0f77088e\") "
Oct 30 01:46:45.678502 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:45.678490    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec7a537-538e-42de-9124-201a0f77088e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec7a537-538e-42de-9124-201a0f77088e-maas-api-openshift-ca\") pod \"1ec7a537-538e-42de-9124-201a0f77088e\" (UID: \"1ec7a537-538e-42de-9124-201a0f77088e\") "
Oct 30 01:46:45.678557 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:45.678516    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ec7a537-538e-42de-9124-201a0f77088e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec7a537-538e-42de-9124-201a0f77088e-maas-api-xks-ca\") pod \"1ec7a537-538e-42de-9124-201a0f77088e\" (UID: \"1ec7a537-538e-42de-9124-201a0f77088e\") "
Oct 30 01:46:45.678860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:45.678831    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec7a537-538e-42de-9124-201a0f77088e-maas-api-openshift-ca" pod "1ec7a537-538e-42de-9124-201a0f77088e" (UID: "1ec7a537-538e-42de-9124-201a0f77088e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:46:45.680419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:45.680394    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec7a537-538e-42de-9124-201a0f77088e-kube-api-access-f8g8v" pod "1ec7a537-538e-42de-9124-201a0f77088e" (UID: "1ec7a537-538e-42de-9124-201a0f77088e"). InnerVolumeSpecName "kube-api-access-f8g8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:46:45.680668 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:45.680647    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ec7a537-538e-42de-9124-201a0f77088e-maas-api-xks-ca" pod "1ec7a537-538e-42de-9124-201a0f77088e" (UID: "1ec7a537-538e-42de-9124-201a0f77088e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:46:45.780388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:45.780358    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec7a537-538e-42de-9124-201a0f77088e-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:45.780388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:45.780388    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec7a537-538e-42de-9124-201a0f77088e-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:45.780564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:45.780401    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8g8v\" (UniqueName: \"kubernetes.io/projected/1ec7a537-538e-42de-9124-201a0f77088e-kube-api-access-f8g8v\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:46:46.411188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:46.411143    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl" event={"ID":"1ec7a537-538e-42de-9124-201a0f77088e","Type":"ContainerDied","Data":"a085db55693b6b9205bdad278b4d2c8b67e5723f9863acbbb7e90691d30a884a"}
Oct 30 01:46:46.411375 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:46.411229    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a085db55693b6b9205bdad278b4d2c8b67e5723f9863acbbb7e90691d30a884a"
Oct 30 01:46:46.411375 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:46:46.411151    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"
Oct 30 01:47:07.369445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.369412    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"]
Oct 30 01:47:07.370138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.370123    4425 cpu_manager.go:414] "containerMap: removing container" podUID="1ec7a537-538e-42de-9124-201a0f77088e" containerName="revoke-keys"
Oct 30 01:47:07.370208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.370139    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec7a537-538e-42de-9124-201a0f77088e" containerName="revoke-keys"
Oct 30 01:47:07.370288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.370278    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec7a537-538e-42de-9124-201a0f77088e" containerName="revoke-keys"
Oct 30 01:47:07.375486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.375468    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:47:07.420414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.420391    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e6fc524e-serving-cert\""
Oct 30 01:47:07.436622 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.436588    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"]
Oct 30 01:47:07.500940 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.500912    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/561f35d5-20a9-46b5-bfe0-004dfec28d66-metrics-certs\") pod \"maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc\" (UID: \"561f35d5-20a9-46b5-bfe0-004dfec28d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:47:07.501100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.500947    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/561f35d5-20a9-46b5-bfe0-004dfec28d66-maas-api-tls\") pod \"maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc\" (UID: \"561f35d5-20a9-46b5-bfe0-004dfec28d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:47:07.501218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.501118    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/561f35d5-20a9-46b5-bfe0-004dfec28d66-openshift-service-ca\") pod \"maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc\" (UID: \"561f35d5-20a9-46b5-bfe0-004dfec28d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:47:07.501409 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.501384    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvd4r\" (UniqueName: \"kubernetes.io/projected/561f35d5-20a9-46b5-bfe0-004dfec28d66-kube-api-access-xvd4r\") pod \"maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc\" (UID: \"561f35d5-20a9-46b5-bfe0-004dfec28d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:47:07.554966 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:47:07.554940    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:47:07.603553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.603519    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvd4r\" (UniqueName: \"kubernetes.io/projected/561f35d5-20a9-46b5-bfe0-004dfec28d66-kube-api-access-xvd4r\") pod \"maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc\" (UID: \"561f35d5-20a9-46b5-bfe0-004dfec28d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:47:07.603738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.603655    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/561f35d5-20a9-46b5-bfe0-004dfec28d66-metrics-certs\") pod \"maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc\" (UID: \"561f35d5-20a9-46b5-bfe0-004dfec28d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:47:07.603738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.603690    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/561f35d5-20a9-46b5-bfe0-004dfec28d66-maas-api-tls\") pod \"maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc\" (UID: \"561f35d5-20a9-46b5-bfe0-004dfec28d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:47:07.603872 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.603754    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/561f35d5-20a9-46b5-bfe0-004dfec28d66-openshift-service-ca\") pod \"maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc\" (UID: \"561f35d5-20a9-46b5-bfe0-004dfec28d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:47:07.605021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.604992    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/561f35d5-20a9-46b5-bfe0-004dfec28d66-openshift-service-ca\") pod \"maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc\" (UID: \"561f35d5-20a9-46b5-bfe0-004dfec28d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:47:07.607145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.607121    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/561f35d5-20a9-46b5-bfe0-004dfec28d66-metrics-certs\") pod \"maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc\" (UID: \"561f35d5-20a9-46b5-bfe0-004dfec28d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:47:07.607298 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.607275    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/561f35d5-20a9-46b5-bfe0-004dfec28d66-maas-api-tls\") pod \"maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc\" (UID: \"561f35d5-20a9-46b5-bfe0-004dfec28d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:47:07.624956 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.624894    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvd4r\" (UniqueName: \"kubernetes.io/projected/561f35d5-20a9-46b5-bfe0-004dfec28d66-kube-api-access-xvd4r\") pod \"maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc\" (UID: \"561f35d5-20a9-46b5-bfe0-004dfec28d66\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:47:07.689174 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:07.689128    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:47:08.000659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:08.000629    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"]
Oct 30 01:47:08.001461 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:47:08.001431    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561f35d5_20a9_46b5_bfe0_004dfec28d66.slice/crio-e39eb355b0e01dff66efce727eda11772a7fbd2adf5933136b326529ddc3c3d4 WatchSource:0}: Error finding container e39eb355b0e01dff66efce727eda11772a7fbd2adf5933136b326529ddc3c3d4: Status 404 returned error can't find the container with id e39eb355b0e01dff66efce727eda11772a7fbd2adf5933136b326529ddc3c3d4
Oct 30 01:47:08.002973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:08.002952    4425 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:47:08.562354 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:08.562333    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:47:08.615742 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:08.615716    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc" event={"ID":"561f35d5-20a9-46b5-bfe0-004dfec28d66","Type":"ContainerStarted","Data":"460e747ea119cd8740b0f475a48b6ea65c5b37434b2def4337e20909c144684a"}
Oct 30 01:47:08.615877 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:08.615779    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:47:08.615877 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:08.615805    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc" event={"ID":"561f35d5-20a9-46b5-bfe0-004dfec28d66","Type":"ContainerStarted","Data":"e39eb355b0e01dff66efce727eda11772a7fbd2adf5933136b326529ddc3c3d4"}
Oct 30 01:47:08.659349 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:08.659285    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc" podStartSLOduration=1.154947426 podStartE2EDuration="1.659265139s" totalImagesPullingTime="504.317713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:08.65574901 +0000 UTC m=+3937.405404319" watchObservedRunningTime="2026-10-30 01:47:08.659265139 +0000 UTC m=+3937.408920444"
Oct 30 01:47:14.629368 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:14.629339    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:47:31.280345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:31.280300    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"]
Oct 30 01:47:31.289131 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:31.289102    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1a4b17-4tdjl"]
Oct 30 01:47:31.424409 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:31.424375    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec7a537-538e-42de-9124-201a0f77088e" path="/var/lib/kubelet/pods/1ec7a537-538e-42de-9124-201a0f77088e/volumes"
Oct 30 01:47:36.178522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:36.178490    4425 scope.go:122] "RemoveContainer" containerID="39c9f888988d58b77e6db19e0049508562993649c29e8b64e3fee720e11dbd24"
Oct 30 01:47:36.188962 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:36.188940    4425 scope.go:122] "RemoveContainer" containerID="d48669e53de67035a8e22fcc5556907934be8f0508cf5d551c06f6aba04ca6fc"
Oct 30 01:47:37.554741 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:47:37.554706    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:47:42.450691 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:42.450655    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"]
Oct 30 01:47:42.455289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:42.455272    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"
Oct 30 01:47:42.478325 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:42.478302    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:47:42.478919 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:42.478900    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:47:42.491438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:42.491413    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"]
Oct 30 01:47:42.609164 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:42.609135    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a9d843e-ll784\" (UID: \"5023f3d5-3dda-44af-88f6-8ce25e6f84f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"
Oct 30 01:47:42.609449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:42.609429    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fmhq\" (UniqueName: \"kubernetes.io/projected/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-kube-api-access-2fmhq\") pod \"maas-api-revoke-sub-1a9d843e-ll784\" (UID: \"5023f3d5-3dda-44af-88f6-8ce25e6f84f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"
Oct 30 01:47:42.609496 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:42.609468    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a9d843e-ll784\" (UID: \"5023f3d5-3dda-44af-88f6-8ce25e6f84f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"
Oct 30 01:47:42.710497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:42.710433    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fmhq\" (UniqueName: \"kubernetes.io/projected/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-kube-api-access-2fmhq\") pod \"maas-api-revoke-sub-1a9d843e-ll784\" (UID: \"5023f3d5-3dda-44af-88f6-8ce25e6f84f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"
Oct 30 01:47:42.710497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:42.710469    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a9d843e-ll784\" (UID: \"5023f3d5-3dda-44af-88f6-8ce25e6f84f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"
Oct 30 01:47:42.710659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:42.710531    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a9d843e-ll784\" (UID: \"5023f3d5-3dda-44af-88f6-8ce25e6f84f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"
Oct 30 01:47:42.711129 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:42.711106    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a9d843e-ll784\" (UID: \"5023f3d5-3dda-44af-88f6-8ce25e6f84f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"
Oct 30 01:47:42.712905 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:42.712881    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a9d843e-ll784\" (UID: \"5023f3d5-3dda-44af-88f6-8ce25e6f84f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"
Oct 30 01:47:42.792571 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:42.792548    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fmhq\" (UniqueName: \"kubernetes.io/projected/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-kube-api-access-2fmhq\") pod \"maas-api-revoke-sub-1a9d843e-ll784\" (UID: \"5023f3d5-3dda-44af-88f6-8ce25e6f84f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"
Oct 30 01:47:43.068757 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:43.068730    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"
Oct 30 01:47:43.376935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:43.376897    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"]
Oct 30 01:47:43.378075 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:47:43.378045    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5023f3d5_3dda_44af_88f6_8ce25e6f84f4.slice/crio-053a76d28c0fd9754ec38f3fd6b599e9413c195feeb0ca8f0d4c07e36bc064ac WatchSource:0}: Error finding container 053a76d28c0fd9754ec38f3fd6b599e9413c195feeb0ca8f0d4c07e36bc064ac: Status 404 returned error can't find the container with id 053a76d28c0fd9754ec38f3fd6b599e9413c195feeb0ca8f0d4c07e36bc064ac
Oct 30 01:47:43.888020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:43.887977    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784" event={"ID":"5023f3d5-3dda-44af-88f6-8ce25e6f84f4","Type":"ContainerStarted","Data":"13ebe953f2986aca078f498c53145780d2b4371047a5ceb834cd085a3a2fbfe3"}
Oct 30 01:47:43.888471 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:43.888045    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784" event={"ID":"5023f3d5-3dda-44af-88f6-8ce25e6f84f4","Type":"ContainerStarted","Data":"053a76d28c0fd9754ec38f3fd6b599e9413c195feeb0ca8f0d4c07e36bc064ac"}
Oct 30 01:47:43.930770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:43.930674    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784" podStartSLOduration=1.9306619189999998 podStartE2EDuration="1.930661919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:43.928600279 +0000 UTC m=+3972.678255586" watchObservedRunningTime="2026-10-30 01:47:43.930661919 +0000 UTC m=+3972.680317226"
Oct 30 01:47:44.897582 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:44.897541    4425 generic.go:356] "Generic (PLEG): container finished" podID="5023f3d5-3dda-44af-88f6-8ce25e6f84f4" containerID="13ebe953f2986aca078f498c53145780d2b4371047a5ceb834cd085a3a2fbfe3" exitCode=0
Oct 30 01:47:44.897998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:44.897617    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784" event={"ID":"5023f3d5-3dda-44af-88f6-8ce25e6f84f4","Type":"ContainerDied","Data":"13ebe953f2986aca078f498c53145780d2b4371047a5ceb834cd085a3a2fbfe3"}
Oct 30 01:47:46.061734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:46.061705    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"
Oct 30 01:47:46.153934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:46.153902    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-maas-api-openshift-ca\") pod \"5023f3d5-3dda-44af-88f6-8ce25e6f84f4\" (UID: \"5023f3d5-3dda-44af-88f6-8ce25e6f84f4\") "
Oct 30 01:47:46.154102 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:46.154046    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-maas-api-xks-ca\") pod \"5023f3d5-3dda-44af-88f6-8ce25e6f84f4\" (UID: \"5023f3d5-3dda-44af-88f6-8ce25e6f84f4\") "
Oct 30 01:47:46.154209 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:46.154191    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-kube-api-access-2fmhq\" (UniqueName: \"kubernetes.io/projected/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-kube-api-access-2fmhq\") pod \"5023f3d5-3dda-44af-88f6-8ce25e6f84f4\" (UID: \"5023f3d5-3dda-44af-88f6-8ce25e6f84f4\") "
Oct 30 01:47:46.154290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:46.154270    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-maas-api-openshift-ca" pod "5023f3d5-3dda-44af-88f6-8ce25e6f84f4" (UID: "5023f3d5-3dda-44af-88f6-8ce25e6f84f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:47:46.154625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:46.154607    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:47:46.156095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:46.156075    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-maas-api-xks-ca" pod "5023f3d5-3dda-44af-88f6-8ce25e6f84f4" (UID: "5023f3d5-3dda-44af-88f6-8ce25e6f84f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:47:46.156171 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:46.156114    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-kube-api-access-2fmhq" pod "5023f3d5-3dda-44af-88f6-8ce25e6f84f4" (UID: "5023f3d5-3dda-44af-88f6-8ce25e6f84f4"). InnerVolumeSpecName "kube-api-access-2fmhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:47:46.255694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:46.255666    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:47:46.255694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:46.255694    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fmhq\" (UniqueName: \"kubernetes.io/projected/5023f3d5-3dda-44af-88f6-8ce25e6f84f4-kube-api-access-2fmhq\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:47:46.917783 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:46.917755    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"
Oct 30 01:47:46.918088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:46.917788    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784" event={"ID":"5023f3d5-3dda-44af-88f6-8ce25e6f84f4","Type":"ContainerDied","Data":"053a76d28c0fd9754ec38f3fd6b599e9413c195feeb0ca8f0d4c07e36bc064ac"}
Oct 30 01:47:46.918265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:46.918111    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="053a76d28c0fd9754ec38f3fd6b599e9413c195feeb0ca8f0d4c07e36bc064ac"
Oct 30 01:47:47.185287 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.185209    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"]
Oct 30 01:47:47.186423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.186399    4425 cpu_manager.go:414] "containerMap: removing container" podUID="5023f3d5-3dda-44af-88f6-8ce25e6f84f4" containerName="revoke-keys"
Oct 30 01:47:47.186493 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.186426    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5023f3d5-3dda-44af-88f6-8ce25e6f84f4" containerName="revoke-keys"
Oct 30 01:47:47.186734 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.186709    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5023f3d5-3dda-44af-88f6-8ce25e6f84f4" containerName="revoke-keys"
Oct 30 01:47:47.191550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.191409    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.206273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.206243    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-0e1a98\"/\"openshift-service-ca.crt\""
Oct 30 01:47:47.207804 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.207789    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-0e1a98\"/\"kube-root-ca.crt\""
Oct 30 01:47:47.223396 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.223366    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"]
Oct 30 01:47:47.269254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.269216    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-tmp-dir\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.269475 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.269332    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-model-cache\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.269475 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.269384    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vprns\" (UniqueName: \"kubernetes.io/projected/7ab1823f-9005-407f-b702-905853eb61b8-kube-api-access-vprns\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.269475 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.269435    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-dshm\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.269659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.269575    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-home\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.269659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.269618    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ab1823f-9005-407f-b702-905853eb61b8-tls-certs\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.285870 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.285753    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-0e1a98\"/\"default-dockercfg-qhcq2\""
Oct 30 01:47:47.286336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.286235    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-0e1a98\"/\"sub-test-model-0e1a98-kserve-self-signed-certs\""
Oct 30 01:47:47.370844 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.370809    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-home\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.371037 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.370870    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ab1823f-9005-407f-b702-905853eb61b8-tls-certs\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.371037 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.370915    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-tmp-dir\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.371037 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.370969    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-model-cache\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.371037 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.371004    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vprns\" (UniqueName: \"kubernetes.io/projected/7ab1823f-9005-407f-b702-905853eb61b8-kube-api-access-vprns\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.371296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.371047    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-dshm\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.371365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.371299    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-home\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.371431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.371373    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-tmp-dir\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.371816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.371787    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-model-cache\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.373558 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.373529    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-dshm\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.373894 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.373878    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ab1823f-9005-407f-b702-905853eb61b8-tls-certs\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.401626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.401598    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vprns\" (UniqueName: \"kubernetes.io/projected/7ab1823f-9005-407f-b702-905853eb61b8-kube-api-access-vprns\") pod \"sub-test-model-0e1a98-kserve-685b77dbb5-8r66z\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") " pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.508901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.508797    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:47.849145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.849117    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"]
Oct 30 01:47:47.849737 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:47:47.849709    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ab1823f_9005_407f_b702_905853eb61b8.slice/crio-9d02e6c0d5b6f37b395b83b3137adf9fbac95f5710fa77d102ee4c0bcd690613 WatchSource:0}: Error finding container 9d02e6c0d5b6f37b395b83b3137adf9fbac95f5710fa77d102ee4c0bcd690613: Status 404 returned error can't find the container with id 9d02e6c0d5b6f37b395b83b3137adf9fbac95f5710fa77d102ee4c0bcd690613
Oct 30 01:47:47.928297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.928261    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z" event={"ID":"7ab1823f-9005-407f-b702-905853eb61b8","Type":"ContainerStarted","Data":"ae507c0d102b4a1be60631414249016ace83263b22fef271a6f75a5d5ec40a4d"}
Oct 30 01:47:47.928402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.928316    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z" event={"ID":"7ab1823f-9005-407f-b702-905853eb61b8","Type":"ContainerStarted","Data":"9d02e6c0d5b6f37b395b83b3137adf9fbac95f5710fa77d102ee4c0bcd690613"}
Oct 30 01:47:47.964994 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:47.964954    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z" podStartSLOduration=0.964942309 podStartE2EDuration="964.942309ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:47.964613871 +0000 UTC m=+3976.714269179" watchObservedRunningTime="2026-10-30 01:47:47.964942309 +0000 UTC m=+3976.714597614"
Oct 30 01:47:48.509667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:48.509638    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:49.436638 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:49.436606    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:47:57.509103 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:57.509072    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:57.521763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:57.521736    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:47:58.017745 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:47:58.017710    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:48:07.556710 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:48:07.556673    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:48:16.313242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.313143    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"]
Oct 30 01:48:16.319320 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.319207    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.333875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.333841    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-1854c4\"/\"openshift-service-ca.crt\""
Oct 30 01:48:16.343721 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.343695    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-1854c4\"/\"kube-root-ca.crt\""
Oct 30 01:48:16.343906 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.343723    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-1854c4\"/\"sub-test-model-1854c4-kserve-self-signed-certs\""
Oct 30 01:48:16.343999 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.343723    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-1854c4\"/\"default-dockercfg-nsxdz\""
Oct 30 01:48:16.348940 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.348915    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"]
Oct 30 01:48:16.361953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.361924    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-home\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.362086 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.362055    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-tmp-dir\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.362168 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.362106    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5qln\" (UniqueName: \"kubernetes.io/projected/723cbbff-c865-4304-9851-d1968c50cd8a-kube-api-access-t5qln\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.362236 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.362206    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-dshm\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.362331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.362307    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/723cbbff-c865-4304-9851-d1968c50cd8a-tls-certs\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.362445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.362431    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-model-cache\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.463671 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.463634    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-model-cache\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.463880 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.463707    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-home\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.463880 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.463791    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-tmp-dir\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.463880 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.463838    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5qln\" (UniqueName: \"kubernetes.io/projected/723cbbff-c865-4304-9851-d1968c50cd8a-kube-api-access-t5qln\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.464072 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.463901    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-dshm\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.464072 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.463955    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/723cbbff-c865-4304-9851-d1968c50cd8a-tls-certs\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.464193 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.464126    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-model-cache\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.464262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.464180    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-tmp-dir\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.464262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.464190    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-home\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.466460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.466431    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-dshm\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.466733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.466711    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/723cbbff-c865-4304-9851-d1968c50cd8a-tls-certs\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.486295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.486266    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5qln\" (UniqueName: \"kubernetes.io/projected/723cbbff-c865-4304-9851-d1968c50cd8a-kube-api-access-t5qln\") pod \"sub-test-model-1854c4-kserve-6db98f484c-kp4k6\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") " pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.634227 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.633705    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:16.954793 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:16.954763    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"]
Oct 30 01:48:16.955430 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:48:16.955406    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod723cbbff_c865_4304_9851_d1968c50cd8a.slice/crio-a18b87385ae354b2936c2b8b3ceb3c60cfa72b5b3cac0d8b36e3eb8765a79168 WatchSource:0}: Error finding container a18b87385ae354b2936c2b8b3ceb3c60cfa72b5b3cac0d8b36e3eb8765a79168: Status 404 returned error can't find the container with id a18b87385ae354b2936c2b8b3ceb3c60cfa72b5b3cac0d8b36e3eb8765a79168
Oct 30 01:48:17.205600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:17.205496    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6" event={"ID":"723cbbff-c865-4304-9851-d1968c50cd8a","Type":"ContainerStarted","Data":"16ec2c0dd9c390b292f642ea37994c89efed4d2b7466a6a97278d4ef4fde79dc"}
Oct 30 01:48:17.205600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:17.205552    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6" event={"ID":"723cbbff-c865-4304-9851-d1968c50cd8a","Type":"ContainerStarted","Data":"a18b87385ae354b2936c2b8b3ceb3c60cfa72b5b3cac0d8b36e3eb8765a79168"}
Oct 30 01:48:17.251621 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:17.251566    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6" podStartSLOduration=1.251552094 podStartE2EDuration="1.251552094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:17.249640082 +0000 UTC m=+4005.999295388" watchObservedRunningTime="2026-10-30 01:48:17.251552094 +0000 UTC m=+4006.001207425"
Oct 30 01:48:17.634640 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:17.634611    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:26.634614 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:26.634584    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:26.647553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:26.647526    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:27.301055 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:27.301024    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:48:30.558395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:30.558360    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:48:33.030000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:33.029964    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"]
Oct 30 01:48:33.039207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:33.039177    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf3dfd77-7fp2k"]
Oct 30 01:48:33.426247 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:33.426149    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d798ab33-6a65-4b02-bf19-28a3f2a1b320" path="/var/lib/kubelet/pods/d798ab33-6a65-4b02-bf19-28a3f2a1b320/volumes"
Oct 30 01:48:34.035458 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:34.035424    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"]
Oct 30 01:48:34.045073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:34.045046    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c3ae11-m2jwn"]
Oct 30 01:48:35.423925 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:35.423892    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6" path="/var/lib/kubelet/pods/a9e1b5f1-badf-4ac4-9152-1f7489f1a8c6/volumes"
Oct 30 01:48:36.220693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:36.220665    4425 scope.go:122] "RemoveContainer" containerID="7b1e3ff21f9030dc6de4b2defb72f035ec50fa4154b84f6af483d2369138c3ae"
Oct 30 01:48:36.231338 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:36.231319    4425 scope.go:122] "RemoveContainer" containerID="ee8975ce0b229cd945eea694535d0a4d033a88e4f020470b3916c67b4783af22"
Oct 30 01:48:36.240825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:36.240800    4425 scope.go:122] "RemoveContainer" containerID="736256b3309b6bd077804159ed73579592374a8a3b0065390e919e0300acd3c9"
Oct 30 01:48:37.554898 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:48:37.554863    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:48:45.030991 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:45.030953    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"]
Oct 30 01:48:45.040067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:45.040037    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"]
Oct 30 01:48:45.044136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:45.044110    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c94a9366-cnvv4"]
Oct 30 01:48:45.047661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:45.047633    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f73a184-bgv6h"]
Oct 30 01:48:45.425087 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:45.425010    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="351e25a6-877b-47ad-8206-b383bf0bad41" path="/var/lib/kubelet/pods/351e25a6-877b-47ad-8206-b383bf0bad41/volumes"
Oct 30 01:48:45.425491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:45.425473    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b3214ea-a3b4-43d8-88b7-01d044b63bb4" path="/var/lib/kubelet/pods/9b3214ea-a3b4-43d8-88b7-01d044b63bb4/volumes"
Oct 30 01:48:53.299293 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.299261    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"]
Oct 30 01:48:53.308521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.308418    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.337067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.337033    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"]
Oct 30 01:48:53.340690 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.340595    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c666c2\"/\"sub-test-model-c666c2-kserve-self-signed-certs\""
Oct 30 01:48:53.340915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.340707    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c666c2\"/\"openshift-service-ca.crt\""
Oct 30 01:48:53.340915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.340767    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c666c2\"/\"default-dockercfg-ckgqh\""
Oct 30 01:48:53.340915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.340831    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c666c2\"/\"kube-root-ca.crt\""
Oct 30 01:48:53.436828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.436796    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj9v8\" (UniqueName: \"kubernetes.io/projected/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-kube-api-access-dj9v8\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.436828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.436840    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-home\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.437056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.437028    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-tls-certs\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.437174 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.437145    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-tmp-dir\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.437247 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.437234    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-model-cache\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.437314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.437298    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-dshm\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.539454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.538835    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-model-cache\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.539454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.538930    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-dshm\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.539454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.539010    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj9v8\" (UniqueName: \"kubernetes.io/projected/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-kube-api-access-dj9v8\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.539454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.539050    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-home\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.539454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.539213    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-tls-certs\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.539454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.539301    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-tmp-dir\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.540339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.539703    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-model-cache\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.540339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.540297    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-tmp-dir\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.540602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.540554    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-home\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.542817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.542787    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-tls-certs\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.548206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.547473    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-dshm\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.568906 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.568833    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj9v8\" (UniqueName: \"kubernetes.io/projected/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-kube-api-access-dj9v8\") pod \"sub-test-model-c666c2-kserve-fbc6c5489-4t9z9\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") " pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.657427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.657390    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:48:53.970832 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:53.970799    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"]
Oct 30 01:48:53.970953 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:48:53.970827    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4e5e67f_7ccd_4744_bf03_dfae00c0f982.slice/crio-e006a666fde4f8c8040fd298df9d7fc2341ec5bc82f588fd35e5eb5848b341da WatchSource:0}: Error finding container e006a666fde4f8c8040fd298df9d7fc2341ec5bc82f588fd35e5eb5848b341da: Status 404 returned error can't find the container with id e006a666fde4f8c8040fd298df9d7fc2341ec5bc82f588fd35e5eb5848b341da
Oct 30 01:48:54.504365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:54.504327    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9" event={"ID":"f4e5e67f-7ccd-4744-bf03-dfae00c0f982","Type":"ContainerStarted","Data":"b394eb5f14c68327f4094cf48f0121fde97af75ce83c6c50780ac3ef8450b61e"}
Oct 30 01:48:54.504365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:54.504372    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9" event={"ID":"f4e5e67f-7ccd-4744-bf03-dfae00c0f982","Type":"ContainerStarted","Data":"e006a666fde4f8c8040fd298df9d7fc2341ec5bc82f588fd35e5eb5848b341da"}
Oct 30 01:48:54.544822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:54.544777    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9" podStartSLOduration=1.544763812 podStartE2EDuration="1.544763812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:54.54143643 +0000 UTC m=+4043.291091738" watchObservedRunningTime="2026-10-30 01:48:54.544763812 +0000 UTC m=+4043.294419119"
Oct 30 01:48:54.658344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:48:54.658281    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:49:03.657796 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:03.657759    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:49:03.670148 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:03.670126    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:49:04.595401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:04.595370    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:49:07.555756 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:49:07.555716    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:49:27.563407 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:27.563374    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:49:36.279354 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:36.279314    4425 scope.go:122] "RemoveContainer" containerID="52f2786eba813dc15ac96ac341732f775e97085e8d7f5a81de1eb864846b083c"
Oct 30 01:49:36.293774 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:36.293746    4425 scope.go:122] "RemoveContainer" containerID="87406ea0119cb37d61f0c02534bd0e03026f8516475cfcf4b9f0ecb5037bd002"
Oct 30 01:49:36.305150 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:36.305130    4425 scope.go:122] "RemoveContainer" containerID="2a644c80518964f21504f9a32e65616e6d2b6de01ae8b9bb430d34440e393c9b"
Oct 30 01:49:36.995088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:36.995050    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"]
Oct 30 01:49:37.001438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:37.001395    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"
Oct 30 01:49:37.019780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:37.019755    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"]
Oct 30 01:49:37.024549 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:37.024522    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:49:37.024549 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:37.024536    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:49:37.076361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:37.076335    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e387cfc-ce48-4b89-8d7f-f258d848f002-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t\" (UID: \"8e387cfc-ce48-4b89-8d7f-f258d848f002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"
Oct 30 01:49:37.076519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:37.076402    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr92z\" (UniqueName: \"kubernetes.io/projected/8e387cfc-ce48-4b89-8d7f-f258d848f002-kube-api-access-cr92z\") pod \"maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t\" (UID: \"8e387cfc-ce48-4b89-8d7f-f258d848f002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"
Oct 30 01:49:37.076519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:37.076457    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e387cfc-ce48-4b89-8d7f-f258d848f002-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t\" (UID: \"8e387cfc-ce48-4b89-8d7f-f258d848f002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"
Oct 30 01:49:37.177746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:37.177715    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e387cfc-ce48-4b89-8d7f-f258d848f002-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t\" (UID: \"8e387cfc-ce48-4b89-8d7f-f258d848f002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"
Oct 30 01:49:37.177928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:37.177787    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr92z\" (UniqueName: \"kubernetes.io/projected/8e387cfc-ce48-4b89-8d7f-f258d848f002-kube-api-access-cr92z\") pod \"maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t\" (UID: \"8e387cfc-ce48-4b89-8d7f-f258d848f002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"
Oct 30 01:49:37.177928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:37.177809    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e387cfc-ce48-4b89-8d7f-f258d848f002-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t\" (UID: \"8e387cfc-ce48-4b89-8d7f-f258d848f002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"
Oct 30 01:49:37.178551 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:37.178525    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e387cfc-ce48-4b89-8d7f-f258d848f002-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t\" (UID: \"8e387cfc-ce48-4b89-8d7f-f258d848f002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"
Oct 30 01:49:37.180285 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:37.180263    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e387cfc-ce48-4b89-8d7f-f258d848f002-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t\" (UID: \"8e387cfc-ce48-4b89-8d7f-f258d848f002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"
Oct 30 01:49:37.198451 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:37.198429    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr92z\" (UniqueName: \"kubernetes.io/projected/8e387cfc-ce48-4b89-8d7f-f258d848f002-kube-api-access-cr92z\") pod \"maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t\" (UID: \"8e387cfc-ce48-4b89-8d7f-f258d848f002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"
Oct 30 01:49:37.316311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:37.316281    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"
Oct 30 01:49:37.555610 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:49:37.555576    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:49:37.619952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:37.619920    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"]
Oct 30 01:49:37.620674 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:49:37.620646    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e387cfc_ce48_4b89_8d7f_f258d848f002.slice/crio-8c159d6f689e05783603f3c9ed1e4716ef609c4df31dc26fa5610c96fafe21ec WatchSource:0}: Error finding container 8c159d6f689e05783603f3c9ed1e4716ef609c4df31dc26fa5610c96fafe21ec: Status 404 returned error can't find the container with id 8c159d6f689e05783603f3c9ed1e4716ef609c4df31dc26fa5610c96fafe21ec
Oct 30 01:49:37.830402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:37.830365    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t" event={"ID":"8e387cfc-ce48-4b89-8d7f-f258d848f002","Type":"ContainerStarted","Data":"8c159d6f689e05783603f3c9ed1e4716ef609c4df31dc26fa5610c96fafe21ec"}
Oct 30 01:49:38.839892 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:38.839810    4425 generic.go:356] "Generic (PLEG): container finished" podID="8e387cfc-ce48-4b89-8d7f-f258d848f002" containerID="6942267e0e6779ee6fc33163673cae9d2d08648aa01d6133ef90b4aaa9b9cecd" exitCode=0
Oct 30 01:49:38.840286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:38.839889    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t" event={"ID":"8e387cfc-ce48-4b89-8d7f-f258d848f002","Type":"ContainerDied","Data":"6942267e0e6779ee6fc33163673cae9d2d08648aa01d6133ef90b4aaa9b9cecd"}
Oct 30 01:49:39.993731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:39.993697    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"
Oct 30 01:49:40.119255 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:40.119175    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e387cfc-ce48-4b89-8d7f-f258d848f002-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e387cfc-ce48-4b89-8d7f-f258d848f002-maas-api-xks-ca\") pod \"8e387cfc-ce48-4b89-8d7f-f258d848f002\" (UID: \"8e387cfc-ce48-4b89-8d7f-f258d848f002\") "
Oct 30 01:49:40.119402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:40.119306    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e387cfc-ce48-4b89-8d7f-f258d848f002-kube-api-access-cr92z\" (UniqueName: \"kubernetes.io/projected/8e387cfc-ce48-4b89-8d7f-f258d848f002-kube-api-access-cr92z\") pod \"8e387cfc-ce48-4b89-8d7f-f258d848f002\" (UID: \"8e387cfc-ce48-4b89-8d7f-f258d848f002\") "
Oct 30 01:49:40.119528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:40.119511    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e387cfc-ce48-4b89-8d7f-f258d848f002-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e387cfc-ce48-4b89-8d7f-f258d848f002-maas-api-openshift-ca\") pod \"8e387cfc-ce48-4b89-8d7f-f258d848f002\" (UID: \"8e387cfc-ce48-4b89-8d7f-f258d848f002\") "
Oct 30 01:49:40.119951 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:40.119912    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e387cfc-ce48-4b89-8d7f-f258d848f002-maas-api-openshift-ca" pod "8e387cfc-ce48-4b89-8d7f-f258d848f002" (UID: "8e387cfc-ce48-4b89-8d7f-f258d848f002"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:49:40.121282 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:40.121260    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e387cfc-ce48-4b89-8d7f-f258d848f002-maas-api-xks-ca" pod "8e387cfc-ce48-4b89-8d7f-f258d848f002" (UID: "8e387cfc-ce48-4b89-8d7f-f258d848f002"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:49:40.121642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:40.121628    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e387cfc-ce48-4b89-8d7f-f258d848f002-kube-api-access-cr92z" pod "8e387cfc-ce48-4b89-8d7f-f258d848f002" (UID: "8e387cfc-ce48-4b89-8d7f-f258d848f002"). InnerVolumeSpecName "kube-api-access-cr92z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:49:40.220880 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:40.220858    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e387cfc-ce48-4b89-8d7f-f258d848f002-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:49:40.220880 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:40.220882    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e387cfc-ce48-4b89-8d7f-f258d848f002-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:49:40.221092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:40.220892    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr92z\" (UniqueName: \"kubernetes.io/projected/8e387cfc-ce48-4b89-8d7f-f258d848f002-kube-api-access-cr92z\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:49:40.855412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:40.855384    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"
Oct 30 01:49:40.855592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:40.855421    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t" event={"ID":"8e387cfc-ce48-4b89-8d7f-f258d848f002","Type":"ContainerDied","Data":"8c159d6f689e05783603f3c9ed1e4716ef609c4df31dc26fa5610c96fafe21ec"}
Oct 30 01:49:40.855592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:40.855473    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c159d6f689e05783603f3c9ed1e4716ef609c4df31dc26fa5610c96fafe21ec"
Oct 30 01:49:47.425526 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:47.425447    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"]
Oct 30 01:49:47.452312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:47.452282    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"]
Oct 30 01:49:52.426746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.426708    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d" podUID="19f75c87-9928-4901-8e16-18086f0760e2" containerName="payload-processing" containerID="cri-o://7b5cda28518cfd15095810a6295650bf8aeb836d27d353396fecfefc35becd6c" gracePeriod=25
Oct 30 01:49:52.452853 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.452822    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l" podUID="30d24304-fb77-479a-9a11-06cc577c4382" containerName="payload-pre-processing" containerID="cri-o://d00723cc413edeecb7b1e297e3119b2e03acc056dedbebda17df325c8ba3cda4" gracePeriod=25
Oct 30 01:49:52.739943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.739911    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"
Oct 30 01:49:52.773958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.773935    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"
Oct 30 01:49:52.892755 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.892724    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d24304-fb77-479a-9a11-06cc577c4382-kube-api-access-pbrx2\" (UniqueName: \"kubernetes.io/projected/30d24304-fb77-479a-9a11-06cc577c4382-kube-api-access-pbrx2\") pod \"30d24304-fb77-479a-9a11-06cc577c4382\" (UID: \"30d24304-fb77-479a-9a11-06cc577c4382\") "
Oct 30 01:49:52.892938 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.892920    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f75c87-9928-4901-8e16-18086f0760e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19f75c87-9928-4901-8e16-18086f0760e2-plugins-config-volume\") pod \"19f75c87-9928-4901-8e16-18086f0760e2\" (UID: \"19f75c87-9928-4901-8e16-18086f0760e2\") "
Oct 30 01:49:52.893015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.893000    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d24304-fb77-479a-9a11-06cc577c4382-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d24304-fb77-479a-9a11-06cc577c4382-plugins-config-volume\") pod \"30d24304-fb77-479a-9a11-06cc577c4382\" (UID: \"30d24304-fb77-479a-9a11-06cc577c4382\") "
Oct 30 01:49:52.893062 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.893036    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f75c87-9928-4901-8e16-18086f0760e2-kube-api-access-kb4td\" (UniqueName: \"kubernetes.io/projected/19f75c87-9928-4901-8e16-18086f0760e2-kube-api-access-kb4td\") pod \"19f75c87-9928-4901-8e16-18086f0760e2\" (UID: \"19f75c87-9928-4901-8e16-18086f0760e2\") "
Oct 30 01:49:52.893392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.893364    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f75c87-9928-4901-8e16-18086f0760e2-plugins-config-volume" pod "19f75c87-9928-4901-8e16-18086f0760e2" (UID: "19f75c87-9928-4901-8e16-18086f0760e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:49:52.893506 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.893370    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d24304-fb77-479a-9a11-06cc577c4382-plugins-config-volume" pod "30d24304-fb77-479a-9a11-06cc577c4382" (UID: "30d24304-fb77-479a-9a11-06cc577c4382"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:49:52.893754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.893731    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19f75c87-9928-4901-8e16-18086f0760e2-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:49:52.893803 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.893759    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d24304-fb77-479a-9a11-06cc577c4382-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:49:52.894997 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.894971    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d24304-fb77-479a-9a11-06cc577c4382-kube-api-access-pbrx2" pod "30d24304-fb77-479a-9a11-06cc577c4382" (UID: "30d24304-fb77-479a-9a11-06cc577c4382"). InnerVolumeSpecName "kube-api-access-pbrx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:49:52.895095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.895044    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f75c87-9928-4901-8e16-18086f0760e2-kube-api-access-kb4td" pod "19f75c87-9928-4901-8e16-18086f0760e2" (UID: "19f75c87-9928-4901-8e16-18086f0760e2"). InnerVolumeSpecName "kube-api-access-kb4td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:49:52.951752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.951691    4425 generic.go:356] "Generic (PLEG): container finished" podID="19f75c87-9928-4901-8e16-18086f0760e2" containerID="7b5cda28518cfd15095810a6295650bf8aeb836d27d353396fecfefc35becd6c" exitCode=0
Oct 30 01:49:52.951752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.951742    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"
Oct 30 01:49:52.951881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.951779    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d" event={"ID":"19f75c87-9928-4901-8e16-18086f0760e2","Type":"ContainerDied","Data":"7b5cda28518cfd15095810a6295650bf8aeb836d27d353396fecfefc35becd6c"}
Oct 30 01:49:52.951881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.951831    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d" event={"ID":"19f75c87-9928-4901-8e16-18086f0760e2","Type":"ContainerDied","Data":"60dc4e68b6466e65e3860d84382d096c5d96a54adbba387f604bfca5ede2b1e3"}
Oct 30 01:49:52.951881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.951859    4425 scope.go:122] "RemoveContainer" containerID="7b5cda28518cfd15095810a6295650bf8aeb836d27d353396fecfefc35becd6c"
Oct 30 01:49:52.953592 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.953569    4425 generic.go:356] "Generic (PLEG): container finished" podID="30d24304-fb77-479a-9a11-06cc577c4382" containerID="d00723cc413edeecb7b1e297e3119b2e03acc056dedbebda17df325c8ba3cda4" exitCode=0
Oct 30 01:49:52.953718 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.953612    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"
Oct 30 01:49:52.953718 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.953648    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l" event={"ID":"30d24304-fb77-479a-9a11-06cc577c4382","Type":"ContainerDied","Data":"d00723cc413edeecb7b1e297e3119b2e03acc056dedbebda17df325c8ba3cda4"}
Oct 30 01:49:52.953718 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.953689    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l" event={"ID":"30d24304-fb77-479a-9a11-06cc577c4382","Type":"ContainerDied","Data":"b93d51f90a4a3a7cbb3f70e74c6cfa5b480d25a21356cfdef1b22a67c43a3f9f"}
Oct 30 01:49:52.965003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.964988    4425 scope.go:122] "RemoveContainer" containerID="7b5cda28518cfd15095810a6295650bf8aeb836d27d353396fecfefc35becd6c"
Oct 30 01:49:52.965312 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:49:52.965294    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5cda28518cfd15095810a6295650bf8aeb836d27d353396fecfefc35becd6c\": container with ID starting with 7b5cda28518cfd15095810a6295650bf8aeb836d27d353396fecfefc35becd6c not found: ID does not exist" containerID="7b5cda28518cfd15095810a6295650bf8aeb836d27d353396fecfefc35becd6c"
Oct 30 01:49:52.965383 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.965319    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5cda28518cfd15095810a6295650bf8aeb836d27d353396fecfefc35becd6c"} err="failed to get container status \"7b5cda28518cfd15095810a6295650bf8aeb836d27d353396fecfefc35becd6c\": rpc error: code = NotFound desc = could not find container \"7b5cda28518cfd15095810a6295650bf8aeb836d27d353396fecfefc35becd6c\": container with ID starting with 7b5cda28518cfd15095810a6295650bf8aeb836d27d353396fecfefc35becd6c not found: ID does not exist"
Oct 30 01:49:52.965383 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.965336    4425 scope.go:122] "RemoveContainer" containerID="d00723cc413edeecb7b1e297e3119b2e03acc056dedbebda17df325c8ba3cda4"
Oct 30 01:49:52.975009 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.974990    4425 scope.go:122] "RemoveContainer" containerID="d00723cc413edeecb7b1e297e3119b2e03acc056dedbebda17df325c8ba3cda4"
Oct 30 01:49:52.975267 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:49:52.975244    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d00723cc413edeecb7b1e297e3119b2e03acc056dedbebda17df325c8ba3cda4\": container with ID starting with d00723cc413edeecb7b1e297e3119b2e03acc056dedbebda17df325c8ba3cda4 not found: ID does not exist" containerID="d00723cc413edeecb7b1e297e3119b2e03acc056dedbebda17df325c8ba3cda4"
Oct 30 01:49:52.975357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.975272    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d00723cc413edeecb7b1e297e3119b2e03acc056dedbebda17df325c8ba3cda4"} err="failed to get container status \"d00723cc413edeecb7b1e297e3119b2e03acc056dedbebda17df325c8ba3cda4\": rpc error: code = NotFound desc = could not find container \"d00723cc413edeecb7b1e297e3119b2e03acc056dedbebda17df325c8ba3cda4\": container with ID starting with d00723cc413edeecb7b1e297e3119b2e03acc056dedbebda17df325c8ba3cda4 not found: ID does not exist"
Oct 30 01:49:52.995261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.995239    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb4td\" (UniqueName: \"kubernetes.io/projected/19f75c87-9928-4901-8e16-18086f0760e2-kube-api-access-kb4td\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:49:52.995356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.995264    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbrx2\" (UniqueName: \"kubernetes.io/projected/30d24304-fb77-479a-9a11-06cc577c4382-kube-api-access-pbrx2\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:49:52.999315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:52.999294    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"]
Oct 30 01:49:53.003403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:53.003382    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6fc524e-7594c7c499-jgc7l"]
Oct 30 01:49:53.031894 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:53.031874    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"]
Oct 30 01:49:53.036958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:53.036936    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e6fc524e-845b759b56-tk84d"]
Oct 30 01:49:53.426004 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:53.425975    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f75c87-9928-4901-8e16-18086f0760e2" path="/var/lib/kubelet/pods/19f75c87-9928-4901-8e16-18086f0760e2/volumes"
Oct 30 01:49:53.426428 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:49:53.426411    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d24304-fb77-479a-9a11-06cc577c4382" path="/var/lib/kubelet/pods/30d24304-fb77-479a-9a11-06cc577c4382/volumes"
Oct 30 01:50:00.040631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:00.040597    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"]
Oct 30 01:50:00.049974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:00.049946    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9f9fb5-6k5t5"]
Oct 30 01:50:01.432581 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:01.432537    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="997e8b87-01db-47e2-b9ef-bf86441a027d" path="/var/lib/kubelet/pods/997e8b87-01db-47e2-b9ef-bf86441a027d/volumes"
Oct 30 01:50:07.554691 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:50:07.554657    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:50:25.256027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:25.255989    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"]
Oct 30 01:50:25.256518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:25.256241    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc" podUID="561f35d5-20a9-46b5-bfe0-004dfec28d66" containerName="maas-api" containerID="cri-o://460e747ea119cd8740b0f475a48b6ea65c5b37434b2def4337e20909c144684a" gracePeriod=30
Oct 30 01:50:25.585227 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:25.585198    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:50:25.675708 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:25.674462    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/561f35d5-20a9-46b5-bfe0-004dfec28d66-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/561f35d5-20a9-46b5-bfe0-004dfec28d66-maas-api-tls\") pod \"561f35d5-20a9-46b5-bfe0-004dfec28d66\" (UID: \"561f35d5-20a9-46b5-bfe0-004dfec28d66\") "
Oct 30 01:50:25.675708 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:25.674552    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/561f35d5-20a9-46b5-bfe0-004dfec28d66-kube-api-access-xvd4r\" (UniqueName: \"kubernetes.io/projected/561f35d5-20a9-46b5-bfe0-004dfec28d66-kube-api-access-xvd4r\") pod \"561f35d5-20a9-46b5-bfe0-004dfec28d66\" (UID: \"561f35d5-20a9-46b5-bfe0-004dfec28d66\") "
Oct 30 01:50:25.675708 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:25.674591    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/561f35d5-20a9-46b5-bfe0-004dfec28d66-metrics-certs\" (UniqueName: \"kubernetes.io/secret/561f35d5-20a9-46b5-bfe0-004dfec28d66-metrics-certs\") pod \"561f35d5-20a9-46b5-bfe0-004dfec28d66\" (UID: \"561f35d5-20a9-46b5-bfe0-004dfec28d66\") "
Oct 30 01:50:25.675708 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:25.674840    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/561f35d5-20a9-46b5-bfe0-004dfec28d66-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/561f35d5-20a9-46b5-bfe0-004dfec28d66-openshift-service-ca\") pod \"561f35d5-20a9-46b5-bfe0-004dfec28d66\" (UID: \"561f35d5-20a9-46b5-bfe0-004dfec28d66\") "
Oct 30 01:50:25.676579 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:25.676278    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561f35d5-20a9-46b5-bfe0-004dfec28d66-openshift-service-ca" pod "561f35d5-20a9-46b5-bfe0-004dfec28d66" (UID: "561f35d5-20a9-46b5-bfe0-004dfec28d66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:50:25.678905 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:25.678876    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561f35d5-20a9-46b5-bfe0-004dfec28d66-kube-api-access-xvd4r" pod "561f35d5-20a9-46b5-bfe0-004dfec28d66" (UID: "561f35d5-20a9-46b5-bfe0-004dfec28d66"). InnerVolumeSpecName "kube-api-access-xvd4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:50:25.682962 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:25.681548    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561f35d5-20a9-46b5-bfe0-004dfec28d66-maas-api-tls" pod "561f35d5-20a9-46b5-bfe0-004dfec28d66" (UID: "561f35d5-20a9-46b5-bfe0-004dfec28d66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:50:25.689522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:25.689498    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561f35d5-20a9-46b5-bfe0-004dfec28d66-metrics-certs" pod "561f35d5-20a9-46b5-bfe0-004dfec28d66" (UID: "561f35d5-20a9-46b5-bfe0-004dfec28d66"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:50:25.776735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:25.776701    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/561f35d5-20a9-46b5-bfe0-004dfec28d66-maas-api-tls\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:50:25.776735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:25.776731    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvd4r\" (UniqueName: \"kubernetes.io/projected/561f35d5-20a9-46b5-bfe0-004dfec28d66-kube-api-access-xvd4r\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:50:25.776735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:25.776741    4425 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/561f35d5-20a9-46b5-bfe0-004dfec28d66-metrics-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:50:25.777065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:25.776751    4425 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/561f35d5-20a9-46b5-bfe0-004dfec28d66-openshift-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:50:26.145975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:26.145926    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"]
Oct 30 01:50:26.149111 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:26.149072    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6fc524e-2td4t"]
Oct 30 01:50:26.207407 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:26.207379    4425 generic.go:356] "Generic (PLEG): container finished" podID="561f35d5-20a9-46b5-bfe0-004dfec28d66" containerID="460e747ea119cd8740b0f475a48b6ea65c5b37434b2def4337e20909c144684a" exitCode=0
Oct 30 01:50:26.207570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:26.207429    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"
Oct 30 01:50:26.207570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:26.207464    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc" event={"ID":"561f35d5-20a9-46b5-bfe0-004dfec28d66","Type":"ContainerDied","Data":"460e747ea119cd8740b0f475a48b6ea65c5b37434b2def4337e20909c144684a"}
Oct 30 01:50:26.207570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:26.207519    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc" event={"ID":"561f35d5-20a9-46b5-bfe0-004dfec28d66","Type":"ContainerDied","Data":"e39eb355b0e01dff66efce727eda11772a7fbd2adf5933136b326529ddc3c3d4"}
Oct 30 01:50:26.207570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:26.207549    4425 scope.go:122] "RemoveContainer" containerID="460e747ea119cd8740b0f475a48b6ea65c5b37434b2def4337e20909c144684a"
Oct 30 01:50:26.219706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:26.219665    4425 scope.go:122] "RemoveContainer" containerID="460e747ea119cd8740b0f475a48b6ea65c5b37434b2def4337e20909c144684a"
Oct 30 01:50:26.219975 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:50:26.219952    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460e747ea119cd8740b0f475a48b6ea65c5b37434b2def4337e20909c144684a\": container with ID starting with 460e747ea119cd8740b0f475a48b6ea65c5b37434b2def4337e20909c144684a not found: ID does not exist" containerID="460e747ea119cd8740b0f475a48b6ea65c5b37434b2def4337e20909c144684a"
Oct 30 01:50:26.220068 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:26.219986    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460e747ea119cd8740b0f475a48b6ea65c5b37434b2def4337e20909c144684a"} err="failed to get container status \"460e747ea119cd8740b0f475a48b6ea65c5b37434b2def4337e20909c144684a\": rpc error: code = NotFound desc = could not find container \"460e747ea119cd8740b0f475a48b6ea65c5b37434b2def4337e20909c144684a\": container with ID starting with 460e747ea119cd8740b0f475a48b6ea65c5b37434b2def4337e20909c144684a not found: ID does not exist"
Oct 30 01:50:26.258895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:26.258871    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"]
Oct 30 01:50:26.262806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:26.262778    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e6fc524e-7c8945bc8-9jjkc"]
Oct 30 01:50:27.424124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:27.424092    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="561f35d5-20a9-46b5-bfe0-004dfec28d66" path="/var/lib/kubelet/pods/561f35d5-20a9-46b5-bfe0-004dfec28d66/volumes"
Oct 30 01:50:27.424566 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:27.424550    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e387cfc-ce48-4b89-8d7f-f258d848f002" path="/var/lib/kubelet/pods/8e387cfc-ce48-4b89-8d7f-f258d848f002/volumes"
Oct 30 01:50:35.831953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.831915    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"]
Oct 30 01:50:35.832649 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.832633    4425 cpu_manager.go:414] "containerMap: removing container" podUID="19f75c87-9928-4901-8e16-18086f0760e2" containerName="payload-processing"
Oct 30 01:50:35.832699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.832649    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f75c87-9928-4901-8e16-18086f0760e2" containerName="payload-processing"
Oct 30 01:50:35.832699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.832658    4425 cpu_manager.go:414] "containerMap: removing container" podUID="561f35d5-20a9-46b5-bfe0-004dfec28d66" containerName="maas-api"
Oct 30 01:50:35.832699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.832663    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561f35d5-20a9-46b5-bfe0-004dfec28d66" containerName="maas-api"
Oct 30 01:50:35.832699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.832681    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8e387cfc-ce48-4b89-8d7f-f258d848f002" containerName="revoke-keys"
Oct 30 01:50:35.832699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.832687    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e387cfc-ce48-4b89-8d7f-f258d848f002" containerName="revoke-keys"
Oct 30 01:50:35.832699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.832699    4425 cpu_manager.go:414] "containerMap: removing container" podUID="30d24304-fb77-479a-9a11-06cc577c4382" containerName="payload-pre-processing"
Oct 30 01:50:35.832908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.832704    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d24304-fb77-479a-9a11-06cc577c4382" containerName="payload-pre-processing"
Oct 30 01:50:35.832908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.832809    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e387cfc-ce48-4b89-8d7f-f258d848f002" containerName="revoke-keys"
Oct 30 01:50:35.832908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.832822    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d24304-fb77-479a-9a11-06cc577c4382" containerName="payload-pre-processing"
Oct 30 01:50:35.832908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.832837    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f75c87-9928-4901-8e16-18086f0760e2" containerName="payload-processing"
Oct 30 01:50:35.832908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.832844    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="561f35d5-20a9-46b5-bfe0-004dfec28d66" containerName="maas-api"
Oct 30 01:50:35.837972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.837950    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"
Oct 30 01:50:35.858927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.858901    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"]
Oct 30 01:50:35.916095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.916067    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:50:35.916272 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.916172    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:50:35.994622 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.994585    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5x68\" (UniqueName: \"kubernetes.io/projected/bd86b83d-af82-4d9c-8acf-6285259f8b0d-kube-api-access-j5x68\") pod \"maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq\" (UID: \"bd86b83d-af82-4d9c-8acf-6285259f8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"
Oct 30 01:50:35.994812 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.994680    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd86b83d-af82-4d9c-8acf-6285259f8b0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq\" (UID: \"bd86b83d-af82-4d9c-8acf-6285259f8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"
Oct 30 01:50:35.994876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:35.994834    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd86b83d-af82-4d9c-8acf-6285259f8b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq\" (UID: \"bd86b83d-af82-4d9c-8acf-6285259f8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"
Oct 30 01:50:36.096426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.096338    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd86b83d-af82-4d9c-8acf-6285259f8b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq\" (UID: \"bd86b83d-af82-4d9c-8acf-6285259f8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"
Oct 30 01:50:36.096607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.096440    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5x68\" (UniqueName: \"kubernetes.io/projected/bd86b83d-af82-4d9c-8acf-6285259f8b0d-kube-api-access-j5x68\") pod \"maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq\" (UID: \"bd86b83d-af82-4d9c-8acf-6285259f8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"
Oct 30 01:50:36.096607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.096471    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd86b83d-af82-4d9c-8acf-6285259f8b0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq\" (UID: \"bd86b83d-af82-4d9c-8acf-6285259f8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"
Oct 30 01:50:36.097051 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.097027    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd86b83d-af82-4d9c-8acf-6285259f8b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq\" (UID: \"bd86b83d-af82-4d9c-8acf-6285259f8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"
Oct 30 01:50:36.098685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.098663    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd86b83d-af82-4d9c-8acf-6285259f8b0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq\" (UID: \"bd86b83d-af82-4d9c-8acf-6285259f8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"
Oct 30 01:50:36.136569 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.136540    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5x68\" (UniqueName: \"kubernetes.io/projected/bd86b83d-af82-4d9c-8acf-6285259f8b0d-kube-api-access-j5x68\") pod \"maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq\" (UID: \"bd86b83d-af82-4d9c-8acf-6285259f8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"
Oct 30 01:50:36.152338 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.152316    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"
Oct 30 01:50:36.182204 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.182177    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"]
Oct 30 01:50:36.188425 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.188402    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.200205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.197420    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-model-cache\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.200205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.197481    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-home\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.200205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.197693    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81efdf05-b9a8-4ee2-9451-7cdc213207fb-tls-certs\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.200205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.197793    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-dshm\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.200205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.197832    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkj2v\" (UniqueName: \"kubernetes.io/projected/81efdf05-b9a8-4ee2-9451-7cdc213207fb-kube-api-access-hkj2v\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.200205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.197946    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-tmp-dir\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.218210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.216248    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"]
Oct 30 01:50:36.218210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.217733    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-4107fc\"/\"default-dockercfg-c2kgb\""
Oct 30 01:50:36.218210 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.217787    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-4107fc\"/\"rate-test-model-4107fc-kserve-self-signed-certs\""
Oct 30 01:50:36.258871 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.255825    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-4107fc\"/\"kube-root-ca.crt\""
Oct 30 01:50:36.258871 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.255933    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-4107fc\"/\"openshift-service-ca.crt\""
Oct 30 01:50:36.299276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.299243    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-dshm\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.299459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.299442    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkj2v\" (UniqueName: \"kubernetes.io/projected/81efdf05-b9a8-4ee2-9451-7cdc213207fb-kube-api-access-hkj2v\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.299614 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.299530    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-tmp-dir\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.299614 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.299608    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-model-cache\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.299814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.299650    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-home\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.299814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.299778    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81efdf05-b9a8-4ee2-9451-7cdc213207fb-tls-certs\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.300428 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.300407    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-tmp-dir\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.300854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.300832    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-model-cache\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.301065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.301047    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-home\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.315516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.315445    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-dshm\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.317018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.316990    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81efdf05-b9a8-4ee2-9451-7cdc213207fb-tls-certs\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.335655 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.335634    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkj2v\" (UniqueName: \"kubernetes.io/projected/81efdf05-b9a8-4ee2-9451-7cdc213207fb-kube-api-access-hkj2v\") pod \"rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") " pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.343271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.343247    4425 scope.go:122] "RemoveContainer" containerID="504d9ff614a544627a7fdde25a0cf8099ed25a5ee11ff5a8136fbbe37763f49b"
Oct 30 01:50:36.359096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.358984    4425 scope.go:122] "RemoveContainer" containerID="7dccfb13bdbe6cf9594df1b7c654a39a003e93e21c3d94d2d7489eece06c6884"
Oct 30 01:50:36.368747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.368721    4425 scope.go:122] "RemoveContainer" containerID="77fa4e1ca2ce3bb5690c7ea6a9a666a12cfa0487765cffb5302c23734b6dc642"
Oct 30 01:50:36.379287 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.379269    4425 scope.go:122] "RemoveContainer" containerID="05a0b5fbaa71e6816eeaed242b9003dbf1d94173988025ad4cecf55222be01d5"
Oct 30 01:50:36.504197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.503918    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"]
Oct 30 01:50:36.505238 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:50:36.505204    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd86b83d_af82_4d9c_8acf_6285259f8b0d.slice/crio-20d1a6a7f475a21beb98c3d53ef32148b46076fed80ff56560ec79c7108772e0 WatchSource:0}: Error finding container 20d1a6a7f475a21beb98c3d53ef32148b46076fed80ff56560ec79c7108772e0: Status 404 returned error can't find the container with id 20d1a6a7f475a21beb98c3d53ef32148b46076fed80ff56560ec79c7108772e0
Oct 30 01:50:36.510354 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.510330    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:36.834957 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:50:36.834931    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81efdf05_b9a8_4ee2_9451_7cdc213207fb.slice/crio-2d422fead3ffa1b781ad3763c6573fe676f19589740c1cb35a9239238c2001fa WatchSource:0}: Error finding container 2d422fead3ffa1b781ad3763c6573fe676f19589740c1cb35a9239238c2001fa: Status 404 returned error can't find the container with id 2d422fead3ffa1b781ad3763c6573fe676f19589740c1cb35a9239238c2001fa
Oct 30 01:50:36.835455 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:36.835422    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"]
Oct 30 01:50:37.302222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:37.302182    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf" event={"ID":"81efdf05-b9a8-4ee2-9451-7cdc213207fb","Type":"ContainerStarted","Data":"b156d65e05f4fcc07484913793a003f366b9e0ab5731e4b2dc1507700bbe6b2d"}
Oct 30 01:50:37.302411 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:37.302236    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf" event={"ID":"81efdf05-b9a8-4ee2-9451-7cdc213207fb","Type":"ContainerStarted","Data":"2d422fead3ffa1b781ad3763c6573fe676f19589740c1cb35a9239238c2001fa"}
Oct 30 01:50:37.304075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:37.304046    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq" event={"ID":"bd86b83d-af82-4d9c-8acf-6285259f8b0d","Type":"ContainerStarted","Data":"6a9ed38eda6ca3973f68f258a54d749ea1fa49ef400a3bf0d18b4cc3fdb7152d"}
Oct 30 01:50:37.304215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:37.304095    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq" event={"ID":"bd86b83d-af82-4d9c-8acf-6285259f8b0d","Type":"ContainerStarted","Data":"20d1a6a7f475a21beb98c3d53ef32148b46076fed80ff56560ec79c7108772e0"}
Oct 30 01:50:37.345004 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:37.344955    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf" podStartSLOduration=1.344938604 podStartE2EDuration="1.344938604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:50:37.343648373 +0000 UTC m=+4146.093303685" watchObservedRunningTime="2026-10-30 01:50:37.344938604 +0000 UTC m=+4146.094593917"
Oct 30 01:50:37.382367 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:37.382325    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq" podStartSLOduration=2.3823108680000002 podStartE2EDuration="2.382310868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:50:37.380796902 +0000 UTC m=+4146.130452209" watchObservedRunningTime="2026-10-30 01:50:37.382310868 +0000 UTC m=+4146.131966176"
Oct 30 01:50:37.511018 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:37.510985    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:37.554542 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:50:37.554470    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:50:38.313507 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:38.313467    4425 generic.go:356] "Generic (PLEG): container finished" podID="bd86b83d-af82-4d9c-8acf-6285259f8b0d" containerID="6a9ed38eda6ca3973f68f258a54d749ea1fa49ef400a3bf0d18b4cc3fdb7152d" exitCode=0
Oct 30 01:50:38.313979 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:38.313548    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq" event={"ID":"bd86b83d-af82-4d9c-8acf-6285259f8b0d","Type":"ContainerDied","Data":"6a9ed38eda6ca3973f68f258a54d749ea1fa49ef400a3bf0d18b4cc3fdb7152d"}
Oct 30 01:50:39.477247 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:39.477223    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"
Oct 30 01:50:39.543087 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:39.543059    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd86b83d-af82-4d9c-8acf-6285259f8b0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd86b83d-af82-4d9c-8acf-6285259f8b0d-maas-api-openshift-ca\") pod \"bd86b83d-af82-4d9c-8acf-6285259f8b0d\" (UID: \"bd86b83d-af82-4d9c-8acf-6285259f8b0d\") "
Oct 30 01:50:39.543254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:39.543199    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd86b83d-af82-4d9c-8acf-6285259f8b0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd86b83d-af82-4d9c-8acf-6285259f8b0d-maas-api-xks-ca\") pod \"bd86b83d-af82-4d9c-8acf-6285259f8b0d\" (UID: \"bd86b83d-af82-4d9c-8acf-6285259f8b0d\") "
Oct 30 01:50:39.543347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:39.543292    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd86b83d-af82-4d9c-8acf-6285259f8b0d-kube-api-access-j5x68\" (UniqueName: \"kubernetes.io/projected/bd86b83d-af82-4d9c-8acf-6285259f8b0d-kube-api-access-j5x68\") pod \"bd86b83d-af82-4d9c-8acf-6285259f8b0d\" (UID: \"bd86b83d-af82-4d9c-8acf-6285259f8b0d\") "
Oct 30 01:50:39.543530 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:39.543498    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd86b83d-af82-4d9c-8acf-6285259f8b0d-maas-api-openshift-ca" pod "bd86b83d-af82-4d9c-8acf-6285259f8b0d" (UID: "bd86b83d-af82-4d9c-8acf-6285259f8b0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:50:39.543973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:39.543946    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd86b83d-af82-4d9c-8acf-6285259f8b0d-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:50:39.545369 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:39.545343    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd86b83d-af82-4d9c-8acf-6285259f8b0d-maas-api-xks-ca" pod "bd86b83d-af82-4d9c-8acf-6285259f8b0d" (UID: "bd86b83d-af82-4d9c-8acf-6285259f8b0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:50:39.545500 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:39.545478    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd86b83d-af82-4d9c-8acf-6285259f8b0d-kube-api-access-j5x68" pod "bd86b83d-af82-4d9c-8acf-6285259f8b0d" (UID: "bd86b83d-af82-4d9c-8acf-6285259f8b0d"). InnerVolumeSpecName "kube-api-access-j5x68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:50:39.645513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:39.645453    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd86b83d-af82-4d9c-8acf-6285259f8b0d-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:50:39.645513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:39.645480    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5x68\" (UniqueName: \"kubernetes.io/projected/bd86b83d-af82-4d9c-8acf-6285259f8b0d-kube-api-access-j5x68\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:50:40.330420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:40.330390    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"
Oct 30 01:50:40.330599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:40.330422    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq" event={"ID":"bd86b83d-af82-4d9c-8acf-6285259f8b0d","Type":"ContainerDied","Data":"20d1a6a7f475a21beb98c3d53ef32148b46076fed80ff56560ec79c7108772e0"}
Oct 30 01:50:40.330599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:40.330471    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20d1a6a7f475a21beb98c3d53ef32148b46076fed80ff56560ec79c7108772e0"
Oct 30 01:50:42.044510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:42.044479    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"]
Oct 30 01:50:42.053672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:42.053646    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d36c4a3-dvsrv"]
Oct 30 01:50:43.425290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:43.425251    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47cfb795-a866-454b-813c-b81eb4ca8271" path="/var/lib/kubelet/pods/47cfb795-a866-454b-813c-b81eb4ca8271/volumes"
Oct 30 01:50:46.285889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:46.285855    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"]
Oct 30 01:50:46.316535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:46.316503    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"]
Oct 30 01:50:46.510778 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:46.510745    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:46.523355 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:46.523332    4425 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:47.397051 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:47.397022    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:50:51.286576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:51.286533    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv" podUID="0aab37bd-b15d-4064-b67b-640c159e9b19" containerName="payload-processing" containerID="cri-o://b04f27da98b3ba48d91a9f179ef3e1856128bcee5ef942c055630e9a40259436" gracePeriod=25
Oct 30 01:50:51.317630 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:51.317595    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx" podUID="b73980e8-28ab-4caf-8be2-9455d2b4335f" containerName="payload-pre-processing" containerID="cri-o://9d282886bbac855c685db09d3529ab51fe49d776a11b964f00193655aef185da" gracePeriod=25
Oct 30 01:50:55.028373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:55.028331    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"]
Oct 30 01:50:55.037880 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:55.037852    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b63fe40d-7k6sm"]
Oct 30 01:50:55.424265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:55.424184    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294" path="/var/lib/kubelet/pods/c4a4f68a-68f3-4bbe-a3fa-58eb92d9a294/volumes"
Oct 30 01:50:55.773200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:55.773136    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv" podUID="0aab37bd-b15d-4064-b67b-640c159e9b19" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:50:55.773413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:55.773227    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx" podUID="b73980e8-28ab-4caf-8be2-9455d2b4335f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:50:56.876933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:50:56.876895    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:51:03.306144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.306110    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"]
Oct 30 01:51:03.306882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.306867    4425 cpu_manager.go:414] "containerMap: removing container" podUID="bd86b83d-af82-4d9c-8acf-6285259f8b0d" containerName="revoke-keys"
Oct 30 01:51:03.306930 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.306883    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd86b83d-af82-4d9c-8acf-6285259f8b0d" containerName="revoke-keys"
Oct 30 01:51:03.307006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.306997    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd86b83d-af82-4d9c-8acf-6285259f8b0d" containerName="revoke-keys"
Oct 30 01:51:03.311822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.311802    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"
Oct 30 01:51:03.333618 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.333592    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xwbf\" (UniqueName: \"kubernetes.io/projected/8c936006-5a77-484b-85d9-d74842f1e607-kube-api-access-4xwbf\") pod \"maas-api-revoke-sub-3a5bd9df-hxvcl\" (UID: \"8c936006-5a77-484b-85d9-d74842f1e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"
Oct 30 01:51:03.333750 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.333683    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c936006-5a77-484b-85d9-d74842f1e607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a5bd9df-hxvcl\" (UID: \"8c936006-5a77-484b-85d9-d74842f1e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"
Oct 30 01:51:03.333848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.333828    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c936006-5a77-484b-85d9-d74842f1e607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a5bd9df-hxvcl\" (UID: \"8c936006-5a77-484b-85d9-d74842f1e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"
Oct 30 01:51:03.341439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.341413    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:51:03.341700 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.341683    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:51:03.347142 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.347116    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"]
Oct 30 01:51:03.434687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.434659    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c936006-5a77-484b-85d9-d74842f1e607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a5bd9df-hxvcl\" (UID: \"8c936006-5a77-484b-85d9-d74842f1e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"
Oct 30 01:51:03.434981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.434962    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xwbf\" (UniqueName: \"kubernetes.io/projected/8c936006-5a77-484b-85d9-d74842f1e607-kube-api-access-4xwbf\") pod \"maas-api-revoke-sub-3a5bd9df-hxvcl\" (UID: \"8c936006-5a77-484b-85d9-d74842f1e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"
Oct 30 01:51:03.435109 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.435087    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c936006-5a77-484b-85d9-d74842f1e607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a5bd9df-hxvcl\" (UID: \"8c936006-5a77-484b-85d9-d74842f1e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"
Oct 30 01:51:03.435623 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.435600    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c936006-5a77-484b-85d9-d74842f1e607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a5bd9df-hxvcl\" (UID: \"8c936006-5a77-484b-85d9-d74842f1e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"
Oct 30 01:51:03.436918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.436901    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c936006-5a77-484b-85d9-d74842f1e607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a5bd9df-hxvcl\" (UID: \"8c936006-5a77-484b-85d9-d74842f1e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"
Oct 30 01:51:03.457420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.457397    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xwbf\" (UniqueName: \"kubernetes.io/projected/8c936006-5a77-484b-85d9-d74842f1e607-kube-api-access-4xwbf\") pod \"maas-api-revoke-sub-3a5bd9df-hxvcl\" (UID: \"8c936006-5a77-484b-85d9-d74842f1e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"
Oct 30 01:51:03.625544 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.625462    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"
Oct 30 01:51:03.933617 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:03.933587    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"]
Oct 30 01:51:03.934326 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:51:03.934291    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c936006_5a77_484b_85d9_d74842f1e607.slice/crio-744cf72af13fde0cb20239da8f0b8b39f54ff3c9ce40d05bd4a47121dab3202c WatchSource:0}: Error finding container 744cf72af13fde0cb20239da8f0b8b39f54ff3c9ce40d05bd4a47121dab3202c: Status 404 returned error can't find the container with id 744cf72af13fde0cb20239da8f0b8b39f54ff3c9ce40d05bd4a47121dab3202c
Oct 30 01:51:04.520591 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:04.520506    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl" event={"ID":"8c936006-5a77-484b-85d9-d74842f1e607","Type":"ContainerStarted","Data":"5df0f7d6c16bc8e5d5fe550bf099e83535d885028a545ffbad2b95a507311ca4"}
Oct 30 01:51:04.520591 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:04.520570    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl" event={"ID":"8c936006-5a77-484b-85d9-d74842f1e607","Type":"ContainerStarted","Data":"744cf72af13fde0cb20239da8f0b8b39f54ff3c9ce40d05bd4a47121dab3202c"}
Oct 30 01:51:04.569194 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:04.569102    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl" podStartSLOduration=1.5690851989999999 podStartE2EDuration="1.569085199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:04.564967456 +0000 UTC m=+4173.314622769" watchObservedRunningTime="2026-10-30 01:51:04.569085199 +0000 UTC m=+4173.318740514"
Oct 30 01:51:05.032118 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:05.032088    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"]
Oct 30 01:51:05.037429 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:05.037404    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e48f727c-njsm2"]
Oct 30 01:51:05.424835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:05.424804    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36dd6581-a435-4715-a354-94959d69e3a6" path="/var/lib/kubelet/pods/36dd6581-a435-4715-a354-94959d69e3a6/volumes"
Oct 30 01:51:05.529586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:05.529556    4425 generic.go:356] "Generic (PLEG): container finished" podID="8c936006-5a77-484b-85d9-d74842f1e607" containerID="5df0f7d6c16bc8e5d5fe550bf099e83535d885028a545ffbad2b95a507311ca4" exitCode=0
Oct 30 01:51:05.529946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:05.529596    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl" event={"ID":"8c936006-5a77-484b-85d9-d74842f1e607","Type":"ContainerDied","Data":"5df0f7d6c16bc8e5d5fe550bf099e83535d885028a545ffbad2b95a507311ca4"}
Oct 30 01:51:05.772869 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:05.772831    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx" podUID="b73980e8-28ab-4caf-8be2-9455d2b4335f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:51:05.773059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:05.772836    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv" podUID="0aab37bd-b15d-4064-b67b-640c159e9b19" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:51:06.701141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:06.701114    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"
Oct 30 01:51:06.778585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:06.778553    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c936006-5a77-484b-85d9-d74842f1e607-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c936006-5a77-484b-85d9-d74842f1e607-maas-api-openshift-ca\") pod \"8c936006-5a77-484b-85d9-d74842f1e607\" (UID: \"8c936006-5a77-484b-85d9-d74842f1e607\") "
Oct 30 01:51:06.778770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:06.778608    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c936006-5a77-484b-85d9-d74842f1e607-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c936006-5a77-484b-85d9-d74842f1e607-maas-api-xks-ca\") pod \"8c936006-5a77-484b-85d9-d74842f1e607\" (UID: \"8c936006-5a77-484b-85d9-d74842f1e607\") "
Oct 30 01:51:06.778770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:06.778712    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c936006-5a77-484b-85d9-d74842f1e607-kube-api-access-4xwbf\" (UniqueName: \"kubernetes.io/projected/8c936006-5a77-484b-85d9-d74842f1e607-kube-api-access-4xwbf\") pod \"8c936006-5a77-484b-85d9-d74842f1e607\" (UID: \"8c936006-5a77-484b-85d9-d74842f1e607\") "
Oct 30 01:51:06.778981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:06.778956    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c936006-5a77-484b-85d9-d74842f1e607-maas-api-openshift-ca" pod "8c936006-5a77-484b-85d9-d74842f1e607" (UID: "8c936006-5a77-484b-85d9-d74842f1e607"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:51:06.779274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:06.779252    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c936006-5a77-484b-85d9-d74842f1e607-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:06.780781 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:06.780755    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c936006-5a77-484b-85d9-d74842f1e607-maas-api-xks-ca" pod "8c936006-5a77-484b-85d9-d74842f1e607" (UID: "8c936006-5a77-484b-85d9-d74842f1e607"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:51:06.780967 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:06.780947    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c936006-5a77-484b-85d9-d74842f1e607-kube-api-access-4xwbf" pod "8c936006-5a77-484b-85d9-d74842f1e607" (UID: "8c936006-5a77-484b-85d9-d74842f1e607"). InnerVolumeSpecName "kube-api-access-4xwbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:51:06.880661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:06.880638    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c936006-5a77-484b-85d9-d74842f1e607-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:06.880661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:06.880662    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xwbf\" (UniqueName: \"kubernetes.io/projected/8c936006-5a77-484b-85d9-d74842f1e607-kube-api-access-4xwbf\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:07.549127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:07.549098    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"
Oct 30 01:51:07.549127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:07.549116    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl" event={"ID":"8c936006-5a77-484b-85d9-d74842f1e607","Type":"ContainerDied","Data":"744cf72af13fde0cb20239da8f0b8b39f54ff3c9ce40d05bd4a47121dab3202c"}
Oct 30 01:51:07.549401 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:07.549192    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="744cf72af13fde0cb20239da8f0b8b39f54ff3c9ce40d05bd4a47121dab3202c"
Oct 30 01:51:07.559015 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:51:07.558986    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:51:11.582906 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.582876    4425 generic.go:356] "Generic (PLEG): container finished" podID="b73980e8-28ab-4caf-8be2-9455d2b4335f" containerID="9d282886bbac855c685db09d3529ab51fe49d776a11b964f00193655aef185da" exitCode=0
Oct 30 01:51:11.583323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.582950    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx" event={"ID":"b73980e8-28ab-4caf-8be2-9455d2b4335f","Type":"ContainerDied","Data":"9d282886bbac855c685db09d3529ab51fe49d776a11b964f00193655aef185da"}
Oct 30 01:51:11.584614 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.584590    4425 generic.go:356] "Generic (PLEG): container finished" podID="0aab37bd-b15d-4064-b67b-640c159e9b19" containerID="b04f27da98b3ba48d91a9f179ef3e1856128bcee5ef942c055630e9a40259436" exitCode=0
Oct 30 01:51:11.584758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.584666    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv" event={"ID":"0aab37bd-b15d-4064-b67b-640c159e9b19","Type":"ContainerDied","Data":"b04f27da98b3ba48d91a9f179ef3e1856128bcee5ef942c055630e9a40259436"}
Oct 30 01:51:11.584758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.584711    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv" event={"ID":"0aab37bd-b15d-4064-b67b-640c159e9b19","Type":"ContainerDied","Data":"d5978ca1d6f5e82fba01cbf0019d567cf3ab9066587c99245b35f9446f0e9891"}
Oct 30 01:51:11.584758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.584728    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5978ca1d6f5e82fba01cbf0019d567cf3ab9066587c99245b35f9446f0e9891"
Oct 30 01:51:11.591984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.591966    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"
Oct 30 01:51:11.620543 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.620527    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"
Oct 30 01:51:11.738649 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.738613    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73980e8-28ab-4caf-8be2-9455d2b4335f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73980e8-28ab-4caf-8be2-9455d2b4335f-plugins-config-volume\") pod \"b73980e8-28ab-4caf-8be2-9455d2b4335f\" (UID: \"b73980e8-28ab-4caf-8be2-9455d2b4335f\") "
Oct 30 01:51:11.738837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.738813    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aab37bd-b15d-4064-b67b-640c159e9b19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aab37bd-b15d-4064-b67b-640c159e9b19-plugins-config-volume\") pod \"0aab37bd-b15d-4064-b67b-640c159e9b19\" (UID: \"0aab37bd-b15d-4064-b67b-640c159e9b19\") "
Oct 30 01:51:11.738914 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.738869    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73980e8-28ab-4caf-8be2-9455d2b4335f-kube-api-access-xnkcx\" (UniqueName: \"kubernetes.io/projected/b73980e8-28ab-4caf-8be2-9455d2b4335f-kube-api-access-xnkcx\") pod \"b73980e8-28ab-4caf-8be2-9455d2b4335f\" (UID: \"b73980e8-28ab-4caf-8be2-9455d2b4335f\") "
Oct 30 01:51:11.738967 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.738910    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aab37bd-b15d-4064-b67b-640c159e9b19-kube-api-access-t5mt2\" (UniqueName: \"kubernetes.io/projected/0aab37bd-b15d-4064-b67b-640c159e9b19-kube-api-access-t5mt2\") pod \"0aab37bd-b15d-4064-b67b-640c159e9b19\" (UID: \"0aab37bd-b15d-4064-b67b-640c159e9b19\") "
Oct 30 01:51:11.739199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.738999    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73980e8-28ab-4caf-8be2-9455d2b4335f-plugins-config-volume" pod "b73980e8-28ab-4caf-8be2-9455d2b4335f" (UID: "b73980e8-28ab-4caf-8be2-9455d2b4335f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:51:11.739348 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.739200    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aab37bd-b15d-4064-b67b-640c159e9b19-plugins-config-volume" pod "0aab37bd-b15d-4064-b67b-640c159e9b19" (UID: "0aab37bd-b15d-4064-b67b-640c159e9b19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:51:11.739517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.739497    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b73980e8-28ab-4caf-8be2-9455d2b4335f-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:11.739576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.739522    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aab37bd-b15d-4064-b67b-640c159e9b19-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:11.740924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.740906    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73980e8-28ab-4caf-8be2-9455d2b4335f-kube-api-access-xnkcx" pod "b73980e8-28ab-4caf-8be2-9455d2b4335f" (UID: "b73980e8-28ab-4caf-8be2-9455d2b4335f"). InnerVolumeSpecName "kube-api-access-xnkcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:51:11.740977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.740941    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aab37bd-b15d-4064-b67b-640c159e9b19-kube-api-access-t5mt2" pod "0aab37bd-b15d-4064-b67b-640c159e9b19" (UID: "0aab37bd-b15d-4064-b67b-640c159e9b19"). InnerVolumeSpecName "kube-api-access-t5mt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:51:11.840454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.840397    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnkcx\" (UniqueName: \"kubernetes.io/projected/b73980e8-28ab-4caf-8be2-9455d2b4335f-kube-api-access-xnkcx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:11.840454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:11.840419    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5mt2\" (UniqueName: \"kubernetes.io/projected/0aab37bd-b15d-4064-b67b-640c159e9b19-kube-api-access-t5mt2\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:12.594325 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:12.594236    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx" event={"ID":"b73980e8-28ab-4caf-8be2-9455d2b4335f","Type":"ContainerDied","Data":"c863832f84a8918e4eabfcc500887ddc323153d2b0dc0f3d063d6c76a12286fa"}
Oct 30 01:51:12.594325 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:12.594308    4425 scope.go:122] "RemoveContainer" containerID="9d282886bbac855c685db09d3529ab51fe49d776a11b964f00193655aef185da"
Oct 30 01:51:12.594832 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:12.594253    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"
Oct 30 01:51:12.594832 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:12.594255    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"
Oct 30 01:51:12.639534 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:12.639513    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"]
Oct 30 01:51:12.649194 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:12.649167    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c7bba9-6988f9fd68-cpbqv"]
Oct 30 01:51:12.675489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:12.675463    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"]
Oct 30 01:51:12.678908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:12.678880    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c7bba9-54cf4579dc-r4xnx"]
Oct 30 01:51:13.427422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.427387    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aab37bd-b15d-4064-b67b-640c159e9b19" path="/var/lib/kubelet/pods/0aab37bd-b15d-4064-b67b-640c159e9b19/volumes"
Oct 30 01:51:13.427843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.427829    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73980e8-28ab-4caf-8be2-9455d2b4335f" path="/var/lib/kubelet/pods/b73980e8-28ab-4caf-8be2-9455d2b4335f/volumes"
Oct 30 01:51:13.795708 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.795674    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"]
Oct 30 01:51:13.796381 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.796367    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8c936006-5a77-484b-85d9-d74842f1e607" containerName="revoke-keys"
Oct 30 01:51:13.796435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.796382    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c936006-5a77-484b-85d9-d74842f1e607" containerName="revoke-keys"
Oct 30 01:51:13.796435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.796408    4425 cpu_manager.go:414] "containerMap: removing container" podUID="0aab37bd-b15d-4064-b67b-640c159e9b19" containerName="payload-processing"
Oct 30 01:51:13.796435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.796414    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aab37bd-b15d-4064-b67b-640c159e9b19" containerName="payload-processing"
Oct 30 01:51:13.796585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.796437    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b73980e8-28ab-4caf-8be2-9455d2b4335f" containerName="payload-pre-processing"
Oct 30 01:51:13.796585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.796442    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73980e8-28ab-4caf-8be2-9455d2b4335f" containerName="payload-pre-processing"
Oct 30 01:51:13.796585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.796557    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aab37bd-b15d-4064-b67b-640c159e9b19" containerName="payload-processing"
Oct 30 01:51:13.796585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.796570    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73980e8-28ab-4caf-8be2-9455d2b4335f" containerName="payload-pre-processing"
Oct 30 01:51:13.796585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.796580    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c936006-5a77-484b-85d9-d74842f1e607" containerName="revoke-keys"
Oct 30 01:51:13.801625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.801608    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"
Oct 30 01:51:13.820981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.820957    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"]
Oct 30 01:51:13.825356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.825329    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:51:13.825505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.825459    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:51:13.866691 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.866663    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f31bab-7494-41c2-9f5e-16a326fa7082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-811266ed-rq5kq\" (UID: \"03f31bab-7494-41c2-9f5e-16a326fa7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"
Oct 30 01:51:13.866818 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.866772    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f31bab-7494-41c2-9f5e-16a326fa7082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-811266ed-rq5kq\" (UID: \"03f31bab-7494-41c2-9f5e-16a326fa7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"
Oct 30 01:51:13.866871 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.866834    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjmkx\" (UniqueName: \"kubernetes.io/projected/03f31bab-7494-41c2-9f5e-16a326fa7082-kube-api-access-mjmkx\") pod \"maas-api-revoke-sub-811266ed-rq5kq\" (UID: \"03f31bab-7494-41c2-9f5e-16a326fa7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"
Oct 30 01:51:13.968402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.968370    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f31bab-7494-41c2-9f5e-16a326fa7082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-811266ed-rq5kq\" (UID: \"03f31bab-7494-41c2-9f5e-16a326fa7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"
Oct 30 01:51:13.968605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.968479    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f31bab-7494-41c2-9f5e-16a326fa7082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-811266ed-rq5kq\" (UID: \"03f31bab-7494-41c2-9f5e-16a326fa7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"
Oct 30 01:51:13.968605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.968525    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmkx\" (UniqueName: \"kubernetes.io/projected/03f31bab-7494-41c2-9f5e-16a326fa7082-kube-api-access-mjmkx\") pod \"maas-api-revoke-sub-811266ed-rq5kq\" (UID: \"03f31bab-7494-41c2-9f5e-16a326fa7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"
Oct 30 01:51:13.969273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.969243    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f31bab-7494-41c2-9f5e-16a326fa7082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-811266ed-rq5kq\" (UID: \"03f31bab-7494-41c2-9f5e-16a326fa7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"
Oct 30 01:51:13.971486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.971442    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f31bab-7494-41c2-9f5e-16a326fa7082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-811266ed-rq5kq\" (UID: \"03f31bab-7494-41c2-9f5e-16a326fa7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"
Oct 30 01:51:13.989692 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:13.989665    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjmkx\" (UniqueName: \"kubernetes.io/projected/03f31bab-7494-41c2-9f5e-16a326fa7082-kube-api-access-mjmkx\") pod \"maas-api-revoke-sub-811266ed-rq5kq\" (UID: \"03f31bab-7494-41c2-9f5e-16a326fa7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"
Oct 30 01:51:14.117229 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:14.116171    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"
Oct 30 01:51:14.453603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:14.453478    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"]
Oct 30 01:51:14.461331 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:51:14.461301    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f31bab_7494_41c2_9f5e_16a326fa7082.slice/crio-4a35e1f64288fc0e193fbadcd28d51a85385bd196ff958f76c49e468c750af86 WatchSource:0}: Error finding container 4a35e1f64288fc0e193fbadcd28d51a85385bd196ff958f76c49e468c750af86: Status 404 returned error can't find the container with id 4a35e1f64288fc0e193fbadcd28d51a85385bd196ff958f76c49e468c750af86
Oct 30 01:51:14.613668 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:14.613627    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq" event={"ID":"03f31bab-7494-41c2-9f5e-16a326fa7082","Type":"ContainerStarted","Data":"4a35e1f64288fc0e193fbadcd28d51a85385bd196ff958f76c49e468c750af86"}
Oct 30 01:51:15.627505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:15.627414    4425 generic.go:356] "Generic (PLEG): container finished" podID="03f31bab-7494-41c2-9f5e-16a326fa7082" containerID="a3d91893cc0ae57e8722d9c8e086719ff1a3f7bcba361270e50e227818b1fb06" exitCode=0
Oct 30 01:51:15.627944 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:15.627623    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq" event={"ID":"03f31bab-7494-41c2-9f5e-16a326fa7082","Type":"ContainerDied","Data":"a3d91893cc0ae57e8722d9c8e086719ff1a3f7bcba361270e50e227818b1fb06"}
Oct 30 01:51:16.791554 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:16.791530    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"
Oct 30 01:51:16.907316 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:16.907231    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f31bab-7494-41c2-9f5e-16a326fa7082-kube-api-access-mjmkx\" (UniqueName: \"kubernetes.io/projected/03f31bab-7494-41c2-9f5e-16a326fa7082-kube-api-access-mjmkx\") pod \"03f31bab-7494-41c2-9f5e-16a326fa7082\" (UID: \"03f31bab-7494-41c2-9f5e-16a326fa7082\") "
Oct 30 01:51:16.907316 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:16.907307    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03f31bab-7494-41c2-9f5e-16a326fa7082-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f31bab-7494-41c2-9f5e-16a326fa7082-maas-api-xks-ca\") pod \"03f31bab-7494-41c2-9f5e-16a326fa7082\" (UID: \"03f31bab-7494-41c2-9f5e-16a326fa7082\") "
Oct 30 01:51:16.907553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:16.907370    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03f31bab-7494-41c2-9f5e-16a326fa7082-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f31bab-7494-41c2-9f5e-16a326fa7082-maas-api-openshift-ca\") pod \"03f31bab-7494-41c2-9f5e-16a326fa7082\" (UID: \"03f31bab-7494-41c2-9f5e-16a326fa7082\") "
Oct 30 01:51:16.907779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:16.907755    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f31bab-7494-41c2-9f5e-16a326fa7082-maas-api-openshift-ca" pod "03f31bab-7494-41c2-9f5e-16a326fa7082" (UID: "03f31bab-7494-41c2-9f5e-16a326fa7082"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:51:16.907983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:16.907969    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03f31bab-7494-41c2-9f5e-16a326fa7082-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:16.909493 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:16.909466    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f31bab-7494-41c2-9f5e-16a326fa7082-maas-api-xks-ca" pod "03f31bab-7494-41c2-9f5e-16a326fa7082" (UID: "03f31bab-7494-41c2-9f5e-16a326fa7082"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:51:16.909613 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:16.909466    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f31bab-7494-41c2-9f5e-16a326fa7082-kube-api-access-mjmkx" pod "03f31bab-7494-41c2-9f5e-16a326fa7082" (UID: "03f31bab-7494-41c2-9f5e-16a326fa7082"). InnerVolumeSpecName "kube-api-access-mjmkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:51:17.009065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:17.009036    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjmkx\" (UniqueName: \"kubernetes.io/projected/03f31bab-7494-41c2-9f5e-16a326fa7082-kube-api-access-mjmkx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:17.009065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:17.009063    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03f31bab-7494-41c2-9f5e-16a326fa7082-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:17.644928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:17.644895    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"
Oct 30 01:51:17.644928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:17.644921    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq" event={"ID":"03f31bab-7494-41c2-9f5e-16a326fa7082","Type":"ContainerDied","Data":"4a35e1f64288fc0e193fbadcd28d51a85385bd196ff958f76c49e468c750af86"}
Oct 30 01:51:17.645307 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:17.644975    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a35e1f64288fc0e193fbadcd28d51a85385bd196ff958f76c49e468c750af86"
Oct 30 01:51:24.749720 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:24.749682    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"]
Oct 30 01:51:24.750407 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:24.750391    4425 cpu_manager.go:414] "containerMap: removing container" podUID="03f31bab-7494-41c2-9f5e-16a326fa7082" containerName="revoke-keys"
Oct 30 01:51:24.750407 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:24.750406    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f31bab-7494-41c2-9f5e-16a326fa7082" containerName="revoke-keys"
Oct 30 01:51:24.750549 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:24.750540    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f31bab-7494-41c2-9f5e-16a326fa7082" containerName="revoke-keys"
Oct 30 01:51:24.754297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:24.754282    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"
Oct 30 01:51:24.766794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:24.766772    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:51:24.776531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:24.776505    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:51:24.780463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:24.780438    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"]
Oct 30 01:51:24.803108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:24.803084    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgwdk\" (UniqueName: \"kubernetes.io/projected/077fb11a-0be0-4b97-ad7b-c630ba538274-kube-api-access-zgwdk\") pod \"maas-api-revoke-sub-5fb235c4-gdbdz\" (UID: \"077fb11a-0be0-4b97-ad7b-c630ba538274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"
Oct 30 01:51:24.803270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:24.803164    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077fb11a-0be0-4b97-ad7b-c630ba538274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb235c4-gdbdz\" (UID: \"077fb11a-0be0-4b97-ad7b-c630ba538274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"
Oct 30 01:51:24.803270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:24.803216    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077fb11a-0be0-4b97-ad7b-c630ba538274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb235c4-gdbdz\" (UID: \"077fb11a-0be0-4b97-ad7b-c630ba538274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"
Oct 30 01:51:24.904798 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:24.904764    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgwdk\" (UniqueName: \"kubernetes.io/projected/077fb11a-0be0-4b97-ad7b-c630ba538274-kube-api-access-zgwdk\") pod \"maas-api-revoke-sub-5fb235c4-gdbdz\" (UID: \"077fb11a-0be0-4b97-ad7b-c630ba538274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"
Oct 30 01:51:24.904939 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:24.904820    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077fb11a-0be0-4b97-ad7b-c630ba538274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb235c4-gdbdz\" (UID: \"077fb11a-0be0-4b97-ad7b-c630ba538274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"
Oct 30 01:51:24.904939 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:24.904849    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077fb11a-0be0-4b97-ad7b-c630ba538274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb235c4-gdbdz\" (UID: \"077fb11a-0be0-4b97-ad7b-c630ba538274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"
Oct 30 01:51:24.905590 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:24.905571    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077fb11a-0be0-4b97-ad7b-c630ba538274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb235c4-gdbdz\" (UID: \"077fb11a-0be0-4b97-ad7b-c630ba538274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"
Oct 30 01:51:24.907145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:24.907126    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077fb11a-0be0-4b97-ad7b-c630ba538274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb235c4-gdbdz\" (UID: \"077fb11a-0be0-4b97-ad7b-c630ba538274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"
Oct 30 01:51:24.925285 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:24.925260    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgwdk\" (UniqueName: \"kubernetes.io/projected/077fb11a-0be0-4b97-ad7b-c630ba538274-kube-api-access-zgwdk\") pod \"maas-api-revoke-sub-5fb235c4-gdbdz\" (UID: \"077fb11a-0be0-4b97-ad7b-c630ba538274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"
Oct 30 01:51:25.068596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:25.068567    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"
Oct 30 01:51:25.402713 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:25.402680    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"]
Oct 30 01:51:25.403992 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:51:25.403967    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077fb11a_0be0_4b97_ad7b_c630ba538274.slice/crio-f4e66da8082592c09d2594fb8224c4a647aebaee8d7af4075a8f495397b1b1ba WatchSource:0}: Error finding container f4e66da8082592c09d2594fb8224c4a647aebaee8d7af4075a8f495397b1b1ba: Status 404 returned error can't find the container with id f4e66da8082592c09d2594fb8224c4a647aebaee8d7af4075a8f495397b1b1ba
Oct 30 01:51:25.709376 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:25.709347    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz" event={"ID":"077fb11a-0be0-4b97-ad7b-c630ba538274","Type":"ContainerStarted","Data":"728a8ef813fce21f24670bb98806507e45ab6561619d42de81d9a83442d005d7"}
Oct 30 01:51:25.709507 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:25.709399    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz" event={"ID":"077fb11a-0be0-4b97-ad7b-c630ba538274","Type":"ContainerStarted","Data":"f4e66da8082592c09d2594fb8224c4a647aebaee8d7af4075a8f495397b1b1ba"}
Oct 30 01:51:25.748079 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:25.748030    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz" podStartSLOduration=1.748015544 podStartE2EDuration="1.748015544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:25.744784241 +0000 UTC m=+4194.494439549" watchObservedRunningTime="2026-10-30 01:51:25.748015544 +0000 UTC m=+4194.497670866"
Oct 30 01:51:26.719318 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:26.719281    4425 generic.go:356] "Generic (PLEG): container finished" podID="077fb11a-0be0-4b97-ad7b-c630ba538274" containerID="728a8ef813fce21f24670bb98806507e45ab6561619d42de81d9a83442d005d7" exitCode=0
Oct 30 01:51:26.719707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:26.719363    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz" event={"ID":"077fb11a-0be0-4b97-ad7b-c630ba538274","Type":"ContainerDied","Data":"728a8ef813fce21f24670bb98806507e45ab6561619d42de81d9a83442d005d7"}
Oct 30 01:51:27.879856 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:27.879828    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"
Oct 30 01:51:27.945588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:27.945550    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/077fb11a-0be0-4b97-ad7b-c630ba538274-kube-api-access-zgwdk\" (UniqueName: \"kubernetes.io/projected/077fb11a-0be0-4b97-ad7b-c630ba538274-kube-api-access-zgwdk\") pod \"077fb11a-0be0-4b97-ad7b-c630ba538274\" (UID: \"077fb11a-0be0-4b97-ad7b-c630ba538274\") "
Oct 30 01:51:27.945775 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:27.945649    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/077fb11a-0be0-4b97-ad7b-c630ba538274-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077fb11a-0be0-4b97-ad7b-c630ba538274-maas-api-xks-ca\") pod \"077fb11a-0be0-4b97-ad7b-c630ba538274\" (UID: \"077fb11a-0be0-4b97-ad7b-c630ba538274\") "
Oct 30 01:51:27.945775 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:27.945704    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/077fb11a-0be0-4b97-ad7b-c630ba538274-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077fb11a-0be0-4b97-ad7b-c630ba538274-maas-api-openshift-ca\") pod \"077fb11a-0be0-4b97-ad7b-c630ba538274\" (UID: \"077fb11a-0be0-4b97-ad7b-c630ba538274\") "
Oct 30 01:51:27.946574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:27.946545    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/077fb11a-0be0-4b97-ad7b-c630ba538274-maas-api-openshift-ca" pod "077fb11a-0be0-4b97-ad7b-c630ba538274" (UID: "077fb11a-0be0-4b97-ad7b-c630ba538274"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:51:27.948492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:27.948463    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077fb11a-0be0-4b97-ad7b-c630ba538274-kube-api-access-zgwdk" pod "077fb11a-0be0-4b97-ad7b-c630ba538274" (UID: "077fb11a-0be0-4b97-ad7b-c630ba538274"). InnerVolumeSpecName "kube-api-access-zgwdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:51:27.957587 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:27.957547    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/077fb11a-0be0-4b97-ad7b-c630ba538274-maas-api-xks-ca" pod "077fb11a-0be0-4b97-ad7b-c630ba538274" (UID: "077fb11a-0be0-4b97-ad7b-c630ba538274"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:51:28.047806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:28.047774    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgwdk\" (UniqueName: \"kubernetes.io/projected/077fb11a-0be0-4b97-ad7b-c630ba538274-kube-api-access-zgwdk\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:28.047806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:28.047804    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/077fb11a-0be0-4b97-ad7b-c630ba538274-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:28.048004 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:28.047815    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/077fb11a-0be0-4b97-ad7b-c630ba538274-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:28.735501 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:28.735471    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"
Oct 30 01:51:28.735674 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:28.735507    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz" event={"ID":"077fb11a-0be0-4b97-ad7b-c630ba538274","Type":"ContainerDied","Data":"f4e66da8082592c09d2594fb8224c4a647aebaee8d7af4075a8f495397b1b1ba"}
Oct 30 01:51:28.735674 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:28.735553    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4e66da8082592c09d2594fb8224c4a647aebaee8d7af4075a8f495397b1b1ba"
Oct 30 01:51:35.174097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.174062    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"]
Oct 30 01:51:35.183955 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.183919    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c7bba9-xspqq"]
Oct 30 01:51:35.250464 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.250431    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"]
Oct 30 01:51:35.251257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.251236    4425 cpu_manager.go:414] "containerMap: removing container" podUID="077fb11a-0be0-4b97-ad7b-c630ba538274" containerName="revoke-keys"
Oct 30 01:51:35.251257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.251257    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="077fb11a-0be0-4b97-ad7b-c630ba538274" containerName="revoke-keys"
Oct 30 01:51:35.251451 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.251417    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="077fb11a-0be0-4b97-ad7b-c630ba538274" containerName="revoke-keys"
Oct 30 01:51:35.255473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.255449    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"
Oct 30 01:51:35.268099 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.268079    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:51:35.268248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.268230    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:51:35.276963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.276941    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"]
Oct 30 01:51:35.334067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.334036    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed14e2e8-91df-43b2-8437-10fb194339ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84f1f1ae-ltgd5\" (UID: \"ed14e2e8-91df-43b2-8437-10fb194339ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"
Oct 30 01:51:35.334067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.334070    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnxst\" (UniqueName: \"kubernetes.io/projected/ed14e2e8-91df-43b2-8437-10fb194339ff-kube-api-access-pnxst\") pod \"maas-api-revoke-sub-84f1f1ae-ltgd5\" (UID: \"ed14e2e8-91df-43b2-8437-10fb194339ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"
Oct 30 01:51:35.334274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.334168    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed14e2e8-91df-43b2-8437-10fb194339ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84f1f1ae-ltgd5\" (UID: \"ed14e2e8-91df-43b2-8437-10fb194339ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"
Oct 30 01:51:35.425929 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.425866    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd86b83d-af82-4d9c-8acf-6285259f8b0d" path="/var/lib/kubelet/pods/bd86b83d-af82-4d9c-8acf-6285259f8b0d/volumes"
Oct 30 01:51:35.434850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.434825    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed14e2e8-91df-43b2-8437-10fb194339ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84f1f1ae-ltgd5\" (UID: \"ed14e2e8-91df-43b2-8437-10fb194339ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"
Oct 30 01:51:35.434968 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.434852    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnxst\" (UniqueName: \"kubernetes.io/projected/ed14e2e8-91df-43b2-8437-10fb194339ff-kube-api-access-pnxst\") pod \"maas-api-revoke-sub-84f1f1ae-ltgd5\" (UID: \"ed14e2e8-91df-43b2-8437-10fb194339ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"
Oct 30 01:51:35.434968 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.434910    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed14e2e8-91df-43b2-8437-10fb194339ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84f1f1ae-ltgd5\" (UID: \"ed14e2e8-91df-43b2-8437-10fb194339ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"
Oct 30 01:51:35.435632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.435610    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed14e2e8-91df-43b2-8437-10fb194339ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84f1f1ae-ltgd5\" (UID: \"ed14e2e8-91df-43b2-8437-10fb194339ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"
Oct 30 01:51:35.437196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.437148    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed14e2e8-91df-43b2-8437-10fb194339ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84f1f1ae-ltgd5\" (UID: \"ed14e2e8-91df-43b2-8437-10fb194339ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"
Oct 30 01:51:35.456028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.456004    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnxst\" (UniqueName: \"kubernetes.io/projected/ed14e2e8-91df-43b2-8437-10fb194339ff-kube-api-access-pnxst\") pod \"maas-api-revoke-sub-84f1f1ae-ltgd5\" (UID: \"ed14e2e8-91df-43b2-8437-10fb194339ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"
Oct 30 01:51:35.569851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.569829    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"
Oct 30 01:51:35.879920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:35.879893    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"]
Oct 30 01:51:35.880535 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:51:35.880510    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded14e2e8_91df_43b2_8437_10fb194339ff.slice/crio-6c39a7131789e310027c50fb95e9644023119df5a792db2d29d5de06a80c6b82 WatchSource:0}: Error finding container 6c39a7131789e310027c50fb95e9644023119df5a792db2d29d5de06a80c6b82: Status 404 returned error can't find the container with id 6c39a7131789e310027c50fb95e9644023119df5a792db2d29d5de06a80c6b82
Oct 30 01:51:36.445225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:36.445127    4425 scope.go:122] "RemoveContainer" containerID="df92f7278efdd867dbcbb341e6ff452c792f9784a505be7288a6a9b1d1600d14"
Oct 30 01:51:36.456970 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:36.456952    4425 scope.go:122] "RemoveContainer" containerID="ef96c8976cf2876ebd30e6be33071db26a63691403b4e0cd0c80c9321d5129e7"
Oct 30 01:51:36.466758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:36.466737    4425 scope.go:122] "RemoveContainer" containerID="8eff35400fe1e71bd83a23308f7a5cf60e0085f028586e97572ed2dce638449f"
Oct 30 01:51:36.476047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:36.476023    4425 scope.go:122] "RemoveContainer" containerID="b04f27da98b3ba48d91a9f179ef3e1856128bcee5ef942c055630e9a40259436"
Oct 30 01:51:36.805413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:36.805373    4425 generic.go:356] "Generic (PLEG): container finished" podID="ed14e2e8-91df-43b2-8437-10fb194339ff" containerID="ce8ca8abcf5212a3b61466152ea83c6a887e25f9dc203121064a1ed8f18522e4" exitCode=0
Oct 30 01:51:36.805587 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:36.805460    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5" event={"ID":"ed14e2e8-91df-43b2-8437-10fb194339ff","Type":"ContainerDied","Data":"ce8ca8abcf5212a3b61466152ea83c6a887e25f9dc203121064a1ed8f18522e4"}
Oct 30 01:51:36.805587 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:36.805508    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5" event={"ID":"ed14e2e8-91df-43b2-8437-10fb194339ff","Type":"ContainerStarted","Data":"6c39a7131789e310027c50fb95e9644023119df5a792db2d29d5de06a80c6b82"}
Oct 30 01:51:37.554951 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:51:37.554912    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:51:37.970889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:37.970861    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"
Oct 30 01:51:38.069878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:38.069784    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed14e2e8-91df-43b2-8437-10fb194339ff-kube-api-access-pnxst\" (UniqueName: \"kubernetes.io/projected/ed14e2e8-91df-43b2-8437-10fb194339ff-kube-api-access-pnxst\") pod \"ed14e2e8-91df-43b2-8437-10fb194339ff\" (UID: \"ed14e2e8-91df-43b2-8437-10fb194339ff\") "
Oct 30 01:51:38.070052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:38.069937    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed14e2e8-91df-43b2-8437-10fb194339ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed14e2e8-91df-43b2-8437-10fb194339ff-maas-api-xks-ca\") pod \"ed14e2e8-91df-43b2-8437-10fb194339ff\" (UID: \"ed14e2e8-91df-43b2-8437-10fb194339ff\") "
Oct 30 01:51:38.070110 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:38.070090    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed14e2e8-91df-43b2-8437-10fb194339ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed14e2e8-91df-43b2-8437-10fb194339ff-maas-api-openshift-ca\") pod \"ed14e2e8-91df-43b2-8437-10fb194339ff\" (UID: \"ed14e2e8-91df-43b2-8437-10fb194339ff\") "
Oct 30 01:51:38.070439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:38.070412    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed14e2e8-91df-43b2-8437-10fb194339ff-maas-api-openshift-ca" pod "ed14e2e8-91df-43b2-8437-10fb194339ff" (UID: "ed14e2e8-91df-43b2-8437-10fb194339ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:51:38.070654 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:38.070630    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed14e2e8-91df-43b2-8437-10fb194339ff-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:38.072148 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:38.072124    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed14e2e8-91df-43b2-8437-10fb194339ff-maas-api-xks-ca" pod "ed14e2e8-91df-43b2-8437-10fb194339ff" (UID: "ed14e2e8-91df-43b2-8437-10fb194339ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:51:38.072424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:38.072404    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed14e2e8-91df-43b2-8437-10fb194339ff-kube-api-access-pnxst" pod "ed14e2e8-91df-43b2-8437-10fb194339ff" (UID: "ed14e2e8-91df-43b2-8437-10fb194339ff"). InnerVolumeSpecName "kube-api-access-pnxst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:51:38.172322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:38.172296    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnxst\" (UniqueName: \"kubernetes.io/projected/ed14e2e8-91df-43b2-8437-10fb194339ff-kube-api-access-pnxst\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:38.172322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:38.172320    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed14e2e8-91df-43b2-8437-10fb194339ff-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:38.823594 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:38.823561    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5" event={"ID":"ed14e2e8-91df-43b2-8437-10fb194339ff","Type":"ContainerDied","Data":"6c39a7131789e310027c50fb95e9644023119df5a792db2d29d5de06a80c6b82"}
Oct 30 01:51:38.824004 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:38.823614    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c39a7131789e310027c50fb95e9644023119df5a792db2d29d5de06a80c6b82"
Oct 30 01:51:38.824004 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:38.823656    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"
Oct 30 01:51:44.475335 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.475298    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"]
Oct 30 01:51:44.476033 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.476018    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ed14e2e8-91df-43b2-8437-10fb194339ff" containerName="revoke-keys"
Oct 30 01:51:44.476077 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.476034    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed14e2e8-91df-43b2-8437-10fb194339ff" containerName="revoke-keys"
Oct 30 01:51:44.476170 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.476148    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed14e2e8-91df-43b2-8437-10fb194339ff" containerName="revoke-keys"
Oct 30 01:51:44.479853 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.479836    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"
Oct 30 01:51:44.498693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.498667    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"]
Oct 30 01:51:44.554978 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.554953    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad21dae-c26b-4f28-a7db-3cde934cbeed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx\" (UID: \"3ad21dae-c26b-4f28-a7db-3cde934cbeed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"
Oct 30 01:51:44.555132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.555068    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad21dae-c26b-4f28-a7db-3cde934cbeed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx\" (UID: \"3ad21dae-c26b-4f28-a7db-3cde934cbeed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"
Oct 30 01:51:44.555272 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.555256    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nllbk\" (UniqueName: \"kubernetes.io/projected/3ad21dae-c26b-4f28-a7db-3cde934cbeed-kube-api-access-nllbk\") pod \"maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx\" (UID: \"3ad21dae-c26b-4f28-a7db-3cde934cbeed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"
Oct 30 01:51:44.564495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.564466    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:51:44.564608 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.564524    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:51:44.656235 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.656199    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nllbk\" (UniqueName: \"kubernetes.io/projected/3ad21dae-c26b-4f28-a7db-3cde934cbeed-kube-api-access-nllbk\") pod \"maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx\" (UID: \"3ad21dae-c26b-4f28-a7db-3cde934cbeed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"
Oct 30 01:51:44.656435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.656302    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad21dae-c26b-4f28-a7db-3cde934cbeed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx\" (UID: \"3ad21dae-c26b-4f28-a7db-3cde934cbeed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"
Oct 30 01:51:44.656435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.656403    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad21dae-c26b-4f28-a7db-3cde934cbeed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx\" (UID: \"3ad21dae-c26b-4f28-a7db-3cde934cbeed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"
Oct 30 01:51:44.657346 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.657316    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad21dae-c26b-4f28-a7db-3cde934cbeed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx\" (UID: \"3ad21dae-c26b-4f28-a7db-3cde934cbeed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"
Oct 30 01:51:44.659651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.659612    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad21dae-c26b-4f28-a7db-3cde934cbeed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx\" (UID: \"3ad21dae-c26b-4f28-a7db-3cde934cbeed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"
Oct 30 01:51:44.680060 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.680025    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nllbk\" (UniqueName: \"kubernetes.io/projected/3ad21dae-c26b-4f28-a7db-3cde934cbeed-kube-api-access-nllbk\") pod \"maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx\" (UID: \"3ad21dae-c26b-4f28-a7db-3cde934cbeed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"
Oct 30 01:51:44.794134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:44.794102    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"
Oct 30 01:51:45.099784 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:45.099756    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"]
Oct 30 01:51:45.100704 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:51:45.100673    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad21dae_c26b_4f28_a7db_3cde934cbeed.slice/crio-0fc1932babdb65c5c12c1fbe3966e70abb831dd2e46a4e5e71eb2954aea4de3d WatchSource:0}: Error finding container 0fc1932babdb65c5c12c1fbe3966e70abb831dd2e46a4e5e71eb2954aea4de3d: Status 404 returned error can't find the container with id 0fc1932babdb65c5c12c1fbe3966e70abb831dd2e46a4e5e71eb2954aea4de3d
Oct 30 01:51:45.892104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:45.892058    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx" event={"ID":"3ad21dae-c26b-4f28-a7db-3cde934cbeed","Type":"ContainerStarted","Data":"506b3f4f4cc565683d558a81d718d65cdb7f3f447c63513ece5a91d3be9e8b88"}
Oct 30 01:51:45.892503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:45.892188    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx" event={"ID":"3ad21dae-c26b-4f28-a7db-3cde934cbeed","Type":"ContainerStarted","Data":"0fc1932babdb65c5c12c1fbe3966e70abb831dd2e46a4e5e71eb2954aea4de3d"}
Oct 30 01:51:45.930964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:45.930917    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx" podStartSLOduration=1.930906116 podStartE2EDuration="1.930906116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:45.930339047 +0000 UTC m=+4214.679994355" watchObservedRunningTime="2026-10-30 01:51:45.930906116 +0000 UTC m=+4214.680561423"
Oct 30 01:51:46.241993 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:46.241907    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"]
Oct 30 01:51:46.902239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:46.902205    4425 generic.go:356] "Generic (PLEG): container finished" podID="3ad21dae-c26b-4f28-a7db-3cde934cbeed" containerID="506b3f4f4cc565683d558a81d718d65cdb7f3f447c63513ece5a91d3be9e8b88" exitCode=0
Oct 30 01:51:46.902681 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:46.902300    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx" event={"ID":"3ad21dae-c26b-4f28-a7db-3cde934cbeed","Type":"ContainerDied","Data":"506b3f4f4cc565683d558a81d718d65cdb7f3f447c63513ece5a91d3be9e8b88"}
Oct 30 01:51:48.067107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:48.067080    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"
Oct 30 01:51:48.100642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:48.100616    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad21dae-c26b-4f28-a7db-3cde934cbeed-kube-api-access-nllbk\" (UniqueName: \"kubernetes.io/projected/3ad21dae-c26b-4f28-a7db-3cde934cbeed-kube-api-access-nllbk\") pod \"3ad21dae-c26b-4f28-a7db-3cde934cbeed\" (UID: \"3ad21dae-c26b-4f28-a7db-3cde934cbeed\") "
Oct 30 01:51:48.100805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:48.100657    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ad21dae-c26b-4f28-a7db-3cde934cbeed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad21dae-c26b-4f28-a7db-3cde934cbeed-maas-api-xks-ca\") pod \"3ad21dae-c26b-4f28-a7db-3cde934cbeed\" (UID: \"3ad21dae-c26b-4f28-a7db-3cde934cbeed\") "
Oct 30 01:51:48.100805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:48.100736    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ad21dae-c26b-4f28-a7db-3cde934cbeed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad21dae-c26b-4f28-a7db-3cde934cbeed-maas-api-openshift-ca\") pod \"3ad21dae-c26b-4f28-a7db-3cde934cbeed\" (UID: \"3ad21dae-c26b-4f28-a7db-3cde934cbeed\") "
Oct 30 01:51:48.101218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:48.101144    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ad21dae-c26b-4f28-a7db-3cde934cbeed-maas-api-openshift-ca" pod "3ad21dae-c26b-4f28-a7db-3cde934cbeed" (UID: "3ad21dae-c26b-4f28-a7db-3cde934cbeed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:51:48.102768 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:48.102741    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ad21dae-c26b-4f28-a7db-3cde934cbeed-maas-api-xks-ca" pod "3ad21dae-c26b-4f28-a7db-3cde934cbeed" (UID: "3ad21dae-c26b-4f28-a7db-3cde934cbeed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:51:48.103064 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:48.103039    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad21dae-c26b-4f28-a7db-3cde934cbeed-kube-api-access-nllbk" pod "3ad21dae-c26b-4f28-a7db-3cde934cbeed" (UID: "3ad21dae-c26b-4f28-a7db-3cde934cbeed"). InnerVolumeSpecName "kube-api-access-nllbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:51:48.203009 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:48.202938    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nllbk\" (UniqueName: \"kubernetes.io/projected/3ad21dae-c26b-4f28-a7db-3cde934cbeed-kube-api-access-nllbk\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:48.203009 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:48.202964    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad21dae-c26b-4f28-a7db-3cde934cbeed-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:48.203009 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:48.202974    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad21dae-c26b-4f28-a7db-3cde934cbeed-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:51:48.918711 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:48.918683    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"
Oct 30 01:51:48.918920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:48.918719    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx" event={"ID":"3ad21dae-c26b-4f28-a7db-3cde934cbeed","Type":"ContainerDied","Data":"0fc1932babdb65c5c12c1fbe3966e70abb831dd2e46a4e5e71eb2954aea4de3d"}
Oct 30 01:51:48.918920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:48.918773    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fc1932babdb65c5c12c1fbe3966e70abb831dd2e46a4e5e71eb2954aea4de3d"
Oct 30 01:51:54.858036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:54.857993    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"]
Oct 30 01:51:54.885910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:54.885878    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"]
Oct 30 01:51:59.858467 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:59.858426    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb" podUID="9cab6c8f-afe1-4908-8aa8-e533837c609a" containerName="payload-processing" containerID="cri-o://af25bf99e691dc67db0ddaab693b76cc2e98f3ebe0c9e2566dbbce9f600a976c" gracePeriod=25
Oct 30 01:51:59.886384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:51:59.886349    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r" podUID="817e5cef-adec-43f8-8d49-439e1379d1be" containerName="payload-pre-processing" containerID="cri-o://3a9db16678e97e6a6ea17db2d4149cdb59021401bd2c465958c71cad8e956e36" gracePeriod=25
Oct 30 01:52:01.266966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.266921    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z" podUID="7ab1823f-9005-407f-b702-905853eb61b8" containerName="main" containerID="cri-o://ae507c0d102b4a1be60631414249016ace83263b22fef271a6f75a5d5ec40a4d" gracePeriod=45
Oct 30 01:52:01.537065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.537006    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:52:01.672109 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.672077    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-tmp-dir\") pod \"7ab1823f-9005-407f-b702-905853eb61b8\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") "
Oct 30 01:52:01.672362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.672149    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ab1823f-9005-407f-b702-905853eb61b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ab1823f-9005-407f-b702-905853eb61b8-tls-certs\") pod \"7ab1823f-9005-407f-b702-905853eb61b8\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") "
Oct 30 01:52:01.672362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.672254    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-dshm\") pod \"7ab1823f-9005-407f-b702-905853eb61b8\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") "
Oct 30 01:52:01.672362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.672290    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-model-cache\") pod \"7ab1823f-9005-407f-b702-905853eb61b8\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") "
Oct 30 01:52:01.672362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.672354    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-home\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-home\") pod \"7ab1823f-9005-407f-b702-905853eb61b8\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") "
Oct 30 01:52:01.672607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.672391    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab1823f-9005-407f-b702-905853eb61b8-kube-api-access-vprns\" (UniqueName: \"kubernetes.io/projected/7ab1823f-9005-407f-b702-905853eb61b8-kube-api-access-vprns\") pod \"7ab1823f-9005-407f-b702-905853eb61b8\" (UID: \"7ab1823f-9005-407f-b702-905853eb61b8\") "
Oct 30 01:52:01.672607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.672409    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-tmp-dir" pod "7ab1823f-9005-407f-b702-905853eb61b8" (UID: "7ab1823f-9005-407f-b702-905853eb61b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:52:01.672685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.672625    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-home" pod "7ab1823f-9005-407f-b702-905853eb61b8" (UID: "7ab1823f-9005-407f-b702-905853eb61b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:52:01.672685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.672638    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-model-cache" pod "7ab1823f-9005-407f-b702-905853eb61b8" (UID: "7ab1823f-9005-407f-b702-905853eb61b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:52:01.673050 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.673029    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:01.673050 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.673048    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:01.673271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.673059    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:01.674310 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.674286    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-dshm" pod "7ab1823f-9005-407f-b702-905853eb61b8" (UID: "7ab1823f-9005-407f-b702-905853eb61b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:52:01.674420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.674349    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab1823f-9005-407f-b702-905853eb61b8-tls-certs" pod "7ab1823f-9005-407f-b702-905853eb61b8" (UID: "7ab1823f-9005-407f-b702-905853eb61b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:52:01.674468 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.674454    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab1823f-9005-407f-b702-905853eb61b8-kube-api-access-vprns" pod "7ab1823f-9005-407f-b702-905853eb61b8" (UID: "7ab1823f-9005-407f-b702-905853eb61b8"). InnerVolumeSpecName "kube-api-access-vprns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:01.774308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.774280    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ab1823f-9005-407f-b702-905853eb61b8-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:01.774308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.774304    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ab1823f-9005-407f-b702-905853eb61b8-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:01.774462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:01.774312    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vprns\" (UniqueName: \"kubernetes.io/projected/7ab1823f-9005-407f-b702-905853eb61b8-kube-api-access-vprns\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:02.018588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:02.018553    4425 generic.go:356] "Generic (PLEG): container finished" podID="7ab1823f-9005-407f-b702-905853eb61b8" containerID="ae507c0d102b4a1be60631414249016ace83263b22fef271a6f75a5d5ec40a4d" exitCode=0
Oct 30 01:52:02.018752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:02.018601    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"
Oct 30 01:52:02.018752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:02.018634    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z" event={"ID":"7ab1823f-9005-407f-b702-905853eb61b8","Type":"ContainerDied","Data":"ae507c0d102b4a1be60631414249016ace83263b22fef271a6f75a5d5ec40a4d"}
Oct 30 01:52:02.018752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:02.018678    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z" event={"ID":"7ab1823f-9005-407f-b702-905853eb61b8","Type":"ContainerDied","Data":"9d02e6c0d5b6f37b395b83b3137adf9fbac95f5710fa77d102ee4c0bcd690613"}
Oct 30 01:52:02.018752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:02.018698    4425 scope.go:122] "RemoveContainer" containerID="ae507c0d102b4a1be60631414249016ace83263b22fef271a6f75a5d5ec40a4d"
Oct 30 01:52:02.029827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:02.029809    4425 scope.go:122] "RemoveContainer" containerID="ae507c0d102b4a1be60631414249016ace83263b22fef271a6f75a5d5ec40a4d"
Oct 30 01:52:02.030075 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:52:02.030060    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae507c0d102b4a1be60631414249016ace83263b22fef271a6f75a5d5ec40a4d\": container with ID starting with ae507c0d102b4a1be60631414249016ace83263b22fef271a6f75a5d5ec40a4d not found: ID does not exist" containerID="ae507c0d102b4a1be60631414249016ace83263b22fef271a6f75a5d5ec40a4d"
Oct 30 01:52:02.030119 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:02.030084    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae507c0d102b4a1be60631414249016ace83263b22fef271a6f75a5d5ec40a4d"} err="failed to get container status \"ae507c0d102b4a1be60631414249016ace83263b22fef271a6f75a5d5ec40a4d\": rpc error: code = NotFound desc = could not find container \"ae507c0d102b4a1be60631414249016ace83263b22fef271a6f75a5d5ec40a4d\": container with ID starting with ae507c0d102b4a1be60631414249016ace83263b22fef271a6f75a5d5ec40a4d not found: ID does not exist"
Oct 30 01:52:02.064175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:02.064131    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"]
Oct 30 01:52:02.073867 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:02.073845    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-0e1a98/sub-test-model-0e1a98-kserve-685b77dbb5-8r66z"]
Oct 30 01:52:03.057706 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:03.057670    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:52:03.424037 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:03.423963    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab1823f-9005-407f-b702-905853eb61b8" path="/var/lib/kubelet/pods/7ab1823f-9005-407f-b702-905853eb61b8/volumes"
Oct 30 01:52:03.461352 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:03.461318    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r" podUID="817e5cef-adec-43f8-8d49-439e1379d1be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:52:03.461498 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:03.461325    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb" podUID="9cab6c8f-afe1-4908-8aa8-e533837c609a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:52:05.644275 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:05.644237    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"]
Oct 30 01:52:05.911634 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:05.911549    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"]
Oct 30 01:52:05.912554 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:05.912525    4425 cpu_manager.go:414] "containerMap: removing container" podUID="3ad21dae-c26b-4f28-a7db-3cde934cbeed" containerName="revoke-keys"
Oct 30 01:52:05.912554 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:05.912550    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad21dae-c26b-4f28-a7db-3cde934cbeed" containerName="revoke-keys"
Oct 30 01:52:05.912769 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:05.912599    4425 cpu_manager.go:414] "containerMap: removing container" podUID="7ab1823f-9005-407f-b702-905853eb61b8" containerName="main"
Oct 30 01:52:05.912769 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:05.912607    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab1823f-9005-407f-b702-905853eb61b8" containerName="main"
Oct 30 01:52:05.912879 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:05.912807    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad21dae-c26b-4f28-a7db-3cde934cbeed" containerName="revoke-keys"
Oct 30 01:52:05.912879 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:05.912827    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab1823f-9005-407f-b702-905853eb61b8" containerName="main"
Oct 30 01:52:05.918355 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:05.918335    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"
Oct 30 01:52:05.932976 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:05.932951    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:52:05.933085 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:05.932974    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:52:05.936585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:05.936562    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"]
Oct 30 01:52:06.023631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:06.023599    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-499482-5llnm\" (UID: \"60b3f3a3-731a-48dc-9f75-7c7de0b3b317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"
Oct 30 01:52:06.023792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:06.023658    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-499482-5llnm\" (UID: \"60b3f3a3-731a-48dc-9f75-7c7de0b3b317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"
Oct 30 01:52:06.023860 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:06.023840    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n86c6\" (UniqueName: \"kubernetes.io/projected/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-kube-api-access-n86c6\") pod \"maas-api-revoke-keys-e2e-extmod-a-499482-5llnm\" (UID: \"60b3f3a3-731a-48dc-9f75-7c7de0b3b317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"
Oct 30 01:52:06.125501 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:06.125464    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-499482-5llnm\" (UID: \"60b3f3a3-731a-48dc-9f75-7c7de0b3b317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"
Oct 30 01:52:06.125694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:06.125535    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-499482-5llnm\" (UID: \"60b3f3a3-731a-48dc-9f75-7c7de0b3b317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"
Oct 30 01:52:06.125694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:06.125681    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n86c6\" (UniqueName: \"kubernetes.io/projected/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-kube-api-access-n86c6\") pod \"maas-api-revoke-keys-e2e-extmod-a-499482-5llnm\" (UID: \"60b3f3a3-731a-48dc-9f75-7c7de0b3b317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"
Oct 30 01:52:06.126579 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:06.126544    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-499482-5llnm\" (UID: \"60b3f3a3-731a-48dc-9f75-7c7de0b3b317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"
Oct 30 01:52:06.128688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:06.128661    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-499482-5llnm\" (UID: \"60b3f3a3-731a-48dc-9f75-7c7de0b3b317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"
Oct 30 01:52:06.146931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:06.146903    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n86c6\" (UniqueName: \"kubernetes.io/projected/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-kube-api-access-n86c6\") pod \"maas-api-revoke-keys-e2e-extmod-a-499482-5llnm\" (UID: \"60b3f3a3-731a-48dc-9f75-7c7de0b3b317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"
Oct 30 01:52:06.235669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:06.235594    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"
Oct 30 01:52:06.557035 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:06.557008    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"]
Oct 30 01:52:06.559101 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:52:06.559071    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b3f3a3_731a_48dc_9f75_7c7de0b3b317.slice/crio-c0bc9584c12f2dba5470d5bc17e7f6dc9c6c8e8513457d9697b606dc192b32ef WatchSource:0}: Error finding container c0bc9584c12f2dba5470d5bc17e7f6dc9c6c8e8513457d9697b606dc192b32ef: Status 404 returned error can't find the container with id c0bc9584c12f2dba5470d5bc17e7f6dc9c6c8e8513457d9697b606dc192b32ef
Oct 30 01:52:07.064669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:07.064636    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm" event={"ID":"60b3f3a3-731a-48dc-9f75-7c7de0b3b317","Type":"ContainerStarted","Data":"1736a92513b42c80bd7d5c427e144d5eae9cf516939dde412c1e05593df37a2c"}
Oct 30 01:52:07.065070 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:07.064689    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm" event={"ID":"60b3f3a3-731a-48dc-9f75-7c7de0b3b317","Type":"ContainerStarted","Data":"c0bc9584c12f2dba5470d5bc17e7f6dc9c6c8e8513457d9697b606dc192b32ef"}
Oct 30 01:52:07.106924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:07.106863    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm" podStartSLOduration=2.106846093 podStartE2EDuration="2.106846093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:07.100560984 +0000 UTC m=+4235.850216288" watchObservedRunningTime="2026-10-30 01:52:07.106846093 +0000 UTC m=+4235.856501402"
Oct 30 01:52:07.558633 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:52:07.558589    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:52:08.073813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:08.073767    4425 generic.go:356] "Generic (PLEG): container finished" podID="60b3f3a3-731a-48dc-9f75-7c7de0b3b317" containerID="1736a92513b42c80bd7d5c427e144d5eae9cf516939dde412c1e05593df37a2c" exitCode=0
Oct 30 01:52:08.074208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:08.073838    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm" event={"ID":"60b3f3a3-731a-48dc-9f75-7c7de0b3b317","Type":"ContainerDied","Data":"1736a92513b42c80bd7d5c427e144d5eae9cf516939dde412c1e05593df37a2c"}
Oct 30 01:52:09.242616 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:09.242595    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"
Oct 30 01:52:09.264172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:09.264131    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-kube-api-access-n86c6\" (UniqueName: \"kubernetes.io/projected/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-kube-api-access-n86c6\") pod \"60b3f3a3-731a-48dc-9f75-7c7de0b3b317\" (UID: \"60b3f3a3-731a-48dc-9f75-7c7de0b3b317\") "
Oct 30 01:52:09.264304 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:09.264176    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-maas-api-xks-ca\") pod \"60b3f3a3-731a-48dc-9f75-7c7de0b3b317\" (UID: \"60b3f3a3-731a-48dc-9f75-7c7de0b3b317\") "
Oct 30 01:52:09.264304 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:09.264209    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-maas-api-openshift-ca\") pod \"60b3f3a3-731a-48dc-9f75-7c7de0b3b317\" (UID: \"60b3f3a3-731a-48dc-9f75-7c7de0b3b317\") "
Oct 30 01:52:09.264602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:09.264579    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-maas-api-openshift-ca" pod "60b3f3a3-731a-48dc-9f75-7c7de0b3b317" (UID: "60b3f3a3-731a-48dc-9f75-7c7de0b3b317"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:09.266147 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:09.266126    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-maas-api-xks-ca" pod "60b3f3a3-731a-48dc-9f75-7c7de0b3b317" (UID: "60b3f3a3-731a-48dc-9f75-7c7de0b3b317"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:52:09.266315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:09.266291    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-kube-api-access-n86c6" pod "60b3f3a3-731a-48dc-9f75-7c7de0b3b317" (UID: "60b3f3a3-731a-48dc-9f75-7c7de0b3b317"). InnerVolumeSpecName "kube-api-access-n86c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:09.365362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:09.365290    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n86c6\" (UniqueName: \"kubernetes.io/projected/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-kube-api-access-n86c6\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:09.365362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:09.365320    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:09.365362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:09.365331    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b3f3a3-731a-48dc-9f75-7c7de0b3b317-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:10.093049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:10.093015    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm" event={"ID":"60b3f3a3-731a-48dc-9f75-7c7de0b3b317","Type":"ContainerDied","Data":"c0bc9584c12f2dba5470d5bc17e7f6dc9c6c8e8513457d9697b606dc192b32ef"}
Oct 30 01:52:10.093049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:10.093036    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"
Oct 30 01:52:10.093305 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:10.093066    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0bc9584c12f2dba5470d5bc17e7f6dc9c6c8e8513457d9697b606dc192b32ef"
Oct 30 01:52:11.366801 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.366770    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"]
Oct 30 01:52:11.367521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.367505    4425 cpu_manager.go:414] "containerMap: removing container" podUID="60b3f3a3-731a-48dc-9f75-7c7de0b3b317" containerName="revoke-keys"
Oct 30 01:52:11.367577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.367522    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b3f3a3-731a-48dc-9f75-7c7de0b3b317" containerName="revoke-keys"
Oct 30 01:52:11.367632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.367624    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b3f3a3-731a-48dc-9f75-7c7de0b3b317" containerName="revoke-keys"
Oct 30 01:52:11.371146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.371126    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"
Oct 30 01:52:11.384382 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.384358    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjk59\" (UniqueName: \"kubernetes.io/projected/d6019518-5851-47d1-8c48-c4c697264d32-kube-api-access-cjk59\") pod \"maas-api-revoke-sub-56c5bfd2-j7xsz\" (UID: \"d6019518-5851-47d1-8c48-c4c697264d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"
Oct 30 01:52:11.384503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.384478    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6019518-5851-47d1-8c48-c4c697264d32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56c5bfd2-j7xsz\" (UID: \"d6019518-5851-47d1-8c48-c4c697264d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"
Oct 30 01:52:11.384503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.384501    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6019518-5851-47d1-8c48-c4c697264d32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56c5bfd2-j7xsz\" (UID: \"d6019518-5851-47d1-8c48-c4c697264d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"
Oct 30 01:52:11.384640 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.384623    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:52:11.394347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.394324    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:52:11.398088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.398064    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"]
Oct 30 01:52:11.485484 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.485456    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjk59\" (UniqueName: \"kubernetes.io/projected/d6019518-5851-47d1-8c48-c4c697264d32-kube-api-access-cjk59\") pod \"maas-api-revoke-sub-56c5bfd2-j7xsz\" (UID: \"d6019518-5851-47d1-8c48-c4c697264d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"
Oct 30 01:52:11.485624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.485610    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6019518-5851-47d1-8c48-c4c697264d32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56c5bfd2-j7xsz\" (UID: \"d6019518-5851-47d1-8c48-c4c697264d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"
Oct 30 01:52:11.485684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.485633    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6019518-5851-47d1-8c48-c4c697264d32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56c5bfd2-j7xsz\" (UID: \"d6019518-5851-47d1-8c48-c4c697264d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"
Oct 30 01:52:11.486323 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.486294    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6019518-5851-47d1-8c48-c4c697264d32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56c5bfd2-j7xsz\" (UID: \"d6019518-5851-47d1-8c48-c4c697264d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"
Oct 30 01:52:11.487950 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.487928    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6019518-5851-47d1-8c48-c4c697264d32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56c5bfd2-j7xsz\" (UID: \"d6019518-5851-47d1-8c48-c4c697264d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"
Oct 30 01:52:11.506652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.506628    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjk59\" (UniqueName: \"kubernetes.io/projected/d6019518-5851-47d1-8c48-c4c697264d32-kube-api-access-cjk59\") pod \"maas-api-revoke-sub-56c5bfd2-j7xsz\" (UID: \"d6019518-5851-47d1-8c48-c4c697264d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"
Oct 30 01:52:11.685297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.685227    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"
Oct 30 01:52:11.990190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:11.990144    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"]
Oct 30 01:52:11.991403 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:52:11.991376    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6019518_5851_47d1_8c48_c4c697264d32.slice/crio-ac6b0e728cb0b7c55fc49438fba891de4183d69e7dbb2d1980c48c5498955d79 WatchSource:0}: Error finding container ac6b0e728cb0b7c55fc49438fba891de4183d69e7dbb2d1980c48c5498955d79: Status 404 returned error can't find the container with id ac6b0e728cb0b7c55fc49438fba891de4183d69e7dbb2d1980c48c5498955d79
Oct 30 01:52:12.111430 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:12.111399    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz" event={"ID":"d6019518-5851-47d1-8c48-c4c697264d32","Type":"ContainerStarted","Data":"ac6b0e728cb0b7c55fc49438fba891de4183d69e7dbb2d1980c48c5498955d79"}
Oct 30 01:52:13.120928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:13.120895    4425 generic.go:356] "Generic (PLEG): container finished" podID="d6019518-5851-47d1-8c48-c4c697264d32" containerID="e97588cd3bd5fb75b3e5cb30149ee59b2c1b38a6a266c0a43489f720233a8076" exitCode=0
Oct 30 01:52:13.121372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:13.120938    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz" event={"ID":"d6019518-5851-47d1-8c48-c4c697264d32","Type":"ContainerDied","Data":"e97588cd3bd5fb75b3e5cb30149ee59b2c1b38a6a266c0a43489f720233a8076"}
Oct 30 01:52:13.461054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:13.461019    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb" podUID="9cab6c8f-afe1-4908-8aa8-e533837c609a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:52:13.461257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:13.461064    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r" podUID="817e5cef-adec-43f8-8d49-439e1379d1be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:52:13.629550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:13.629517    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"]
Oct 30 01:52:13.634010 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:13.633985    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"
Oct 30 01:52:13.692747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:13.692718    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"]
Oct 30 01:52:13.713350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:13.713287    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s5qq\" (UniqueName: \"kubernetes.io/projected/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-kube-api-access-8s5qq\") pod \"maas-api-revoke-sub-b05436d4-gmdz5\" (UID: \"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"
Oct 30 01:52:13.713483 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:13.713350    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b05436d4-gmdz5\" (UID: \"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"
Oct 30 01:52:13.713618 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:13.713595    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b05436d4-gmdz5\" (UID: \"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"
Oct 30 01:52:13.814591 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:13.814551    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b05436d4-gmdz5\" (UID: \"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"
Oct 30 01:52:13.814755 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:13.814646    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s5qq\" (UniqueName: \"kubernetes.io/projected/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-kube-api-access-8s5qq\") pod \"maas-api-revoke-sub-b05436d4-gmdz5\" (UID: \"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"
Oct 30 01:52:13.814755 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:13.814690    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b05436d4-gmdz5\" (UID: \"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"
Oct 30 01:52:13.815428 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:13.815405    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b05436d4-gmdz5\" (UID: \"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"
Oct 30 01:52:13.816916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:13.816894    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b05436d4-gmdz5\" (UID: \"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"
Oct 30 01:52:13.839357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:13.839333    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s5qq\" (UniqueName: \"kubernetes.io/projected/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-kube-api-access-8s5qq\") pod \"maas-api-revoke-sub-b05436d4-gmdz5\" (UID: \"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"
Oct 30 01:52:13.948571 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:13.948541    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"
Oct 30 01:52:14.255699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:14.255667    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"]
Oct 30 01:52:14.256080 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:52:14.256049    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeefe3b1e_3c62_47c6_a2e5_a6ca8ec197ba.slice/crio-31d99c8cffcd4a7641fadd0bddb7664a3dc31ac7734fb7df5534d2b658ceba3a WatchSource:0}: Error finding container 31d99c8cffcd4a7641fadd0bddb7664a3dc31ac7734fb7df5534d2b658ceba3a: Status 404 returned error can't find the container with id 31d99c8cffcd4a7641fadd0bddb7664a3dc31ac7734fb7df5534d2b658ceba3a
Oct 30 01:52:14.278462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:14.278436    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"
Oct 30 01:52:14.325193 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:14.325141    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6019518-5851-47d1-8c48-c4c697264d32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6019518-5851-47d1-8c48-c4c697264d32-maas-api-xks-ca\") pod \"d6019518-5851-47d1-8c48-c4c697264d32\" (UID: \"d6019518-5851-47d1-8c48-c4c697264d32\") "
Oct 30 01:52:14.325534 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:14.325508    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6019518-5851-47d1-8c48-c4c697264d32-kube-api-access-cjk59\" (UniqueName: \"kubernetes.io/projected/d6019518-5851-47d1-8c48-c4c697264d32-kube-api-access-cjk59\") pod \"d6019518-5851-47d1-8c48-c4c697264d32\" (UID: \"d6019518-5851-47d1-8c48-c4c697264d32\") "
Oct 30 01:52:14.325910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:14.325573    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6019518-5851-47d1-8c48-c4c697264d32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6019518-5851-47d1-8c48-c4c697264d32-maas-api-openshift-ca\") pod \"d6019518-5851-47d1-8c48-c4c697264d32\" (UID: \"d6019518-5851-47d1-8c48-c4c697264d32\") "
Oct 30 01:52:14.326396 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:14.326354    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6019518-5851-47d1-8c48-c4c697264d32-maas-api-openshift-ca" pod "d6019518-5851-47d1-8c48-c4c697264d32" (UID: "d6019518-5851-47d1-8c48-c4c697264d32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:14.327769 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:14.327739    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6019518-5851-47d1-8c48-c4c697264d32-maas-api-xks-ca" pod "d6019518-5851-47d1-8c48-c4c697264d32" (UID: "d6019518-5851-47d1-8c48-c4c697264d32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:52:14.327993 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:14.327966    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6019518-5851-47d1-8c48-c4c697264d32-kube-api-access-cjk59" pod "d6019518-5851-47d1-8c48-c4c697264d32" (UID: "d6019518-5851-47d1-8c48-c4c697264d32"). InnerVolumeSpecName "kube-api-access-cjk59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:14.427808 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:14.427773    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6019518-5851-47d1-8c48-c4c697264d32-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:14.427808 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:14.427801    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjk59\" (UniqueName: \"kubernetes.io/projected/d6019518-5851-47d1-8c48-c4c697264d32-kube-api-access-cjk59\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:14.427808 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:14.427813    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6019518-5851-47d1-8c48-c4c697264d32-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:15.139846 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:15.139769    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"
Oct 30 01:52:15.139846 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:15.139811    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz" event={"ID":"d6019518-5851-47d1-8c48-c4c697264d32","Type":"ContainerDied","Data":"ac6b0e728cb0b7c55fc49438fba891de4183d69e7dbb2d1980c48c5498955d79"}
Oct 30 01:52:15.140086 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:15.139862    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac6b0e728cb0b7c55fc49438fba891de4183d69e7dbb2d1980c48c5498955d79"
Oct 30 01:52:15.141520 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:15.141494    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5" event={"ID":"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba","Type":"ContainerStarted","Data":"28bc8cdd986cdfe4ac61a2e911f949c77e289bd58a23487c3bdde0fbcfd2db06"}
Oct 30 01:52:15.141661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:15.141534    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5" event={"ID":"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba","Type":"ContainerStarted","Data":"31d99c8cffcd4a7641fadd0bddb7664a3dc31ac7734fb7df5534d2b658ceba3a"}
Oct 30 01:52:15.180428 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:15.180389    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5" podStartSLOduration=2.180377966 podStartE2EDuration="2.180377966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:15.179649031 +0000 UTC m=+4243.929304338" watchObservedRunningTime="2026-10-30 01:52:15.180377966 +0000 UTC m=+4243.930033286"
Oct 30 01:52:16.150589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:16.150553    4425 generic.go:356] "Generic (PLEG): container finished" podID="eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba" containerID="28bc8cdd986cdfe4ac61a2e911f949c77e289bd58a23487c3bdde0fbcfd2db06" exitCode=0
Oct 30 01:52:16.150589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:16.150578    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5" event={"ID":"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba","Type":"ContainerDied","Data":"28bc8cdd986cdfe4ac61a2e911f949c77e289bd58a23487c3bdde0fbcfd2db06"}
Oct 30 01:52:16.332088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:16.332051    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"]
Oct 30 01:52:16.351973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:16.351936    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"]
Oct 30 01:52:17.315193 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:17.315147    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"
Oct 30 01:52:17.360041 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:17.360012    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-kube-api-access-8s5qq\" (UniqueName: \"kubernetes.io/projected/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-kube-api-access-8s5qq\") pod \"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba\" (UID: \"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba\") "
Oct 30 01:52:17.360249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:17.360068    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-maas-api-xks-ca\") pod \"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba\" (UID: \"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba\") "
Oct 30 01:52:17.360327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:17.360271    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-maas-api-openshift-ca\") pod \"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba\" (UID: \"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba\") "
Oct 30 01:52:17.360650 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:17.360624    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-maas-api-openshift-ca" pod "eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba" (UID: "eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:17.360776 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:17.360760    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:17.362301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:17.362276    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-maas-api-xks-ca" pod "eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba" (UID: "eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:52:17.362477 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:17.362451    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-kube-api-access-8s5qq" pod "eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba" (UID: "eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba"). InnerVolumeSpecName "kube-api-access-8s5qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:17.462413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:17.462380    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s5qq\" (UniqueName: \"kubernetes.io/projected/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-kube-api-access-8s5qq\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:17.462413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:17.462407    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:18.167437 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:18.167407    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"
Oct 30 01:52:18.167728 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:18.167485    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5" event={"ID":"eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba","Type":"ContainerDied","Data":"31d99c8cffcd4a7641fadd0bddb7664a3dc31ac7734fb7df5534d2b658ceba3a"}
Oct 30 01:52:18.168025 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:18.167999    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31d99c8cffcd4a7641fadd0bddb7664a3dc31ac7734fb7df5534d2b658ceba3a"
Oct 30 01:52:20.176293 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.176264    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"
Oct 30 01:52:20.186314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.186287    4425 generic.go:356] "Generic (PLEG): container finished" podID="9cab6c8f-afe1-4908-8aa8-e533837c609a" containerID="af25bf99e691dc67db0ddaab693b76cc2e98f3ebe0c9e2566dbbce9f600a976c" exitCode=0
Oct 30 01:52:20.186457 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.186337    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"
Oct 30 01:52:20.186457 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.186379    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb" event={"ID":"9cab6c8f-afe1-4908-8aa8-e533837c609a","Type":"ContainerDied","Data":"af25bf99e691dc67db0ddaab693b76cc2e98f3ebe0c9e2566dbbce9f600a976c"}
Oct 30 01:52:20.186457 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.186448    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb" event={"ID":"9cab6c8f-afe1-4908-8aa8-e533837c609a","Type":"ContainerDied","Data":"b8596a64a89f60d6dfaad46e5623410a2d8a92b855d97548daddb71c2622081e"}
Oct 30 01:52:20.186600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.186478    4425 scope.go:122] "RemoveContainer" containerID="af25bf99e691dc67db0ddaab693b76cc2e98f3ebe0c9e2566dbbce9f600a976c"
Oct 30 01:52:20.188413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.188389    4425 generic.go:356] "Generic (PLEG): container finished" podID="817e5cef-adec-43f8-8d49-439e1379d1be" containerID="3a9db16678e97e6a6ea17db2d4149cdb59021401bd2c465958c71cad8e956e36" exitCode=0
Oct 30 01:52:20.188504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.188472    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r" event={"ID":"817e5cef-adec-43f8-8d49-439e1379d1be","Type":"ContainerDied","Data":"3a9db16678e97e6a6ea17db2d4149cdb59021401bd2c465958c71cad8e956e36"}
Oct 30 01:52:20.197866 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.197840    4425 scope.go:122] "RemoveContainer" containerID="af25bf99e691dc67db0ddaab693b76cc2e98f3ebe0c9e2566dbbce9f600a976c"
Oct 30 01:52:20.198127 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:52:20.198100    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af25bf99e691dc67db0ddaab693b76cc2e98f3ebe0c9e2566dbbce9f600a976c\": container with ID starting with af25bf99e691dc67db0ddaab693b76cc2e98f3ebe0c9e2566dbbce9f600a976c not found: ID does not exist" containerID="af25bf99e691dc67db0ddaab693b76cc2e98f3ebe0c9e2566dbbce9f600a976c"
Oct 30 01:52:20.198256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.198123    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af25bf99e691dc67db0ddaab693b76cc2e98f3ebe0c9e2566dbbce9f600a976c"} err="failed to get container status \"af25bf99e691dc67db0ddaab693b76cc2e98f3ebe0c9e2566dbbce9f600a976c\": rpc error: code = NotFound desc = could not find container \"af25bf99e691dc67db0ddaab693b76cc2e98f3ebe0c9e2566dbbce9f600a976c\": container with ID starting with af25bf99e691dc67db0ddaab693b76cc2e98f3ebe0c9e2566dbbce9f600a976c not found: ID does not exist"
Oct 30 01:52:20.221413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.221390    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"
Oct 30 01:52:20.295226 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.295195    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cab6c8f-afe1-4908-8aa8-e533837c609a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cab6c8f-afe1-4908-8aa8-e533837c609a-plugins-config-volume\") pod \"9cab6c8f-afe1-4908-8aa8-e533837c609a\" (UID: \"9cab6c8f-afe1-4908-8aa8-e533837c609a\") "
Oct 30 01:52:20.295424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.295339    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/817e5cef-adec-43f8-8d49-439e1379d1be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/817e5cef-adec-43f8-8d49-439e1379d1be-plugins-config-volume\") pod \"817e5cef-adec-43f8-8d49-439e1379d1be\" (UID: \"817e5cef-adec-43f8-8d49-439e1379d1be\") "
Oct 30 01:52:20.295424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.295371    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cab6c8f-afe1-4908-8aa8-e533837c609a-kube-api-access-f488s\" (UniqueName: \"kubernetes.io/projected/9cab6c8f-afe1-4908-8aa8-e533837c609a-kube-api-access-f488s\") pod \"9cab6c8f-afe1-4908-8aa8-e533837c609a\" (UID: \"9cab6c8f-afe1-4908-8aa8-e533837c609a\") "
Oct 30 01:52:20.295424 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.295417    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/817e5cef-adec-43f8-8d49-439e1379d1be-kube-api-access-zp5v6\" (UniqueName: \"kubernetes.io/projected/817e5cef-adec-43f8-8d49-439e1379d1be-kube-api-access-zp5v6\") pod \"817e5cef-adec-43f8-8d49-439e1379d1be\" (UID: \"817e5cef-adec-43f8-8d49-439e1379d1be\") "
Oct 30 01:52:20.302180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.295630    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cab6c8f-afe1-4908-8aa8-e533837c609a-plugins-config-volume" pod "9cab6c8f-afe1-4908-8aa8-e533837c609a" (UID: "9cab6c8f-afe1-4908-8aa8-e533837c609a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:20.302180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.296515    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817e5cef-adec-43f8-8d49-439e1379d1be-plugins-config-volume" pod "817e5cef-adec-43f8-8d49-439e1379d1be" (UID: "817e5cef-adec-43f8-8d49-439e1379d1be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:20.302180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.296761    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cab6c8f-afe1-4908-8aa8-e533837c609a-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:20.302180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.296784    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/817e5cef-adec-43f8-8d49-439e1379d1be-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:20.302180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.298574    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cab6c8f-afe1-4908-8aa8-e533837c609a-kube-api-access-f488s" pod "9cab6c8f-afe1-4908-8aa8-e533837c609a" (UID: "9cab6c8f-afe1-4908-8aa8-e533837c609a"). InnerVolumeSpecName "kube-api-access-f488s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:20.302180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.300081    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/817e5cef-adec-43f8-8d49-439e1379d1be-kube-api-access-zp5v6" pod "817e5cef-adec-43f8-8d49-439e1379d1be" (UID: "817e5cef-adec-43f8-8d49-439e1379d1be"). InnerVolumeSpecName "kube-api-access-zp5v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:20.398441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.398349    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f488s\" (UniqueName: \"kubernetes.io/projected/9cab6c8f-afe1-4908-8aa8-e533837c609a-kube-api-access-f488s\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:20.398441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.398387    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp5v6\" (UniqueName: \"kubernetes.io/projected/817e5cef-adec-43f8-8d49-439e1379d1be-kube-api-access-zp5v6\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:20.528932 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.528899    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"]
Oct 30 01:52:20.538055 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.538031    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-94ea1e-7dfd8f5888-fmzbb"]
Oct 30 01:52:20.667495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.667417    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6" podUID="723cbbff-c865-4304-9851-d1968c50cd8a" containerName="main" containerID="cri-o://16ec2c0dd9c390b292f642ea37994c89efed4d2b7466a6a97278d4ef4fde79dc" gracePeriod=45
Oct 30 01:52:20.946933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:20.946906    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:52:21.004783 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.004754    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723cbbff-c865-4304-9851-d1968c50cd8a-kube-api-access-t5qln\" (UniqueName: \"kubernetes.io/projected/723cbbff-c865-4304-9851-d1968c50cd8a-kube-api-access-t5qln\") pod \"723cbbff-c865-4304-9851-d1968c50cd8a\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") "
Oct 30 01:52:21.004950 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.004836    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/723cbbff-c865-4304-9851-d1968c50cd8a-tls-certs\" (UniqueName: \"kubernetes.io/secret/723cbbff-c865-4304-9851-d1968c50cd8a-tls-certs\") pod \"723cbbff-c865-4304-9851-d1968c50cd8a\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") "
Oct 30 01:52:21.004950 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.004897    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-home\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-home\") pod \"723cbbff-c865-4304-9851-d1968c50cd8a\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") "
Oct 30 01:52:21.004950 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.004922    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-model-cache\") pod \"723cbbff-c865-4304-9851-d1968c50cd8a\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") "
Oct 30 01:52:21.005128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.004961    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-tmp-dir\") pod \"723cbbff-c865-4304-9851-d1968c50cd8a\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") "
Oct 30 01:52:21.005128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.005008    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-dshm\") pod \"723cbbff-c865-4304-9851-d1968c50cd8a\" (UID: \"723cbbff-c865-4304-9851-d1968c50cd8a\") "
Oct 30 01:52:21.005626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.005470    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-model-cache" pod "723cbbff-c865-4304-9851-d1968c50cd8a" (UID: "723cbbff-c865-4304-9851-d1968c50cd8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:52:21.005853 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.005831    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:21.005944 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.005908    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-home" pod "723cbbff-c865-4304-9851-d1968c50cd8a" (UID: "723cbbff-c865-4304-9851-d1968c50cd8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:52:21.006008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.005962    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-tmp-dir" pod "723cbbff-c865-4304-9851-d1968c50cd8a" (UID: "723cbbff-c865-4304-9851-d1968c50cd8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:52:21.007195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.007170    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723cbbff-c865-4304-9851-d1968c50cd8a-kube-api-access-t5qln" pod "723cbbff-c865-4304-9851-d1968c50cd8a" (UID: "723cbbff-c865-4304-9851-d1968c50cd8a"). InnerVolumeSpecName "kube-api-access-t5qln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:21.007365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.007339    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-dshm" pod "723cbbff-c865-4304-9851-d1968c50cd8a" (UID: "723cbbff-c865-4304-9851-d1968c50cd8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:52:21.007434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.007396    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/723cbbff-c865-4304-9851-d1968c50cd8a-tls-certs" pod "723cbbff-c865-4304-9851-d1968c50cd8a" (UID: "723cbbff-c865-4304-9851-d1968c50cd8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:52:21.106753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.106721    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:21.106753 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.106748    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:21.106910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.106762    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5qln\" (UniqueName: \"kubernetes.io/projected/723cbbff-c865-4304-9851-d1968c50cd8a-kube-api-access-t5qln\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:21.106910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.106777    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/723cbbff-c865-4304-9851-d1968c50cd8a-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:21.106910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.106788    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/723cbbff-c865-4304-9851-d1968c50cd8a-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:21.197237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.197169    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"
Oct 30 01:52:21.197237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.197202    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r" event={"ID":"817e5cef-adec-43f8-8d49-439e1379d1be","Type":"ContainerDied","Data":"6f754b17d9bfafba4c72416a0487de1d979fbe9d01c8c7e5b82831f71aa44948"}
Oct 30 01:52:21.197651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.197267    4425 scope.go:122] "RemoveContainer" containerID="3a9db16678e97e6a6ea17db2d4149cdb59021401bd2c465958c71cad8e956e36"
Oct 30 01:52:21.198959 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.198885    4425 generic.go:356] "Generic (PLEG): container finished" podID="723cbbff-c865-4304-9851-d1968c50cd8a" containerID="16ec2c0dd9c390b292f642ea37994c89efed4d2b7466a6a97278d4ef4fde79dc" exitCode=0
Oct 30 01:52:21.199085 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.199068    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"
Oct 30 01:52:21.199149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.199103    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6" event={"ID":"723cbbff-c865-4304-9851-d1968c50cd8a","Type":"ContainerDied","Data":"16ec2c0dd9c390b292f642ea37994c89efed4d2b7466a6a97278d4ef4fde79dc"}
Oct 30 01:52:21.199149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.199135    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6" event={"ID":"723cbbff-c865-4304-9851-d1968c50cd8a","Type":"ContainerDied","Data":"a18b87385ae354b2936c2b8b3ceb3c60cfa72b5b3cac0d8b36e3eb8765a79168"}
Oct 30 01:52:21.210855 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.210837    4425 scope.go:122] "RemoveContainer" containerID="16ec2c0dd9c390b292f642ea37994c89efed4d2b7466a6a97278d4ef4fde79dc"
Oct 30 01:52:21.220399 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.220379    4425 scope.go:122] "RemoveContainer" containerID="16ec2c0dd9c390b292f642ea37994c89efed4d2b7466a6a97278d4ef4fde79dc"
Oct 30 01:52:21.220629 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:52:21.220611    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ec2c0dd9c390b292f642ea37994c89efed4d2b7466a6a97278d4ef4fde79dc\": container with ID starting with 16ec2c0dd9c390b292f642ea37994c89efed4d2b7466a6a97278d4ef4fde79dc not found: ID does not exist" containerID="16ec2c0dd9c390b292f642ea37994c89efed4d2b7466a6a97278d4ef4fde79dc"
Oct 30 01:52:21.220693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.220635    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ec2c0dd9c390b292f642ea37994c89efed4d2b7466a6a97278d4ef4fde79dc"} err="failed to get container status \"16ec2c0dd9c390b292f642ea37994c89efed4d2b7466a6a97278d4ef4fde79dc\": rpc error: code = NotFound desc = could not find container \"16ec2c0dd9c390b292f642ea37994c89efed4d2b7466a6a97278d4ef4fde79dc\": container with ID starting with 16ec2c0dd9c390b292f642ea37994c89efed4d2b7466a6a97278d4ef4fde79dc not found: ID does not exist"
Oct 30 01:52:21.244620 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.244596    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"]
Oct 30 01:52:21.248324 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.248303    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1854c4/sub-test-model-1854c4-kserve-6db98f484c-kp4k6"]
Oct 30 01:52:21.276448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.276427    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"]
Oct 30 01:52:21.281532 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.281508    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-94ea1e-76964cb787-8rs5r"]
Oct 30 01:52:21.332915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.332875    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm" podUID="d2fecb6d-0b3b-4b1c-9806-74e68559265a" containerName="payload-processing" containerID="cri-o://0ad555280f5a67d164712906536283da051527390cd413001c6b33790be0ff44" gracePeriod=25
Oct 30 01:52:21.352286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.352252    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t" podUID="8b86ac41-fb47-4c59-8723-9223d01f49f3" containerName="payload-pre-processing" containerID="cri-o://c58c54a09c3ae43aa427c5d4177dacfd9fa0fcc23903af8ed1e20a9cecc66f41" gracePeriod=25
Oct 30 01:52:21.423054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.423027    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="723cbbff-c865-4304-9851-d1968c50cd8a" path="/var/lib/kubelet/pods/723cbbff-c865-4304-9851-d1968c50cd8a/volumes"
Oct 30 01:52:21.423500 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.423481    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="817e5cef-adec-43f8-8d49-439e1379d1be" path="/var/lib/kubelet/pods/817e5cef-adec-43f8-8d49-439e1379d1be/volumes"
Oct 30 01:52:21.423912 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.423894    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cab6c8f-afe1-4908-8aa8-e533837c609a" path="/var/lib/kubelet/pods/9cab6c8f-afe1-4908-8aa8-e533837c609a/volumes"
Oct 30 01:52:21.696873 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.696839    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"]
Oct 30 01:52:21.697577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.697560    4425 cpu_manager.go:414] "containerMap: removing container" podUID="817e5cef-adec-43f8-8d49-439e1379d1be" containerName="payload-pre-processing"
Oct 30 01:52:21.697652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.697577    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="817e5cef-adec-43f8-8d49-439e1379d1be" containerName="payload-pre-processing"
Oct 30 01:52:21.697652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.697587    4425 cpu_manager.go:414] "containerMap: removing container" podUID="723cbbff-c865-4304-9851-d1968c50cd8a" containerName="main"
Oct 30 01:52:21.697652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.697593    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="723cbbff-c865-4304-9851-d1968c50cd8a" containerName="main"
Oct 30 01:52:21.697652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.697610    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d6019518-5851-47d1-8c48-c4c697264d32" containerName="revoke-keys"
Oct 30 01:52:21.697652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.697615    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6019518-5851-47d1-8c48-c4c697264d32" containerName="revoke-keys"
Oct 30 01:52:21.697652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.697625    4425 cpu_manager.go:414] "containerMap: removing container" podUID="eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba" containerName="revoke-keys"
Oct 30 01:52:21.697652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.697630    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba" containerName="revoke-keys"
Oct 30 01:52:21.697652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.697644    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9cab6c8f-afe1-4908-8aa8-e533837c609a" containerName="payload-processing"
Oct 30 01:52:21.697652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.697649    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cab6c8f-afe1-4908-8aa8-e533837c609a" containerName="payload-processing"
Oct 30 01:52:21.697972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.697739    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="817e5cef-adec-43f8-8d49-439e1379d1be" containerName="payload-pre-processing"
Oct 30 01:52:21.697972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.697747    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="723cbbff-c865-4304-9851-d1968c50cd8a" containerName="main"
Oct 30 01:52:21.697972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.697756    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6019518-5851-47d1-8c48-c4c697264d32" containerName="revoke-keys"
Oct 30 01:52:21.697972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.697764    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba" containerName="revoke-keys"
Oct 30 01:52:21.697972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.697771    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cab6c8f-afe1-4908-8aa8-e533837c609a" containerName="payload-processing"
Oct 30 01:52:21.702730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.702710    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"
Oct 30 01:52:21.717286 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.717264    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:52:21.717389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.717352    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:52:21.721876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.721856    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"]
Oct 30 01:52:21.815430 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.815390    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d693b69-f319-4f95-9844-1ce2e5df425d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a6d91ec-kmfrp\" (UID: \"5d693b69-f319-4f95-9844-1ce2e5df425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"
Oct 30 01:52:21.815570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.815461    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d693b69-f319-4f95-9844-1ce2e5df425d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a6d91ec-kmfrp\" (UID: \"5d693b69-f319-4f95-9844-1ce2e5df425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"
Oct 30 01:52:21.815625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.815576    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh2c8\" (UniqueName: \"kubernetes.io/projected/5d693b69-f319-4f95-9844-1ce2e5df425d-kube-api-access-nh2c8\") pod \"maas-api-revoke-sub-3a6d91ec-kmfrp\" (UID: \"5d693b69-f319-4f95-9844-1ce2e5df425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"
Oct 30 01:52:21.916582 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.916554    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh2c8\" (UniqueName: \"kubernetes.io/projected/5d693b69-f319-4f95-9844-1ce2e5df425d-kube-api-access-nh2c8\") pod \"maas-api-revoke-sub-3a6d91ec-kmfrp\" (UID: \"5d693b69-f319-4f95-9844-1ce2e5df425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"
Oct 30 01:52:21.916778 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.916726    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d693b69-f319-4f95-9844-1ce2e5df425d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a6d91ec-kmfrp\" (UID: \"5d693b69-f319-4f95-9844-1ce2e5df425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"
Oct 30 01:52:21.916853 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.916800    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d693b69-f319-4f95-9844-1ce2e5df425d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a6d91ec-kmfrp\" (UID: \"5d693b69-f319-4f95-9844-1ce2e5df425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"
Oct 30 01:52:21.917519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.917495    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d693b69-f319-4f95-9844-1ce2e5df425d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a6d91ec-kmfrp\" (UID: \"5d693b69-f319-4f95-9844-1ce2e5df425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"
Oct 30 01:52:21.918956 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.918935    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d693b69-f319-4f95-9844-1ce2e5df425d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a6d91ec-kmfrp\" (UID: \"5d693b69-f319-4f95-9844-1ce2e5df425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"
Oct 30 01:52:21.937427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:21.937405    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh2c8\" (UniqueName: \"kubernetes.io/projected/5d693b69-f319-4f95-9844-1ce2e5df425d-kube-api-access-nh2c8\") pod \"maas-api-revoke-sub-3a6d91ec-kmfrp\" (UID: \"5d693b69-f319-4f95-9844-1ce2e5df425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"
Oct 30 01:52:22.017550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:22.017513    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"
Oct 30 01:52:22.322765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:22.322733    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"]
Oct 30 01:52:22.324101 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:52:22.324071    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d693b69_f319_4f95_9844_1ce2e5df425d.slice/crio-aec725cb99abad49c2427ac7c0799a301dd61aef86af038492477bea64c23952 WatchSource:0}: Error finding container aec725cb99abad49c2427ac7c0799a301dd61aef86af038492477bea64c23952: Status 404 returned error can't find the container with id aec725cb99abad49c2427ac7c0799a301dd61aef86af038492477bea64c23952
Oct 30 01:52:23.219992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:23.219958    4425 generic.go:356] "Generic (PLEG): container finished" podID="5d693b69-f319-4f95-9844-1ce2e5df425d" containerID="f982d0cda0ad4faf112b1441ba0401b679cab86fe9eb515c1c4ad9e86a33a214" exitCode=0
Oct 30 01:52:23.220123 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:23.220000    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp" event={"ID":"5d693b69-f319-4f95-9844-1ce2e5df425d","Type":"ContainerDied","Data":"f982d0cda0ad4faf112b1441ba0401b679cab86fe9eb515c1c4ad9e86a33a214"}
Oct 30 01:52:23.220123 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:23.220056    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp" event={"ID":"5d693b69-f319-4f95-9844-1ce2e5df425d","Type":"ContainerStarted","Data":"aec725cb99abad49c2427ac7c0799a301dd61aef86af038492477bea64c23952"}
Oct 30 01:52:23.392092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:23.392054    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t" podUID="8b86ac41-fb47-4c59-8723-9223d01f49f3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:52:23.392595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:23.392136    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm" podUID="d2fecb6d-0b3b-4b1c-9806-74e68559265a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:52:24.127838 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.127801    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"]
Oct 30 01:52:24.132634 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.132613    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"
Oct 30 01:52:24.151970 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.151939    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"]
Oct 30 01:52:24.244948 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.244917    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb8a211-m57ld\" (UID: \"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"
Oct 30 01:52:24.245113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.244978    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qllvv\" (UniqueName: \"kubernetes.io/projected/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-kube-api-access-qllvv\") pod \"maas-api-revoke-sub-cdb8a211-m57ld\" (UID: \"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"
Oct 30 01:52:24.245202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.245179    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb8a211-m57ld\" (UID: \"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"
Oct 30 01:52:24.346903 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.346862    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb8a211-m57ld\" (UID: \"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"
Oct 30 01:52:24.347776 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.347658    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb8a211-m57ld\" (UID: \"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"
Oct 30 01:52:24.347902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.347881    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb8a211-m57ld\" (UID: \"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"
Oct 30 01:52:24.347964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.347938    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qllvv\" (UniqueName: \"kubernetes.io/projected/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-kube-api-access-qllvv\") pod \"maas-api-revoke-sub-cdb8a211-m57ld\" (UID: \"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"
Oct 30 01:52:24.350406 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.350380    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb8a211-m57ld\" (UID: \"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"
Oct 30 01:52:24.369300 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.369269    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qllvv\" (UniqueName: \"kubernetes.io/projected/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-kube-api-access-qllvv\") pod \"maas-api-revoke-sub-cdb8a211-m57ld\" (UID: \"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"
Oct 30 01:52:24.398457 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.398404    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"
Oct 30 01:52:24.447408 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.447385    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"
Oct 30 01:52:24.549720 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.549681    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d693b69-f319-4f95-9844-1ce2e5df425d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d693b69-f319-4f95-9844-1ce2e5df425d-maas-api-xks-ca\") pod \"5d693b69-f319-4f95-9844-1ce2e5df425d\" (UID: \"5d693b69-f319-4f95-9844-1ce2e5df425d\") "
Oct 30 01:52:24.549885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.549791    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d693b69-f319-4f95-9844-1ce2e5df425d-kube-api-access-nh2c8\" (UniqueName: \"kubernetes.io/projected/5d693b69-f319-4f95-9844-1ce2e5df425d-kube-api-access-nh2c8\") pod \"5d693b69-f319-4f95-9844-1ce2e5df425d\" (UID: \"5d693b69-f319-4f95-9844-1ce2e5df425d\") "
Oct 30 01:52:24.549885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.549867    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d693b69-f319-4f95-9844-1ce2e5df425d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d693b69-f319-4f95-9844-1ce2e5df425d-maas-api-openshift-ca\") pod \"5d693b69-f319-4f95-9844-1ce2e5df425d\" (UID: \"5d693b69-f319-4f95-9844-1ce2e5df425d\") "
Oct 30 01:52:24.550296 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.550268    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d693b69-f319-4f95-9844-1ce2e5df425d-maas-api-openshift-ca" pod "5d693b69-f319-4f95-9844-1ce2e5df425d" (UID: "5d693b69-f319-4f95-9844-1ce2e5df425d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:24.550926 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.550620    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d693b69-f319-4f95-9844-1ce2e5df425d-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:24.551992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.551966    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d693b69-f319-4f95-9844-1ce2e5df425d-kube-api-access-nh2c8" pod "5d693b69-f319-4f95-9844-1ce2e5df425d" (UID: "5d693b69-f319-4f95-9844-1ce2e5df425d"). InnerVolumeSpecName "kube-api-access-nh2c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:24.552372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.552348    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d693b69-f319-4f95-9844-1ce2e5df425d-maas-api-xks-ca" pod "5d693b69-f319-4f95-9844-1ce2e5df425d" (UID: "5d693b69-f319-4f95-9844-1ce2e5df425d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:52:24.651918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.651855    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d693b69-f319-4f95-9844-1ce2e5df425d-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:24.651918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.651882    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh2c8\" (UniqueName: \"kubernetes.io/projected/5d693b69-f319-4f95-9844-1ce2e5df425d-kube-api-access-nh2c8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:24.756670 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:24.756638    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"]
Oct 30 01:52:24.757063 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:52:24.757028    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2da52c_5c3c_40c0_a3a1_b2a68c1501fa.slice/crio-6630620a81b3acb2bc95f011786e0cf0b319dece5827a13720c4bd09a733206c WatchSource:0}: Error finding container 6630620a81b3acb2bc95f011786e0cf0b319dece5827a13720c4bd09a733206c: Status 404 returned error can't find the container with id 6630620a81b3acb2bc95f011786e0cf0b319dece5827a13720c4bd09a733206c
Oct 30 01:52:25.236824 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:25.236742    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"
Oct 30 01:52:25.236824 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:25.236768    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp" event={"ID":"5d693b69-f319-4f95-9844-1ce2e5df425d","Type":"ContainerDied","Data":"aec725cb99abad49c2427ac7c0799a301dd61aef86af038492477bea64c23952"}
Oct 30 01:52:25.236824 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:25.236814    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aec725cb99abad49c2427ac7c0799a301dd61aef86af038492477bea64c23952"
Oct 30 01:52:25.238622 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:25.238595    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld" event={"ID":"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa","Type":"ContainerStarted","Data":"d9270c4fb38fc8d9f0fe3397707dbc7664578c6ef3db6585d08cf26406e56f04"}
Oct 30 01:52:25.238770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:25.238637    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld" event={"ID":"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa","Type":"ContainerStarted","Data":"6630620a81b3acb2bc95f011786e0cf0b319dece5827a13720c4bd09a733206c"}
Oct 30 01:52:25.275310 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:25.275267    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld" podStartSLOduration=1.275256306 podStartE2EDuration="1.275256306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:25.274315708 +0000 UTC m=+4254.023971015" watchObservedRunningTime="2026-10-30 01:52:25.275256306 +0000 UTC m=+4254.024911643"
Oct 30 01:52:26.247452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:26.247415    4425 generic.go:356] "Generic (PLEG): container finished" podID="6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa" containerID="d9270c4fb38fc8d9f0fe3397707dbc7664578c6ef3db6585d08cf26406e56f04" exitCode=0
Oct 30 01:52:26.247857 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:26.247470    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld" event={"ID":"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa","Type":"ContainerDied","Data":"d9270c4fb38fc8d9f0fe3397707dbc7664578c6ef3db6585d08cf26406e56f04"}
Oct 30 01:52:26.508884 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:26.508797    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"]
Oct 30 01:52:26.509060 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:26.509025    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b" podUID="68f8c428-fa70-411a-bd16-2861555890bf" containerName="maas-api" containerID="cri-o://2bf5c4af154cfabaf72b0c2251e266888655b006fbf876e31a94d15cd114734a" gracePeriod=30
Oct 30 01:52:26.803013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:26.802981    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:52:26.875774 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:26.875737    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f8c428-fa70-411a-bd16-2861555890bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68f8c428-fa70-411a-bd16-2861555890bf-openshift-service-ca\") pod \"68f8c428-fa70-411a-bd16-2861555890bf\" (UID: \"68f8c428-fa70-411a-bd16-2861555890bf\") "
Oct 30 01:52:26.875958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:26.875816    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f8c428-fa70-411a-bd16-2861555890bf-kube-api-access-8xxgf\" (UniqueName: \"kubernetes.io/projected/68f8c428-fa70-411a-bd16-2861555890bf-kube-api-access-8xxgf\") pod \"68f8c428-fa70-411a-bd16-2861555890bf\" (UID: \"68f8c428-fa70-411a-bd16-2861555890bf\") "
Oct 30 01:52:26.876041 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:26.876002    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68f8c428-fa70-411a-bd16-2861555890bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f8c428-fa70-411a-bd16-2861555890bf-metrics-certs\") pod \"68f8c428-fa70-411a-bd16-2861555890bf\" (UID: \"68f8c428-fa70-411a-bd16-2861555890bf\") "
Oct 30 01:52:26.876041 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:26.876032    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68f8c428-fa70-411a-bd16-2861555890bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68f8c428-fa70-411a-bd16-2861555890bf-maas-api-tls\") pod \"68f8c428-fa70-411a-bd16-2861555890bf\" (UID: \"68f8c428-fa70-411a-bd16-2861555890bf\") "
Oct 30 01:52:26.876425 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:26.876390    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f8c428-fa70-411a-bd16-2861555890bf-openshift-service-ca" pod "68f8c428-fa70-411a-bd16-2861555890bf" (UID: "68f8c428-fa70-411a-bd16-2861555890bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:26.876674 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:26.876650    4425 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68f8c428-fa70-411a-bd16-2861555890bf-openshift-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:26.878747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:26.878716    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f8c428-fa70-411a-bd16-2861555890bf-kube-api-access-8xxgf" pod "68f8c428-fa70-411a-bd16-2861555890bf" (UID: "68f8c428-fa70-411a-bd16-2861555890bf"). InnerVolumeSpecName "kube-api-access-8xxgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:26.878864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:26.878844    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f8c428-fa70-411a-bd16-2861555890bf-maas-api-tls" pod "68f8c428-fa70-411a-bd16-2861555890bf" (UID: "68f8c428-fa70-411a-bd16-2861555890bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:52:26.879104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:26.879082    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f8c428-fa70-411a-bd16-2861555890bf-metrics-certs" pod "68f8c428-fa70-411a-bd16-2861555890bf" (UID: "68f8c428-fa70-411a-bd16-2861555890bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:52:26.978182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:26.978113    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xxgf\" (UniqueName: \"kubernetes.io/projected/68f8c428-fa70-411a-bd16-2861555890bf-kube-api-access-8xxgf\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:26.978182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:26.978147    4425 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f8c428-fa70-411a-bd16-2861555890bf-metrics-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:26.978182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:26.978177    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68f8c428-fa70-411a-bd16-2861555890bf-maas-api-tls\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:27.210405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.210316    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"]
Oct 30 01:52:27.215872 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.215839    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-94ea1e-b28wx"]
Oct 30 01:52:27.261448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.261406    4425 generic.go:356] "Generic (PLEG): container finished" podID="68f8c428-fa70-411a-bd16-2861555890bf" containerID="2bf5c4af154cfabaf72b0c2251e266888655b006fbf876e31a94d15cd114734a" exitCode=0
Oct 30 01:52:27.261863 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.261450    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"
Oct 30 01:52:27.261863 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.261481    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b" event={"ID":"68f8c428-fa70-411a-bd16-2861555890bf","Type":"ContainerDied","Data":"2bf5c4af154cfabaf72b0c2251e266888655b006fbf876e31a94d15cd114734a"}
Oct 30 01:52:27.261863 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.261556    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b" event={"ID":"68f8c428-fa70-411a-bd16-2861555890bf","Type":"ContainerDied","Data":"b652ae6ed1be311ee7bfffbe7b9b1d5f4f4ef63a1685226eb70d91ddc265152e"}
Oct 30 01:52:27.261863 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.261582    4425 scope.go:122] "RemoveContainer" containerID="2bf5c4af154cfabaf72b0c2251e266888655b006fbf876e31a94d15cd114734a"
Oct 30 01:52:27.274052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.274032    4425 scope.go:122] "RemoveContainer" containerID="2bf5c4af154cfabaf72b0c2251e266888655b006fbf876e31a94d15cd114734a"
Oct 30 01:52:27.274798 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:52:27.274776    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf5c4af154cfabaf72b0c2251e266888655b006fbf876e31a94d15cd114734a\": container with ID starting with 2bf5c4af154cfabaf72b0c2251e266888655b006fbf876e31a94d15cd114734a not found: ID does not exist" containerID="2bf5c4af154cfabaf72b0c2251e266888655b006fbf876e31a94d15cd114734a"
Oct 30 01:52:27.274874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.274809    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf5c4af154cfabaf72b0c2251e266888655b006fbf876e31a94d15cd114734a"} err="failed to get container status \"2bf5c4af154cfabaf72b0c2251e266888655b006fbf876e31a94d15cd114734a\": rpc error: code = NotFound desc = could not find container \"2bf5c4af154cfabaf72b0c2251e266888655b006fbf876e31a94d15cd114734a\": container with ID starting with 2bf5c4af154cfabaf72b0c2251e266888655b006fbf876e31a94d15cd114734a not found: ID does not exist"
Oct 30 01:52:27.312642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.312613    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"]
Oct 30 01:52:27.316374 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.316350    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-94ea1e-5ccb778f58-p2d7b"]
Oct 30 01:52:27.400428 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.400401    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"
Oct 30 01:52:27.423882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.423852    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ad21dae-c26b-4f28-a7db-3cde934cbeed" path="/var/lib/kubelet/pods/3ad21dae-c26b-4f28-a7db-3cde934cbeed/volumes"
Oct 30 01:52:27.424473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.424448    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f8c428-fa70-411a-bd16-2861555890bf" path="/var/lib/kubelet/pods/68f8c428-fa70-411a-bd16-2861555890bf/volumes"
Oct 30 01:52:27.485588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.485505    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-kube-api-access-qllvv\" (UniqueName: \"kubernetes.io/projected/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-kube-api-access-qllvv\") pod \"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa\" (UID: \"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa\") "
Oct 30 01:52:27.485588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.485585    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-maas-api-xks-ca\") pod \"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa\" (UID: \"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa\") "
Oct 30 01:52:27.485829 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.485678    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-maas-api-openshift-ca\") pod \"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa\" (UID: \"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa\") "
Oct 30 01:52:27.486054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.486025    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-maas-api-openshift-ca" pod "6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa" (UID: "6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:27.486274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.486255    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:27.487624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.487602    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-kube-api-access-qllvv" pod "6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa" (UID: "6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa"). InnerVolumeSpecName "kube-api-access-qllvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:27.487701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.487615    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-maas-api-xks-ca" pod "6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa" (UID: "6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:52:27.587650 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.587620    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qllvv\" (UniqueName: \"kubernetes.io/projected/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-kube-api-access-qllvv\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:27.587650 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:27.587649    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:28.272125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:28.272093    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"
Oct 30 01:52:28.272125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:28.272122    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld" event={"ID":"6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa","Type":"ContainerDied","Data":"6630620a81b3acb2bc95f011786e0cf0b319dece5827a13720c4bd09a733206c"}
Oct 30 01:52:28.272545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:28.272202    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6630620a81b3acb2bc95f011786e0cf0b319dece5827a13720c4bd09a733206c"
Oct 30 01:52:33.392189 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:33.392135    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm" podUID="d2fecb6d-0b3b-4b1c-9806-74e68559265a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:52:33.392557 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:33.392264    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t" podUID="8b86ac41-fb47-4c59-8723-9223d01f49f3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:52:34.977431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:34.977395    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"]
Oct 30 01:52:34.978268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:34.978244    4425 cpu_manager.go:414] "containerMap: removing container" podUID="6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa" containerName="revoke-keys"
Oct 30 01:52:34.978268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:34.978268    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa" containerName="revoke-keys"
Oct 30 01:52:34.978474 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:34.978309    4425 cpu_manager.go:414] "containerMap: removing container" podUID="5d693b69-f319-4f95-9844-1ce2e5df425d" containerName="revoke-keys"
Oct 30 01:52:34.978474 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:34.978318    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d693b69-f319-4f95-9844-1ce2e5df425d" containerName="revoke-keys"
Oct 30 01:52:34.978474 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:34.978372    4425 cpu_manager.go:414] "containerMap: removing container" podUID="68f8c428-fa70-411a-bd16-2861555890bf" containerName="maas-api"
Oct 30 01:52:34.978474 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:34.978379    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f8c428-fa70-411a-bd16-2861555890bf" containerName="maas-api"
Oct 30 01:52:34.978679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:34.978559    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f8c428-fa70-411a-bd16-2861555890bf" containerName="maas-api"
Oct 30 01:52:34.978679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:34.978577    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa" containerName="revoke-keys"
Oct 30 01:52:34.978679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:34.978594    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d693b69-f319-4f95-9844-1ce2e5df425d" containerName="revoke-keys"
Oct 30 01:52:34.983790 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:34.983769    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"
Oct 30 01:52:34.997405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:34.997387    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:52:35.007343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:35.007321    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:52:35.012218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:35.012192    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"]
Oct 30 01:52:35.078336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:35.078299    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kknjx\" (UniqueName: \"kubernetes.io/projected/e622d47c-b9a0-4307-863f-84241f4bc9c8-kube-api-access-kknjx\") pod \"maas-api-revoke-sub-9db93ee8-88lmk\" (UID: \"e622d47c-b9a0-4307-863f-84241f4bc9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"
Oct 30 01:52:35.078578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:35.078557    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e622d47c-b9a0-4307-863f-84241f4bc9c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9db93ee8-88lmk\" (UID: \"e622d47c-b9a0-4307-863f-84241f4bc9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"
Oct 30 01:52:35.078649 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:35.078605    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e622d47c-b9a0-4307-863f-84241f4bc9c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9db93ee8-88lmk\" (UID: \"e622d47c-b9a0-4307-863f-84241f4bc9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"
Oct 30 01:52:35.180415 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:35.180376    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e622d47c-b9a0-4307-863f-84241f4bc9c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9db93ee8-88lmk\" (UID: \"e622d47c-b9a0-4307-863f-84241f4bc9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"
Oct 30 01:52:35.180609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:35.180440    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e622d47c-b9a0-4307-863f-84241f4bc9c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9db93ee8-88lmk\" (UID: \"e622d47c-b9a0-4307-863f-84241f4bc9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"
Oct 30 01:52:35.180609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:35.180489    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kknjx\" (UniqueName: \"kubernetes.io/projected/e622d47c-b9a0-4307-863f-84241f4bc9c8-kube-api-access-kknjx\") pod \"maas-api-revoke-sub-9db93ee8-88lmk\" (UID: \"e622d47c-b9a0-4307-863f-84241f4bc9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"
Oct 30 01:52:35.181229 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:35.181204    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e622d47c-b9a0-4307-863f-84241f4bc9c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9db93ee8-88lmk\" (UID: \"e622d47c-b9a0-4307-863f-84241f4bc9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"
Oct 30 01:52:35.182864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:35.182839    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e622d47c-b9a0-4307-863f-84241f4bc9c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9db93ee8-88lmk\" (UID: \"e622d47c-b9a0-4307-863f-84241f4bc9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"
Oct 30 01:52:35.203800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:35.203775    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kknjx\" (UniqueName: \"kubernetes.io/projected/e622d47c-b9a0-4307-863f-84241f4bc9c8-kube-api-access-kknjx\") pod \"maas-api-revoke-sub-9db93ee8-88lmk\" (UID: \"e622d47c-b9a0-4307-863f-84241f4bc9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"
Oct 30 01:52:35.297598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:35.297570    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"
Oct 30 01:52:35.606632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:35.606573    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"]
Oct 30 01:52:35.607330 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:52:35.607296    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode622d47c_b9a0_4307_863f_84241f4bc9c8.slice/crio-2c160832736bd4ad6cbf21cea76b96397ee3999e9726d4c34ae6564125bc9628 WatchSource:0}: Error finding container 2c160832736bd4ad6cbf21cea76b96397ee3999e9726d4c34ae6564125bc9628: Status 404 returned error can't find the container with id 2c160832736bd4ad6cbf21cea76b96397ee3999e9726d4c34ae6564125bc9628
Oct 30 01:52:36.198384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.198305    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"]
Oct 30 01:52:36.204249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.204227    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"
Oct 30 01:52:36.225357 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.225330    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"]
Oct 30 01:52:36.297246 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.297208    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ec74f1-5a05-48eb-967d-c478b39ec884-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h\" (UID: \"c1ec74f1-5a05-48eb-967d-c478b39ec884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"
Oct 30 01:52:36.297420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.297295    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ec74f1-5a05-48eb-967d-c478b39ec884-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h\" (UID: \"c1ec74f1-5a05-48eb-967d-c478b39ec884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"
Oct 30 01:52:36.297420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.297362    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmxbh\" (UniqueName: \"kubernetes.io/projected/c1ec74f1-5a05-48eb-967d-c478b39ec884-kube-api-access-qmxbh\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h\" (UID: \"c1ec74f1-5a05-48eb-967d-c478b39ec884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"
Oct 30 01:52:36.333516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.333479    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk" event={"ID":"e622d47c-b9a0-4307-863f-84241f4bc9c8","Type":"ContainerStarted","Data":"09390c33e53705893963c0fefd4bb0a17091d12d05d5074254042efbf8f1a093"}
Oct 30 01:52:36.333677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.333533    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk" event={"ID":"e622d47c-b9a0-4307-863f-84241f4bc9c8","Type":"ContainerStarted","Data":"2c160832736bd4ad6cbf21cea76b96397ee3999e9726d4c34ae6564125bc9628"}
Oct 30 01:52:36.371826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.371775    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk" podStartSLOduration=2.371761627 podStartE2EDuration="2.371761627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:36.368721498 +0000 UTC m=+4265.118376819" watchObservedRunningTime="2026-10-30 01:52:36.371761627 +0000 UTC m=+4265.121416969"
Oct 30 01:52:36.391722 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.391689    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"]
Oct 30 01:52:36.397435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.397407    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"
Oct 30 01:52:36.398251 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.398225    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ec74f1-5a05-48eb-967d-c478b39ec884-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h\" (UID: \"c1ec74f1-5a05-48eb-967d-c478b39ec884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"
Oct 30 01:52:36.398362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.398258    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ec74f1-5a05-48eb-967d-c478b39ec884-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h\" (UID: \"c1ec74f1-5a05-48eb-967d-c478b39ec884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"
Oct 30 01:52:36.398362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.398307    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmxbh\" (UniqueName: \"kubernetes.io/projected/c1ec74f1-5a05-48eb-967d-c478b39ec884-kube-api-access-qmxbh\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h\" (UID: \"c1ec74f1-5a05-48eb-967d-c478b39ec884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"
Oct 30 01:52:36.399059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.399035    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ec74f1-5a05-48eb-967d-c478b39ec884-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h\" (UID: \"c1ec74f1-5a05-48eb-967d-c478b39ec884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"
Oct 30 01:52:36.400830 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.400803    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ec74f1-5a05-48eb-967d-c478b39ec884-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h\" (UID: \"c1ec74f1-5a05-48eb-967d-c478b39ec884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"
Oct 30 01:52:36.422208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.421931    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"]
Oct 30 01:52:36.436695 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.436667    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmxbh\" (UniqueName: \"kubernetes.io/projected/c1ec74f1-5a05-48eb-967d-c478b39ec884-kube-api-access-qmxbh\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h\" (UID: \"c1ec74f1-5a05-48eb-967d-c478b39ec884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"
Oct 30 01:52:36.499621 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.499586    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06fcef97-63bc-454d-a993-5dc46557cf81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f\" (UID: \"06fcef97-63bc-454d-a993-5dc46557cf81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"
Oct 30 01:52:36.499812 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.499789    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsq27\" (UniqueName: \"kubernetes.io/projected/06fcef97-63bc-454d-a993-5dc46557cf81-kube-api-access-nsq27\") pod \"maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f\" (UID: \"06fcef97-63bc-454d-a993-5dc46557cf81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"
Oct 30 01:52:36.499876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.499825    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06fcef97-63bc-454d-a993-5dc46557cf81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f\" (UID: \"06fcef97-63bc-454d-a993-5dc46557cf81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"
Oct 30 01:52:36.520082 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.520051    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"
Oct 30 01:52:36.528098 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.528071    4425 scope.go:122] "RemoveContainer" containerID="0bafa7f2ccf0330d40868e8c0fbd1becb609eea6ffd4f0d5e0c3c50767df4c42"
Oct 30 01:52:36.563850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.563818    4425 scope.go:122] "RemoveContainer" containerID="986e335a83caac437e4d0377cc1f9bb1a8ef2b628995aab5559748b07f7d16fd"
Oct 30 01:52:36.579980 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.579919    4425 scope.go:122] "RemoveContainer" containerID="7f33d774e4f19c3416c2be3f4fc69e3714111e6c603209b86883129b9bda237d"
Oct 30 01:52:36.614222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.602045    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsq27\" (UniqueName: \"kubernetes.io/projected/06fcef97-63bc-454d-a993-5dc46557cf81-kube-api-access-nsq27\") pod \"maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f\" (UID: \"06fcef97-63bc-454d-a993-5dc46557cf81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"
Oct 30 01:52:36.614222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.602101    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06fcef97-63bc-454d-a993-5dc46557cf81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f\" (UID: \"06fcef97-63bc-454d-a993-5dc46557cf81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"
Oct 30 01:52:36.614222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.602205    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06fcef97-63bc-454d-a993-5dc46557cf81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f\" (UID: \"06fcef97-63bc-454d-a993-5dc46557cf81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"
Oct 30 01:52:36.614222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.602995    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06fcef97-63bc-454d-a993-5dc46557cf81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f\" (UID: \"06fcef97-63bc-454d-a993-5dc46557cf81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"
Oct 30 01:52:36.614222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.613530    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06fcef97-63bc-454d-a993-5dc46557cf81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f\" (UID: \"06fcef97-63bc-454d-a993-5dc46557cf81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"
Oct 30 01:52:36.623425 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.623401    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsq27\" (UniqueName: \"kubernetes.io/projected/06fcef97-63bc-454d-a993-5dc46557cf81-kube-api-access-nsq27\") pod \"maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f\" (UID: \"06fcef97-63bc-454d-a993-5dc46557cf81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"
Oct 30 01:52:36.639006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.638982    4425 scope.go:122] "RemoveContainer" containerID="bdb67249427b4ded225c6defe86b76ee70702296882f40360f5d29af99a74316"
Oct 30 01:52:36.776305 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.776278    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"
Oct 30 01:52:36.835681 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:36.835639    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"]
Oct 30 01:52:36.842435 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:52:36.839998    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ec74f1_5a05_48eb_967d_c478b39ec884.slice/crio-512ab62d978cba9f616f9d0694a50c7c0d127f087ada38b556ecb27a54ab1a48 WatchSource:0}: Error finding container 512ab62d978cba9f616f9d0694a50c7c0d127f087ada38b556ecb27a54ab1a48: Status 404 returned error can't find the container with id 512ab62d978cba9f616f9d0694a50c7c0d127f087ada38b556ecb27a54ab1a48
Oct 30 01:52:37.079581 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:37.079552    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"]
Oct 30 01:52:37.080226 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:52:37.080196    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06fcef97_63bc_454d_a993_5dc46557cf81.slice/crio-24f16eca26d4bd244f576b6d3444016691d4f74d9f171cb416085c812c45e885 WatchSource:0}: Error finding container 24f16eca26d4bd244f576b6d3444016691d4f74d9f171cb416085c812c45e885: Status 404 returned error can't find the container with id 24f16eca26d4bd244f576b6d3444016691d4f74d9f171cb416085c812c45e885
Oct 30 01:52:37.342177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:37.342073    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f" event={"ID":"06fcef97-63bc-454d-a993-5dc46557cf81","Type":"ContainerStarted","Data":"24f16eca26d4bd244f576b6d3444016691d4f74d9f171cb416085c812c45e885"}
Oct 30 01:52:37.343975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:37.343934    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h" event={"ID":"c1ec74f1-5a05-48eb-967d-c478b39ec884","Type":"ContainerStarted","Data":"26dece2db76b053e0dd6bf7c346095ecbe703c222157987e5321241268f01bb8"}
Oct 30 01:52:37.344135 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:37.343986    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h" event={"ID":"c1ec74f1-5a05-48eb-967d-c478b39ec884","Type":"ContainerStarted","Data":"512ab62d978cba9f616f9d0694a50c7c0d127f087ada38b556ecb27a54ab1a48"}
Oct 30 01:52:37.345762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:37.345738    4425 generic.go:356] "Generic (PLEG): container finished" podID="e622d47c-b9a0-4307-863f-84241f4bc9c8" containerID="09390c33e53705893963c0fefd4bb0a17091d12d05d5074254042efbf8f1a093" exitCode=0
Oct 30 01:52:37.345875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:37.345810    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk" event={"ID":"e622d47c-b9a0-4307-863f-84241f4bc9c8","Type":"ContainerDied","Data":"09390c33e53705893963c0fefd4bb0a17091d12d05d5074254042efbf8f1a093"}
Oct 30 01:52:37.379232 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:37.379193    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h" podStartSLOduration=1.379181692 podStartE2EDuration="1.379181692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:37.378801716 +0000 UTC m=+4266.128457026" watchObservedRunningTime="2026-10-30 01:52:37.379181692 +0000 UTC m=+4266.128837028"
Oct 30 01:52:37.554516 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:52:37.554482    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:52:38.357358 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:38.357321    4425 generic.go:356] "Generic (PLEG): container finished" podID="c1ec74f1-5a05-48eb-967d-c478b39ec884" containerID="26dece2db76b053e0dd6bf7c346095ecbe703c222157987e5321241268f01bb8" exitCode=0
Oct 30 01:52:38.357758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:38.357403    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h" event={"ID":"c1ec74f1-5a05-48eb-967d-c478b39ec884","Type":"ContainerDied","Data":"26dece2db76b053e0dd6bf7c346095ecbe703c222157987e5321241268f01bb8"}
Oct 30 01:52:38.359165 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:38.359137    4425 generic.go:356] "Generic (PLEG): container finished" podID="06fcef97-63bc-454d-a993-5dc46557cf81" containerID="3080327d50bb7796cf0e5c90aa80f7014f9224d31e77df86b1afd1924514478e" exitCode=0
Oct 30 01:52:38.359304 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:38.359197    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f" event={"ID":"06fcef97-63bc-454d-a993-5dc46557cf81","Type":"ContainerDied","Data":"3080327d50bb7796cf0e5c90aa80f7014f9224d31e77df86b1afd1924514478e"}
Oct 30 01:52:38.516364 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:38.516332    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"
Oct 30 01:52:38.627230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:38.627122    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e622d47c-b9a0-4307-863f-84241f4bc9c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e622d47c-b9a0-4307-863f-84241f4bc9c8-maas-api-xks-ca\") pod \"e622d47c-b9a0-4307-863f-84241f4bc9c8\" (UID: \"e622d47c-b9a0-4307-863f-84241f4bc9c8\") "
Oct 30 01:52:38.627230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:38.627216    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e622d47c-b9a0-4307-863f-84241f4bc9c8-kube-api-access-kknjx\" (UniqueName: \"kubernetes.io/projected/e622d47c-b9a0-4307-863f-84241f4bc9c8-kube-api-access-kknjx\") pod \"e622d47c-b9a0-4307-863f-84241f4bc9c8\" (UID: \"e622d47c-b9a0-4307-863f-84241f4bc9c8\") "
Oct 30 01:52:38.627420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:38.627389    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e622d47c-b9a0-4307-863f-84241f4bc9c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e622d47c-b9a0-4307-863f-84241f4bc9c8-maas-api-openshift-ca\") pod \"e622d47c-b9a0-4307-863f-84241f4bc9c8\" (UID: \"e622d47c-b9a0-4307-863f-84241f4bc9c8\") "
Oct 30 01:52:38.627756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:38.627730    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e622d47c-b9a0-4307-863f-84241f4bc9c8-maas-api-openshift-ca" pod "e622d47c-b9a0-4307-863f-84241f4bc9c8" (UID: "e622d47c-b9a0-4307-863f-84241f4bc9c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:38.629148 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:38.629125    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e622d47c-b9a0-4307-863f-84241f4bc9c8-maas-api-xks-ca" pod "e622d47c-b9a0-4307-863f-84241f4bc9c8" (UID: "e622d47c-b9a0-4307-863f-84241f4bc9c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:52:38.629273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:38.629246    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e622d47c-b9a0-4307-863f-84241f4bc9c8-kube-api-access-kknjx" pod "e622d47c-b9a0-4307-863f-84241f4bc9c8" (UID: "e622d47c-b9a0-4307-863f-84241f4bc9c8"). InnerVolumeSpecName "kube-api-access-kknjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:38.728925 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:38.728895    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kknjx\" (UniqueName: \"kubernetes.io/projected/e622d47c-b9a0-4307-863f-84241f4bc9c8-kube-api-access-kknjx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:38.728925 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:38.728922    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e622d47c-b9a0-4307-863f-84241f4bc9c8-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:38.728925 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:38.728933    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e622d47c-b9a0-4307-863f-84241f4bc9c8-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:39.368556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.368522    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"
Oct 30 01:52:39.368556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.368531    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk" event={"ID":"e622d47c-b9a0-4307-863f-84241f4bc9c8","Type":"ContainerDied","Data":"2c160832736bd4ad6cbf21cea76b96397ee3999e9726d4c34ae6564125bc9628"}
Oct 30 01:52:39.369091 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.368587    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c160832736bd4ad6cbf21cea76b96397ee3999e9726d4c34ae6564125bc9628"
Oct 30 01:52:39.530104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.530071    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"
Oct 30 01:52:39.561627 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.561605    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"
Oct 30 01:52:39.640478 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.640391    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ec74f1-5a05-48eb-967d-c478b39ec884-kube-api-access-qmxbh\" (UniqueName: \"kubernetes.io/projected/c1ec74f1-5a05-48eb-967d-c478b39ec884-kube-api-access-qmxbh\") pod \"c1ec74f1-5a05-48eb-967d-c478b39ec884\" (UID: \"c1ec74f1-5a05-48eb-967d-c478b39ec884\") "
Oct 30 01:52:39.640478 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.640467    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1ec74f1-5a05-48eb-967d-c478b39ec884-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ec74f1-5a05-48eb-967d-c478b39ec884-maas-api-xks-ca\") pod \"c1ec74f1-5a05-48eb-967d-c478b39ec884\" (UID: \"c1ec74f1-5a05-48eb-967d-c478b39ec884\") "
Oct 30 01:52:39.640723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.640632    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06fcef97-63bc-454d-a993-5dc46557cf81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06fcef97-63bc-454d-a993-5dc46557cf81-maas-api-openshift-ca\") pod \"06fcef97-63bc-454d-a993-5dc46557cf81\" (UID: \"06fcef97-63bc-454d-a993-5dc46557cf81\") "
Oct 30 01:52:39.640862 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.640838    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06fcef97-63bc-454d-a993-5dc46557cf81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06fcef97-63bc-454d-a993-5dc46557cf81-maas-api-xks-ca\") pod \"06fcef97-63bc-454d-a993-5dc46557cf81\" (UID: \"06fcef97-63bc-454d-a993-5dc46557cf81\") "
Oct 30 01:52:39.641025 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.641007    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ec74f1-5a05-48eb-967d-c478b39ec884-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ec74f1-5a05-48eb-967d-c478b39ec884-maas-api-openshift-ca\") pod \"c1ec74f1-5a05-48eb-967d-c478b39ec884\" (UID: \"c1ec74f1-5a05-48eb-967d-c478b39ec884\") "
Oct 30 01:52:39.641195 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.641171    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06fcef97-63bc-454d-a993-5dc46557cf81-kube-api-access-nsq27\" (UniqueName: \"kubernetes.io/projected/06fcef97-63bc-454d-a993-5dc46557cf81-kube-api-access-nsq27\") pod \"06fcef97-63bc-454d-a993-5dc46557cf81\" (UID: \"06fcef97-63bc-454d-a993-5dc46557cf81\") "
Oct 30 01:52:39.641431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.641015    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06fcef97-63bc-454d-a993-5dc46557cf81-maas-api-openshift-ca" pod "06fcef97-63bc-454d-a993-5dc46557cf81" (UID: "06fcef97-63bc-454d-a993-5dc46557cf81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:39.641431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.641368    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ec74f1-5a05-48eb-967d-c478b39ec884-maas-api-openshift-ca" pod "c1ec74f1-5a05-48eb-967d-c478b39ec884" (UID: "c1ec74f1-5a05-48eb-967d-c478b39ec884"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:39.643462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.643436    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1ec74f1-5a05-48eb-967d-c478b39ec884-maas-api-xks-ca" pod "c1ec74f1-5a05-48eb-967d-c478b39ec884" (UID: "c1ec74f1-5a05-48eb-967d-c478b39ec884"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:52:39.643764 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.643737    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06fcef97-63bc-454d-a993-5dc46557cf81-maas-api-xks-ca" pod "06fcef97-63bc-454d-a993-5dc46557cf81" (UID: "06fcef97-63bc-454d-a993-5dc46557cf81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:52:39.644120 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.644102    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ec74f1-5a05-48eb-967d-c478b39ec884-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:39.644820 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.644125    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06fcef97-63bc-454d-a993-5dc46557cf81-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:39.644820 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.644139    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06fcef97-63bc-454d-a993-5dc46557cf81-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:39.644820 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.644189    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ec74f1-5a05-48eb-967d-c478b39ec884-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:39.644820 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.644436    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ec74f1-5a05-48eb-967d-c478b39ec884-kube-api-access-qmxbh" pod "c1ec74f1-5a05-48eb-967d-c478b39ec884" (UID: "c1ec74f1-5a05-48eb-967d-c478b39ec884"). InnerVolumeSpecName "kube-api-access-qmxbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:39.645125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.645104    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06fcef97-63bc-454d-a993-5dc46557cf81-kube-api-access-nsq27" pod "06fcef97-63bc-454d-a993-5dc46557cf81" (UID: "06fcef97-63bc-454d-a993-5dc46557cf81"). InnerVolumeSpecName "kube-api-access-nsq27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:39.745466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.745439    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsq27\" (UniqueName: \"kubernetes.io/projected/06fcef97-63bc-454d-a993-5dc46557cf81-kube-api-access-nsq27\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:39.745466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:39.745466    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmxbh\" (UniqueName: \"kubernetes.io/projected/c1ec74f1-5a05-48eb-967d-c478b39ec884-kube-api-access-qmxbh\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:40.377329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:40.377297    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"
Oct 30 01:52:40.377329 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:40.377319    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f" event={"ID":"06fcef97-63bc-454d-a993-5dc46557cf81","Type":"ContainerDied","Data":"24f16eca26d4bd244f576b6d3444016691d4f74d9f171cb416085c812c45e885"}
Oct 30 01:52:40.377893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:40.377369    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24f16eca26d4bd244f576b6d3444016691d4f74d9f171cb416085c812c45e885"
Oct 30 01:52:40.378877 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:40.378850    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"
Oct 30 01:52:40.378877 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:40.378858    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h" event={"ID":"c1ec74f1-5a05-48eb-967d-c478b39ec884","Type":"ContainerDied","Data":"512ab62d978cba9f616f9d0694a50c7c0d127f087ada38b556ecb27a54ab1a48"}
Oct 30 01:52:40.379030 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:40.378903    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="512ab62d978cba9f616f9d0694a50c7c0d127f087ada38b556ecb27a54ab1a48"
Oct 30 01:52:41.629524 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:41.629497    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"
Oct 30 01:52:41.661015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:41.660989    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"
Oct 30 01:52:41.666629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:41.666605    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2fecb6d-0b3b-4b1c-9806-74e68559265a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fecb6d-0b3b-4b1c-9806-74e68559265a-plugins-config-volume\") pod \"d2fecb6d-0b3b-4b1c-9806-74e68559265a\" (UID: \"d2fecb6d-0b3b-4b1c-9806-74e68559265a\") "
Oct 30 01:52:41.666762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:41.666666    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2fecb6d-0b3b-4b1c-9806-74e68559265a-kube-api-access-hmrcz\" (UniqueName: \"kubernetes.io/projected/d2fecb6d-0b3b-4b1c-9806-74e68559265a-kube-api-access-hmrcz\") pod \"d2fecb6d-0b3b-4b1c-9806-74e68559265a\" (UID: \"d2fecb6d-0b3b-4b1c-9806-74e68559265a\") "
Oct 30 01:52:41.667011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:41.666983    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2fecb6d-0b3b-4b1c-9806-74e68559265a-plugins-config-volume" pod "d2fecb6d-0b3b-4b1c-9806-74e68559265a" (UID: "d2fecb6d-0b3b-4b1c-9806-74e68559265a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:41.667185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:41.667169    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2fecb6d-0b3b-4b1c-9806-74e68559265a-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:41.668771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:41.668754    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fecb6d-0b3b-4b1c-9806-74e68559265a-kube-api-access-hmrcz" pod "d2fecb6d-0b3b-4b1c-9806-74e68559265a" (UID: "d2fecb6d-0b3b-4b1c-9806-74e68559265a"). InnerVolumeSpecName "kube-api-access-hmrcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:41.767946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:41.767908    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b86ac41-fb47-4c59-8723-9223d01f49f3-kube-api-access-snzqf\" (UniqueName: \"kubernetes.io/projected/8b86ac41-fb47-4c59-8723-9223d01f49f3-kube-api-access-snzqf\") pod \"8b86ac41-fb47-4c59-8723-9223d01f49f3\" (UID: \"8b86ac41-fb47-4c59-8723-9223d01f49f3\") "
Oct 30 01:52:41.768117 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:41.768108    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b86ac41-fb47-4c59-8723-9223d01f49f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b86ac41-fb47-4c59-8723-9223d01f49f3-plugins-config-volume\") pod \"8b86ac41-fb47-4c59-8723-9223d01f49f3\" (UID: \"8b86ac41-fb47-4c59-8723-9223d01f49f3\") "
Oct 30 01:52:41.768498 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:41.768469    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b86ac41-fb47-4c59-8723-9223d01f49f3-plugins-config-volume" pod "8b86ac41-fb47-4c59-8723-9223d01f49f3" (UID: "8b86ac41-fb47-4c59-8723-9223d01f49f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:41.768635 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:41.768584    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b86ac41-fb47-4c59-8723-9223d01f49f3-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:41.768635 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:41.768597    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmrcz\" (UniqueName: \"kubernetes.io/projected/d2fecb6d-0b3b-4b1c-9806-74e68559265a-kube-api-access-hmrcz\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:41.770006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:41.769979    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b86ac41-fb47-4c59-8723-9223d01f49f3-kube-api-access-snzqf" pod "8b86ac41-fb47-4c59-8723-9223d01f49f3" (UID: "8b86ac41-fb47-4c59-8723-9223d01f49f3"). InnerVolumeSpecName "kube-api-access-snzqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:41.869660 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:41.869597    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snzqf\" (UniqueName: \"kubernetes.io/projected/8b86ac41-fb47-4c59-8723-9223d01f49f3-kube-api-access-snzqf\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:42.402732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.402691    4425 generic.go:356] "Generic (PLEG): container finished" podID="8b86ac41-fb47-4c59-8723-9223d01f49f3" containerID="c58c54a09c3ae43aa427c5d4177dacfd9fa0fcc23903af8ed1e20a9cecc66f41" exitCode=0
Oct 30 01:52:42.403040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.403011    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"
Oct 30 01:52:42.403214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.403189    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t" event={"ID":"8b86ac41-fb47-4c59-8723-9223d01f49f3","Type":"ContainerDied","Data":"c58c54a09c3ae43aa427c5d4177dacfd9fa0fcc23903af8ed1e20a9cecc66f41"}
Oct 30 01:52:42.403449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.403430    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t" event={"ID":"8b86ac41-fb47-4c59-8723-9223d01f49f3","Type":"ContainerDied","Data":"13d1f9b8292c9ceac4da0e5d3ebf424853264f4d2ab0a8ef0128d755e02e80dc"}
Oct 30 01:52:42.403590 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.403576    4425 scope.go:122] "RemoveContainer" containerID="c58c54a09c3ae43aa427c5d4177dacfd9fa0fcc23903af8ed1e20a9cecc66f41"
Oct 30 01:52:42.405739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.405712    4425 generic.go:356] "Generic (PLEG): container finished" podID="d2fecb6d-0b3b-4b1c-9806-74e68559265a" containerID="0ad555280f5a67d164712906536283da051527390cd413001c6b33790be0ff44" exitCode=0
Oct 30 01:52:42.405992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.405972    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm" event={"ID":"d2fecb6d-0b3b-4b1c-9806-74e68559265a","Type":"ContainerDied","Data":"0ad555280f5a67d164712906536283da051527390cd413001c6b33790be0ff44"}
Oct 30 01:52:42.406079 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.406035    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm" event={"ID":"d2fecb6d-0b3b-4b1c-9806-74e68559265a","Type":"ContainerDied","Data":"66e8b7cc31f04549b6d80a4a59dea67601afcc29d3f5561d0238797bc399ec13"}
Oct 30 01:52:42.406142 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.406128    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"
Oct 30 01:52:42.420005 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.419988    4425 scope.go:122] "RemoveContainer" containerID="c58c54a09c3ae43aa427c5d4177dacfd9fa0fcc23903af8ed1e20a9cecc66f41"
Oct 30 01:52:42.420278 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:52:42.420257    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58c54a09c3ae43aa427c5d4177dacfd9fa0fcc23903af8ed1e20a9cecc66f41\": container with ID starting with c58c54a09c3ae43aa427c5d4177dacfd9fa0fcc23903af8ed1e20a9cecc66f41 not found: ID does not exist" containerID="c58c54a09c3ae43aa427c5d4177dacfd9fa0fcc23903af8ed1e20a9cecc66f41"
Oct 30 01:52:42.420351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.420285    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58c54a09c3ae43aa427c5d4177dacfd9fa0fcc23903af8ed1e20a9cecc66f41"} err="failed to get container status \"c58c54a09c3ae43aa427c5d4177dacfd9fa0fcc23903af8ed1e20a9cecc66f41\": rpc error: code = NotFound desc = could not find container \"c58c54a09c3ae43aa427c5d4177dacfd9fa0fcc23903af8ed1e20a9cecc66f41\": container with ID starting with c58c54a09c3ae43aa427c5d4177dacfd9fa0fcc23903af8ed1e20a9cecc66f41 not found: ID does not exist"
Oct 30 01:52:42.420351 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.420301    4425 scope.go:122] "RemoveContainer" containerID="0ad555280f5a67d164712906536283da051527390cd413001c6b33790be0ff44"
Oct 30 01:52:42.429815 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.429798    4425 scope.go:122] "RemoveContainer" containerID="0ad555280f5a67d164712906536283da051527390cd413001c6b33790be0ff44"
Oct 30 01:52:42.430057 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:52:42.430038    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad555280f5a67d164712906536283da051527390cd413001c6b33790be0ff44\": container with ID starting with 0ad555280f5a67d164712906536283da051527390cd413001c6b33790be0ff44 not found: ID does not exist" containerID="0ad555280f5a67d164712906536283da051527390cd413001c6b33790be0ff44"
Oct 30 01:52:42.430127 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.430062    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad555280f5a67d164712906536283da051527390cd413001c6b33790be0ff44"} err="failed to get container status \"0ad555280f5a67d164712906536283da051527390cd413001c6b33790be0ff44\": rpc error: code = NotFound desc = could not find container \"0ad555280f5a67d164712906536283da051527390cd413001c6b33790be0ff44\": container with ID starting with 0ad555280f5a67d164712906536283da051527390cd413001c6b33790be0ff44 not found: ID does not exist"
Oct 30 01:52:42.465081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.465058    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"]
Oct 30 01:52:42.474229 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.474207    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-499482-79b5ccd7dc-hjw6t"]
Oct 30 01:52:42.503822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.503800    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"]
Oct 30 01:52:42.510200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:42.510176    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-499482-688697fd76-v4hzm"]
Oct 30 01:52:43.423679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:43.423645    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b86ac41-fb47-4c59-8723-9223d01f49f3" path="/var/lib/kubelet/pods/8b86ac41-fb47-4c59-8723-9223d01f49f3/volumes"
Oct 30 01:52:43.424205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:43.424188    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2fecb6d-0b3b-4b1c-9806-74e68559265a" path="/var/lib/kubelet/pods/d2fecb6d-0b3b-4b1c-9806-74e68559265a/volumes"
Oct 30 01:52:45.600464 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.600385    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"]
Oct 30 01:52:45.601044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.601030    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8b86ac41-fb47-4c59-8723-9223d01f49f3" containerName="payload-pre-processing"
Oct 30 01:52:45.601108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.601044    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b86ac41-fb47-4c59-8723-9223d01f49f3" containerName="payload-pre-processing"
Oct 30 01:52:45.601108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.601057    4425 cpu_manager.go:414] "containerMap: removing container" podUID="06fcef97-63bc-454d-a993-5dc46557cf81" containerName="revoke-keys"
Oct 30 01:52:45.601108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.601062    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06fcef97-63bc-454d-a993-5dc46557cf81" containerName="revoke-keys"
Oct 30 01:52:45.601108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.601082    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d2fecb6d-0b3b-4b1c-9806-74e68559265a" containerName="payload-processing"
Oct 30 01:52:45.601108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.601087    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2fecb6d-0b3b-4b1c-9806-74e68559265a" containerName="payload-processing"
Oct 30 01:52:45.601108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.601103    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c1ec74f1-5a05-48eb-967d-c478b39ec884" containerName="revoke-keys"
Oct 30 01:52:45.601108 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.601108    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ec74f1-5a05-48eb-967d-c478b39ec884" containerName="revoke-keys"
Oct 30 01:52:45.601509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.601117    4425 cpu_manager.go:414] "containerMap: removing container" podUID="e622d47c-b9a0-4307-863f-84241f4bc9c8" containerName="revoke-keys"
Oct 30 01:52:45.601509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.601124    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e622d47c-b9a0-4307-863f-84241f4bc9c8" containerName="revoke-keys"
Oct 30 01:52:45.601509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.601281    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b86ac41-fb47-4c59-8723-9223d01f49f3" containerName="payload-pre-processing"
Oct 30 01:52:45.601509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.601297    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="06fcef97-63bc-454d-a993-5dc46557cf81" containerName="revoke-keys"
Oct 30 01:52:45.601509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.601313    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2fecb6d-0b3b-4b1c-9806-74e68559265a" containerName="payload-processing"
Oct 30 01:52:45.601509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.601328    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ec74f1-5a05-48eb-967d-c478b39ec884" containerName="revoke-keys"
Oct 30 01:52:45.601509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.601335    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e622d47c-b9a0-4307-863f-84241f4bc9c8" containerName="revoke-keys"
Oct 30 01:52:45.606513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.606495    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"
Oct 30 01:52:45.624669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.624400    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"]
Oct 30 01:52:45.635187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.629828    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:52:45.635187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.629975    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:52:45.715461 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.715428    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6164abfa-2fd7-4e60-946d-19efeb886751-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19f51ebe-fp2vg\" (UID: \"6164abfa-2fd7-4e60-946d-19efeb886751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"
Oct 30 01:52:45.715604 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.715495    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6164abfa-2fd7-4e60-946d-19efeb886751-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19f51ebe-fp2vg\" (UID: \"6164abfa-2fd7-4e60-946d-19efeb886751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"
Oct 30 01:52:45.715654 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.715616    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc44c\" (UniqueName: \"kubernetes.io/projected/6164abfa-2fd7-4e60-946d-19efeb886751-kube-api-access-hc44c\") pod \"maas-api-revoke-sub-19f51ebe-fp2vg\" (UID: \"6164abfa-2fd7-4e60-946d-19efeb886751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"
Oct 30 01:52:45.817382 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.817346    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6164abfa-2fd7-4e60-946d-19efeb886751-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19f51ebe-fp2vg\" (UID: \"6164abfa-2fd7-4e60-946d-19efeb886751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"
Oct 30 01:52:45.817572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.817429    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc44c\" (UniqueName: \"kubernetes.io/projected/6164abfa-2fd7-4e60-946d-19efeb886751-kube-api-access-hc44c\") pod \"maas-api-revoke-sub-19f51ebe-fp2vg\" (UID: \"6164abfa-2fd7-4e60-946d-19efeb886751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"
Oct 30 01:52:45.817624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.817582    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6164abfa-2fd7-4e60-946d-19efeb886751-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19f51ebe-fp2vg\" (UID: \"6164abfa-2fd7-4e60-946d-19efeb886751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"
Oct 30 01:52:45.818234 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.818204    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6164abfa-2fd7-4e60-946d-19efeb886751-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19f51ebe-fp2vg\" (UID: \"6164abfa-2fd7-4e60-946d-19efeb886751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"
Oct 30 01:52:45.820539 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.820510    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6164abfa-2fd7-4e60-946d-19efeb886751-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19f51ebe-fp2vg\" (UID: \"6164abfa-2fd7-4e60-946d-19efeb886751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"
Oct 30 01:52:45.838754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.838721    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc44c\" (UniqueName: \"kubernetes.io/projected/6164abfa-2fd7-4e60-946d-19efeb886751-kube-api-access-hc44c\") pod \"maas-api-revoke-sub-19f51ebe-fp2vg\" (UID: \"6164abfa-2fd7-4e60-946d-19efeb886751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"
Oct 30 01:52:45.925136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:45.925075    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"
Oct 30 01:52:46.036222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:46.036185    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"]
Oct 30 01:52:46.042036 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:46.042001    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a9d843e-ll784"]
Oct 30 01:52:46.226932 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:52:46.226904    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6164abfa_2fd7_4e60_946d_19efeb886751.slice/crio-b07d5d7d5c0b53f9d0ab579ff1cb2ef15d5ae443d2f0b1952d1c262d871b8e26 WatchSource:0}: Error finding container b07d5d7d5c0b53f9d0ab579ff1cb2ef15d5ae443d2f0b1952d1c262d871b8e26: Status 404 returned error can't find the container with id b07d5d7d5c0b53f9d0ab579ff1cb2ef15d5ae443d2f0b1952d1c262d871b8e26
Oct 30 01:52:46.227483 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:46.227460    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"]
Oct 30 01:52:46.438573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:46.438528    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg" event={"ID":"6164abfa-2fd7-4e60-946d-19efeb886751","Type":"ContainerStarted","Data":"b07d5d7d5c0b53f9d0ab579ff1cb2ef15d5ae443d2f0b1952d1c262d871b8e26"}
Oct 30 01:52:46.621020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:46.620992    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"]
Oct 30 01:52:46.654732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:46.654706    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"]
Oct 30 01:52:47.426049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:47.425968    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5023f3d5-3dda-44af-88f6-8ce25e6f84f4" path="/var/lib/kubelet/pods/5023f3d5-3dda-44af-88f6-8ce25e6f84f4/volumes"
Oct 30 01:52:47.435723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:47.435698    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"]
Oct 30 01:52:47.448610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:47.448576    4425 generic.go:356] "Generic (PLEG): container finished" podID="6164abfa-2fd7-4e60-946d-19efeb886751" containerID="c02fae2976a9dfcd04299623ae07ca44cb6052daff2e59caa0e816f7611f7226" exitCode=0
Oct 30 01:52:47.448767 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:47.448672    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg" event={"ID":"6164abfa-2fd7-4e60-946d-19efeb886751","Type":"ContainerDied","Data":"c02fae2976a9dfcd04299623ae07ca44cb6052daff2e59caa0e816f7611f7226"}
Oct 30 01:52:48.610643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:48.610614    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"
Oct 30 01:52:48.756015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:48.755911    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6164abfa-2fd7-4e60-946d-19efeb886751-kube-api-access-hc44c\" (UniqueName: \"kubernetes.io/projected/6164abfa-2fd7-4e60-946d-19efeb886751-kube-api-access-hc44c\") pod \"6164abfa-2fd7-4e60-946d-19efeb886751\" (UID: \"6164abfa-2fd7-4e60-946d-19efeb886751\") "
Oct 30 01:52:48.756015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:48.755967    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6164abfa-2fd7-4e60-946d-19efeb886751-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6164abfa-2fd7-4e60-946d-19efeb886751-maas-api-xks-ca\") pod \"6164abfa-2fd7-4e60-946d-19efeb886751\" (UID: \"6164abfa-2fd7-4e60-946d-19efeb886751\") "
Oct 30 01:52:48.756279 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:48.756127    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6164abfa-2fd7-4e60-946d-19efeb886751-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6164abfa-2fd7-4e60-946d-19efeb886751-maas-api-openshift-ca\") pod \"6164abfa-2fd7-4e60-946d-19efeb886751\" (UID: \"6164abfa-2fd7-4e60-946d-19efeb886751\") "
Oct 30 01:52:48.756545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:48.756517    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6164abfa-2fd7-4e60-946d-19efeb886751-maas-api-openshift-ca" pod "6164abfa-2fd7-4e60-946d-19efeb886751" (UID: "6164abfa-2fd7-4e60-946d-19efeb886751"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:48.756675 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:48.756655    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6164abfa-2fd7-4e60-946d-19efeb886751-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:48.758502 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:48.758463    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6164abfa-2fd7-4e60-946d-19efeb886751-maas-api-xks-ca" pod "6164abfa-2fd7-4e60-946d-19efeb886751" (UID: "6164abfa-2fd7-4e60-946d-19efeb886751"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:52:48.758756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:48.758727    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6164abfa-2fd7-4e60-946d-19efeb886751-kube-api-access-hc44c" pod "6164abfa-2fd7-4e60-946d-19efeb886751" (UID: "6164abfa-2fd7-4e60-946d-19efeb886751"). InnerVolumeSpecName "kube-api-access-hc44c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:48.857849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:48.857817    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc44c\" (UniqueName: \"kubernetes.io/projected/6164abfa-2fd7-4e60-946d-19efeb886751-kube-api-access-hc44c\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:48.857849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:48.857845    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6164abfa-2fd7-4e60-946d-19efeb886751-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:49.464839 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:49.464815    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"
Oct 30 01:52:49.465003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:49.464809    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg" event={"ID":"6164abfa-2fd7-4e60-946d-19efeb886751","Type":"ContainerDied","Data":"b07d5d7d5c0b53f9d0ab579ff1cb2ef15d5ae443d2f0b1952d1c262d871b8e26"}
Oct 30 01:52:49.465003 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:49.464948    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b07d5d7d5c0b53f9d0ab579ff1cb2ef15d5ae443d2f0b1952d1c262d871b8e26"
Oct 30 01:52:51.511451 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:51.511420    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"]
Oct 30 01:52:51.520881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:51.520856    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-499482-5llnm"]
Oct 30 01:52:51.622240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:51.622162    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt" podUID="892671e5-857b-4816-9376-f6862e47dd40" containerName="payload-processing" containerID="cri-o://eaab8886600a3a447a871bd80c08af7cc20054d24eb777cc39e8c53d6fa82b95" gracePeriod=25
Oct 30 01:52:51.655492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:51.655451    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm" podUID="fde72706-a81b-4b93-96c6-f02dd6a96f3c" containerName="payload-pre-processing" containerID="cri-o://3f4f2ef4769034a517ef1a0b01cff5625b68888378cbdfa8f47546388451c002" gracePeriod=25
Oct 30 01:52:51.936574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:51.936544    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"
Oct 30 01:52:51.971244 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:51.971224    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"
Oct 30 01:52:52.095722 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.095641    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde72706-a81b-4b93-96c6-f02dd6a96f3c-kube-api-access-47qhb\" (UniqueName: \"kubernetes.io/projected/fde72706-a81b-4b93-96c6-f02dd6a96f3c-kube-api-access-47qhb\") pod \"fde72706-a81b-4b93-96c6-f02dd6a96f3c\" (UID: \"fde72706-a81b-4b93-96c6-f02dd6a96f3c\") "
Oct 30 01:52:52.095722 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.095710    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/892671e5-857b-4816-9376-f6862e47dd40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/892671e5-857b-4816-9376-f6862e47dd40-plugins-config-volume\") pod \"892671e5-857b-4816-9376-f6862e47dd40\" (UID: \"892671e5-857b-4816-9376-f6862e47dd40\") "
Oct 30 01:52:52.095943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.095833    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892671e5-857b-4816-9376-f6862e47dd40-kube-api-access-2htcp\" (UniqueName: \"kubernetes.io/projected/892671e5-857b-4816-9376-f6862e47dd40-kube-api-access-2htcp\") pod \"892671e5-857b-4816-9376-f6862e47dd40\" (UID: \"892671e5-857b-4816-9376-f6862e47dd40\") "
Oct 30 01:52:52.095943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.095854    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fde72706-a81b-4b93-96c6-f02dd6a96f3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde72706-a81b-4b93-96c6-f02dd6a96f3c-plugins-config-volume\") pod \"fde72706-a81b-4b93-96c6-f02dd6a96f3c\" (UID: \"fde72706-a81b-4b93-96c6-f02dd6a96f3c\") "
Oct 30 01:52:52.096144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.096116    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892671e5-857b-4816-9376-f6862e47dd40-plugins-config-volume" pod "892671e5-857b-4816-9376-f6862e47dd40" (UID: "892671e5-857b-4816-9376-f6862e47dd40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:52.096297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.096277    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/892671e5-857b-4816-9376-f6862e47dd40-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:52.096400 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.096377    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde72706-a81b-4b93-96c6-f02dd6a96f3c-plugins-config-volume" pod "fde72706-a81b-4b93-96c6-f02dd6a96f3c" (UID: "fde72706-a81b-4b93-96c6-f02dd6a96f3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:52.097842 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.097823    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde72706-a81b-4b93-96c6-f02dd6a96f3c-kube-api-access-47qhb" pod "fde72706-a81b-4b93-96c6-f02dd6a96f3c" (UID: "fde72706-a81b-4b93-96c6-f02dd6a96f3c"). InnerVolumeSpecName "kube-api-access-47qhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:52.097926 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.097857    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892671e5-857b-4816-9376-f6862e47dd40-kube-api-access-2htcp" pod "892671e5-857b-4816-9376-f6862e47dd40" (UID: "892671e5-857b-4816-9376-f6862e47dd40"). InnerVolumeSpecName "kube-api-access-2htcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:52.197875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.197818    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2htcp\" (UniqueName: \"kubernetes.io/projected/892671e5-857b-4816-9376-f6862e47dd40-kube-api-access-2htcp\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:52.197875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.197872    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde72706-a81b-4b93-96c6-f02dd6a96f3c-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:52.197875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.197881    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47qhb\" (UniqueName: \"kubernetes.io/projected/fde72706-a81b-4b93-96c6-f02dd6a96f3c-kube-api-access-47qhb\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:52.437002 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.436921    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx" podUID="1addf2b0-5855-4a72-acba-a10a5bcc5ec0" containerName="payload-processing" containerID="cri-o://a641f90cd456e23891883c5fd9059ea39a467a76eaa9caabf844504b64c49bfe" gracePeriod=25
Oct 30 01:52:52.487397 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.487363    4425 generic.go:356] "Generic (PLEG): container finished" podID="892671e5-857b-4816-9376-f6862e47dd40" containerID="eaab8886600a3a447a871bd80c08af7cc20054d24eb777cc39e8c53d6fa82b95" exitCode=0
Oct 30 01:52:52.487522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.487422    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"
Oct 30 01:52:52.487522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.487450    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt" event={"ID":"892671e5-857b-4816-9376-f6862e47dd40","Type":"ContainerDied","Data":"eaab8886600a3a447a871bd80c08af7cc20054d24eb777cc39e8c53d6fa82b95"}
Oct 30 01:52:52.487522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.487506    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt" event={"ID":"892671e5-857b-4816-9376-f6862e47dd40","Type":"ContainerDied","Data":"579b6c97e66f1bbe4a78bfd12dcaa3c7af2f1252bbea59b2b5aba4f476bdfa08"}
Oct 30 01:52:52.487900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.487534    4425 scope.go:122] "RemoveContainer" containerID="eaab8886600a3a447a871bd80c08af7cc20054d24eb777cc39e8c53d6fa82b95"
Oct 30 01:52:52.489059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.489028    4425 generic.go:356] "Generic (PLEG): container finished" podID="fde72706-a81b-4b93-96c6-f02dd6a96f3c" containerID="3f4f2ef4769034a517ef1a0b01cff5625b68888378cbdfa8f47546388451c002" exitCode=0
Oct 30 01:52:52.489188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.489053    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm" event={"ID":"fde72706-a81b-4b93-96c6-f02dd6a96f3c","Type":"ContainerDied","Data":"3f4f2ef4769034a517ef1a0b01cff5625b68888378cbdfa8f47546388451c002"}
Oct 30 01:52:52.489188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.489081    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"
Oct 30 01:52:52.489188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.489105    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm" event={"ID":"fde72706-a81b-4b93-96c6-f02dd6a96f3c","Type":"ContainerDied","Data":"a966910e15cbea6d8ac459a7303638785082006c35a268cf17e2dded9592fdfe"}
Oct 30 01:52:52.546700 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.546679    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"]
Oct 30 01:52:52.552586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.552559    4425 scope.go:122] "RemoveContainer" containerID="eaab8886600a3a447a871bd80c08af7cc20054d24eb777cc39e8c53d6fa82b95"
Oct 30 01:52:52.552893 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:52:52.552866    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaab8886600a3a447a871bd80c08af7cc20054d24eb777cc39e8c53d6fa82b95\": container with ID starting with eaab8886600a3a447a871bd80c08af7cc20054d24eb777cc39e8c53d6fa82b95 not found: ID does not exist" containerID="eaab8886600a3a447a871bd80c08af7cc20054d24eb777cc39e8c53d6fa82b95"
Oct 30 01:52:52.552949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.552907    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaab8886600a3a447a871bd80c08af7cc20054d24eb777cc39e8c53d6fa82b95"} err="failed to get container status \"eaab8886600a3a447a871bd80c08af7cc20054d24eb777cc39e8c53d6fa82b95\": rpc error: code = NotFound desc = could not find container \"eaab8886600a3a447a871bd80c08af7cc20054d24eb777cc39e8c53d6fa82b95\": container with ID starting with eaab8886600a3a447a871bd80c08af7cc20054d24eb777cc39e8c53d6fa82b95 not found: ID does not exist"
Oct 30 01:52:52.552949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.552933    4425 scope.go:122] "RemoveContainer" containerID="3f4f2ef4769034a517ef1a0b01cff5625b68888378cbdfa8f47546388451c002"
Oct 30 01:52:52.555324 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.555305    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1cb829-675559d77f-c76wm"]
Oct 30 01:52:52.562984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.562967    4425 scope.go:122] "RemoveContainer" containerID="3f4f2ef4769034a517ef1a0b01cff5625b68888378cbdfa8f47546388451c002"
Oct 30 01:52:52.563291 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:52:52.563271    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f4f2ef4769034a517ef1a0b01cff5625b68888378cbdfa8f47546388451c002\": container with ID starting with 3f4f2ef4769034a517ef1a0b01cff5625b68888378cbdfa8f47546388451c002 not found: ID does not exist" containerID="3f4f2ef4769034a517ef1a0b01cff5625b68888378cbdfa8f47546388451c002"
Oct 30 01:52:52.563368 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.563300    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f4f2ef4769034a517ef1a0b01cff5625b68888378cbdfa8f47546388451c002"} err="failed to get container status \"3f4f2ef4769034a517ef1a0b01cff5625b68888378cbdfa8f47546388451c002\": rpc error: code = NotFound desc = could not find container \"3f4f2ef4769034a517ef1a0b01cff5625b68888378cbdfa8f47546388451c002\": container with ID starting with 3f4f2ef4769034a517ef1a0b01cff5625b68888378cbdfa8f47546388451c002 not found: ID does not exist"
Oct 30 01:52:52.578379 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.578354    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"]
Oct 30 01:52:52.581940 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.581919    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1cb829-6545959fd-5ksrt"]
Oct 30 01:52:52.710220 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.710191    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"
Oct 30 01:52:52.803391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.803358    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1addf2b0-5855-4a72-acba-a10a5bcc5ec0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1addf2b0-5855-4a72-acba-a10a5bcc5ec0-plugins-config-volume\") pod \"1addf2b0-5855-4a72-acba-a10a5bcc5ec0\" (UID: \"1addf2b0-5855-4a72-acba-a10a5bcc5ec0\") "
Oct 30 01:52:52.803568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.803408    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1addf2b0-5855-4a72-acba-a10a5bcc5ec0-kube-api-access-wmnm7\" (UniqueName: \"kubernetes.io/projected/1addf2b0-5855-4a72-acba-a10a5bcc5ec0-kube-api-access-wmnm7\") pod \"1addf2b0-5855-4a72-acba-a10a5bcc5ec0\" (UID: \"1addf2b0-5855-4a72-acba-a10a5bcc5ec0\") "
Oct 30 01:52:52.803726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.803704    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1addf2b0-5855-4a72-acba-a10a5bcc5ec0-plugins-config-volume" pod "1addf2b0-5855-4a72-acba-a10a5bcc5ec0" (UID: "1addf2b0-5855-4a72-acba-a10a5bcc5ec0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:52:52.803901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.803886    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1addf2b0-5855-4a72-acba-a10a5bcc5ec0-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:52.805634 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.805607    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1addf2b0-5855-4a72-acba-a10a5bcc5ec0-kube-api-access-wmnm7" pod "1addf2b0-5855-4a72-acba-a10a5bcc5ec0" (UID: "1addf2b0-5855-4a72-acba-a10a5bcc5ec0"). InnerVolumeSpecName "kube-api-access-wmnm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:52:52.904821 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:52.904788    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmnm7\" (UniqueName: \"kubernetes.io/projected/1addf2b0-5855-4a72-acba-a10a5bcc5ec0-kube-api-access-wmnm7\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:52:53.425546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:53.425512    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b3f3a3-731a-48dc-9f75-7c7de0b3b317" path="/var/lib/kubelet/pods/60b3f3a3-731a-48dc-9f75-7c7de0b3b317/volumes"
Oct 30 01:52:53.425969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:53.425954    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892671e5-857b-4816-9376-f6862e47dd40" path="/var/lib/kubelet/pods/892671e5-857b-4816-9376-f6862e47dd40/volumes"
Oct 30 01:52:53.426366 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:53.426342    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde72706-a81b-4b93-96c6-f02dd6a96f3c" path="/var/lib/kubelet/pods/fde72706-a81b-4b93-96c6-f02dd6a96f3c/volumes"
Oct 30 01:52:53.499312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:53.499283    4425 generic.go:356] "Generic (PLEG): container finished" podID="1addf2b0-5855-4a72-acba-a10a5bcc5ec0" containerID="a641f90cd456e23891883c5fd9059ea39a467a76eaa9caabf844504b64c49bfe" exitCode=0
Oct 30 01:52:53.499470 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:53.499330    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"
Oct 30 01:52:53.499470 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:53.499368    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx" event={"ID":"1addf2b0-5855-4a72-acba-a10a5bcc5ec0","Type":"ContainerDied","Data":"a641f90cd456e23891883c5fd9059ea39a467a76eaa9caabf844504b64c49bfe"}
Oct 30 01:52:53.499470 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:53.499424    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx" event={"ID":"1addf2b0-5855-4a72-acba-a10a5bcc5ec0","Type":"ContainerDied","Data":"afb0ea4d5ac4a0d748d91cc9a765080fbd301eaf67f70190b7a8e861418d589e"}
Oct 30 01:52:53.499470 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:53.499453    4425 scope.go:122] "RemoveContainer" containerID="a641f90cd456e23891883c5fd9059ea39a467a76eaa9caabf844504b64c49bfe"
Oct 30 01:52:53.509806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:53.509775    4425 scope.go:122] "RemoveContainer" containerID="a641f90cd456e23891883c5fd9059ea39a467a76eaa9caabf844504b64c49bfe"
Oct 30 01:52:53.510072 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:52:53.510050    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a641f90cd456e23891883c5fd9059ea39a467a76eaa9caabf844504b64c49bfe\": container with ID starting with a641f90cd456e23891883c5fd9059ea39a467a76eaa9caabf844504b64c49bfe not found: ID does not exist" containerID="a641f90cd456e23891883c5fd9059ea39a467a76eaa9caabf844504b64c49bfe"
Oct 30 01:52:53.510140 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:53.510081    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a641f90cd456e23891883c5fd9059ea39a467a76eaa9caabf844504b64c49bfe"} err="failed to get container status \"a641f90cd456e23891883c5fd9059ea39a467a76eaa9caabf844504b64c49bfe\": rpc error: code = NotFound desc = could not find container \"a641f90cd456e23891883c5fd9059ea39a467a76eaa9caabf844504b64c49bfe\": container with ID starting with a641f90cd456e23891883c5fd9059ea39a467a76eaa9caabf844504b64c49bfe not found: ID does not exist"
Oct 30 01:52:53.551668 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:53.551646    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"]
Oct 30 01:52:53.561443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:53.561412    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-df4db1-767c9cf6f6-wqzzx"]
Oct 30 01:52:55.423514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:52:55.423484    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1addf2b0-5855-4a72-acba-a10a5bcc5ec0" path="/var/lib/kubelet/pods/1addf2b0-5855-4a72-acba-a10a5bcc5ec0/volumes"
Oct 30 01:53:00.798259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.798227    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"]
Oct 30 01:53:00.798839 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.798826    4425 cpu_manager.go:414] "containerMap: removing container" podUID="6164abfa-2fd7-4e60-946d-19efeb886751" containerName="revoke-keys"
Oct 30 01:53:00.798890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.798842    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6164abfa-2fd7-4e60-946d-19efeb886751" containerName="revoke-keys"
Oct 30 01:53:00.798890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.798866    4425 cpu_manager.go:414] "containerMap: removing container" podUID="1addf2b0-5855-4a72-acba-a10a5bcc5ec0" containerName="payload-processing"
Oct 30 01:53:00.798890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.798872    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1addf2b0-5855-4a72-acba-a10a5bcc5ec0" containerName="payload-processing"
Oct 30 01:53:00.798890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.798883    4425 cpu_manager.go:414] "containerMap: removing container" podUID="fde72706-a81b-4b93-96c6-f02dd6a96f3c" containerName="payload-pre-processing"
Oct 30 01:53:00.798890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.798887    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde72706-a81b-4b93-96c6-f02dd6a96f3c" containerName="payload-pre-processing"
Oct 30 01:53:00.799066 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.798905    4425 cpu_manager.go:414] "containerMap: removing container" podUID="892671e5-857b-4816-9376-f6862e47dd40" containerName="payload-processing"
Oct 30 01:53:00.799066 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.798910    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892671e5-857b-4816-9376-f6862e47dd40" containerName="payload-processing"
Oct 30 01:53:00.799066 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.798998    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6164abfa-2fd7-4e60-946d-19efeb886751" containerName="revoke-keys"
Oct 30 01:53:00.799066 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.799009    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="1addf2b0-5855-4a72-acba-a10a5bcc5ec0" containerName="payload-processing"
Oct 30 01:53:00.799066 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.799016    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde72706-a81b-4b93-96c6-f02dd6a96f3c" containerName="payload-pre-processing"
Oct 30 01:53:00.799066 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.799024    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="892671e5-857b-4816-9376-f6862e47dd40" containerName="payload-processing"
Oct 30 01:53:00.804782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.804762    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"
Oct 30 01:53:00.822473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.822450    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"]
Oct 30 01:53:00.827859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.827840    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:53:00.828061 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.828042    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:53:00.894324 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.894295    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668d389b-833e-42dc-90e7-951366e7b5cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr\" (UID: \"668d389b-833e-42dc-90e7-951366e7b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"
Oct 30 01:53:00.894519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.894499    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668d389b-833e-42dc-90e7-951366e7b5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr\" (UID: \"668d389b-833e-42dc-90e7-951366e7b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"
Oct 30 01:53:00.894564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.894532    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxzts\" (UniqueName: \"kubernetes.io/projected/668d389b-833e-42dc-90e7-951366e7b5cf-kube-api-access-mxzts\") pod \"maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr\" (UID: \"668d389b-833e-42dc-90e7-951366e7b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"
Oct 30 01:53:00.996360 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.996321    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668d389b-833e-42dc-90e7-951366e7b5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr\" (UID: \"668d389b-833e-42dc-90e7-951366e7b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"
Oct 30 01:53:00.996528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.996368    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxzts\" (UniqueName: \"kubernetes.io/projected/668d389b-833e-42dc-90e7-951366e7b5cf-kube-api-access-mxzts\") pod \"maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr\" (UID: \"668d389b-833e-42dc-90e7-951366e7b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"
Oct 30 01:53:00.996528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.996416    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668d389b-833e-42dc-90e7-951366e7b5cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr\" (UID: \"668d389b-833e-42dc-90e7-951366e7b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"
Oct 30 01:53:00.996984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.996960    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668d389b-833e-42dc-90e7-951366e7b5cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr\" (UID: \"668d389b-833e-42dc-90e7-951366e7b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"
Oct 30 01:53:00.998828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:00.998794    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668d389b-833e-42dc-90e7-951366e7b5cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr\" (UID: \"668d389b-833e-42dc-90e7-951366e7b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"
Oct 30 01:53:01.017599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.017564    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxzts\" (UniqueName: \"kubernetes.io/projected/668d389b-833e-42dc-90e7-951366e7b5cf-kube-api-access-mxzts\") pod \"maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr\" (UID: \"668d389b-833e-42dc-90e7-951366e7b5cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"
Oct 30 01:53:01.119073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.119008    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"
Oct 30 01:53:01.234141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.234104    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"]
Oct 30 01:53:01.436838 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.436757    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"]
Oct 30 01:53:01.441512 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:53:01.441479    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668d389b_833e_42dc_90e7_951366e7b5cf.slice/crio-5fcbfd70b4d1baf03b258096d4778656464994ef918400985434d793cd502c2a WatchSource:0}: Error finding container 5fcbfd70b4d1baf03b258096d4778656464994ef918400985434d793cd502c2a: Status 404 returned error can't find the container with id 5fcbfd70b4d1baf03b258096d4778656464994ef918400985434d793cd502c2a
Oct 30 01:53:01.538316 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.538281    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"]
Oct 30 01:53:01.544996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.544975    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"
Oct 30 01:53:01.559257 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.558509    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr" event={"ID":"668d389b-833e-42dc-90e7-951366e7b5cf","Type":"ContainerStarted","Data":"5fcbfd70b4d1baf03b258096d4778656464994ef918400985434d793cd502c2a"}
Oct 30 01:53:01.560965 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.560942    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"]
Oct 30 01:53:01.602233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.602206    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd9467ca-1998-4000-87e2-e64b3bb98c3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9\" (UID: \"bd9467ca-1998-4000-87e2-e64b3bb98c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"
Oct 30 01:53:01.602402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.602279    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcgdq\" (UniqueName: \"kubernetes.io/projected/bd9467ca-1998-4000-87e2-e64b3bb98c3c-kube-api-access-kcgdq\") pod \"maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9\" (UID: \"bd9467ca-1998-4000-87e2-e64b3bb98c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"
Oct 30 01:53:01.602478 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.602411    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd9467ca-1998-4000-87e2-e64b3bb98c3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9\" (UID: \"bd9467ca-1998-4000-87e2-e64b3bb98c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"
Oct 30 01:53:01.703897 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.703825    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd9467ca-1998-4000-87e2-e64b3bb98c3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9\" (UID: \"bd9467ca-1998-4000-87e2-e64b3bb98c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"
Oct 30 01:53:01.704048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.703934    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgdq\" (UniqueName: \"kubernetes.io/projected/bd9467ca-1998-4000-87e2-e64b3bb98c3c-kube-api-access-kcgdq\") pod \"maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9\" (UID: \"bd9467ca-1998-4000-87e2-e64b3bb98c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"
Oct 30 01:53:01.704048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.704010    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd9467ca-1998-4000-87e2-e64b3bb98c3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9\" (UID: \"bd9467ca-1998-4000-87e2-e64b3bb98c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"
Oct 30 01:53:01.704698 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.704672    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd9467ca-1998-4000-87e2-e64b3bb98c3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9\" (UID: \"bd9467ca-1998-4000-87e2-e64b3bb98c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"
Oct 30 01:53:01.706234 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.706214    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd9467ca-1998-4000-87e2-e64b3bb98c3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9\" (UID: \"bd9467ca-1998-4000-87e2-e64b3bb98c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"
Oct 30 01:53:01.725288 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.725263    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcgdq\" (UniqueName: \"kubernetes.io/projected/bd9467ca-1998-4000-87e2-e64b3bb98c3c-kube-api-access-kcgdq\") pod \"maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9\" (UID: \"bd9467ca-1998-4000-87e2-e64b3bb98c3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"
Oct 30 01:53:01.861810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:01.861778    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"
Oct 30 01:53:02.178642 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:02.178613    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"]
Oct 30 01:53:02.180215 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:53:02.180185    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9467ca_1998_4000_87e2_e64b3bb98c3c.slice/crio-7d861108ddd194d8dd39d02194fe44930ca0c97d2caa2e2bb310b837501cf447 WatchSource:0}: Error finding container 7d861108ddd194d8dd39d02194fe44930ca0c97d2caa2e2bb310b837501cf447: Status 404 returned error can't find the container with id 7d861108ddd194d8dd39d02194fe44930ca0c97d2caa2e2bb310b837501cf447
Oct 30 01:53:02.569841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:02.569814    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9" event={"ID":"bd9467ca-1998-4000-87e2-e64b3bb98c3c","Type":"ContainerStarted","Data":"7d861108ddd194d8dd39d02194fe44930ca0c97d2caa2e2bb310b837501cf447"}
Oct 30 01:53:02.572337 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:02.572309    4425 generic.go:356] "Generic (PLEG): container finished" podID="668d389b-833e-42dc-90e7-951366e7b5cf" containerID="f7d52d61772669b9fe17215332bb14d109aaac7d87836689ce880ceb89861cb8" exitCode=0
Oct 30 01:53:02.572447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:02.572369    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr" event={"ID":"668d389b-833e-42dc-90e7-951366e7b5cf","Type":"ContainerDied","Data":"f7d52d61772669b9fe17215332bb14d109aaac7d87836689ce880ceb89861cb8"}
Oct 30 01:53:03.581528 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:03.581494    4425 generic.go:356] "Generic (PLEG): container finished" podID="bd9467ca-1998-4000-87e2-e64b3bb98c3c" containerID="0a28f2dbb8ba90069c37bff99957fd881e276bba4504fb8ab269512e3587cb58" exitCode=0
Oct 30 01:53:03.581893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:03.581573    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9" event={"ID":"bd9467ca-1998-4000-87e2-e64b3bb98c3c","Type":"ContainerDied","Data":"0a28f2dbb8ba90069c37bff99957fd881e276bba4504fb8ab269512e3587cb58"}
Oct 30 01:53:03.743438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:03.743412    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"
Oct 30 01:53:03.838109 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:03.838021    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668d389b-833e-42dc-90e7-951366e7b5cf-kube-api-access-mxzts\" (UniqueName: \"kubernetes.io/projected/668d389b-833e-42dc-90e7-951366e7b5cf-kube-api-access-mxzts\") pod \"668d389b-833e-42dc-90e7-951366e7b5cf\" (UID: \"668d389b-833e-42dc-90e7-951366e7b5cf\") "
Oct 30 01:53:03.838109 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:03.838095    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/668d389b-833e-42dc-90e7-951366e7b5cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668d389b-833e-42dc-90e7-951366e7b5cf-maas-api-xks-ca\") pod \"668d389b-833e-42dc-90e7-951366e7b5cf\" (UID: \"668d389b-833e-42dc-90e7-951366e7b5cf\") "
Oct 30 01:53:03.838365 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:03.838252    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/668d389b-833e-42dc-90e7-951366e7b5cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668d389b-833e-42dc-90e7-951366e7b5cf-maas-api-openshift-ca\") pod \"668d389b-833e-42dc-90e7-951366e7b5cf\" (UID: \"668d389b-833e-42dc-90e7-951366e7b5cf\") "
Oct 30 01:53:03.838582 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:03.838557    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668d389b-833e-42dc-90e7-951366e7b5cf-maas-api-openshift-ca" pod "668d389b-833e-42dc-90e7-951366e7b5cf" (UID: "668d389b-833e-42dc-90e7-951366e7b5cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:03.838726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:03.838652    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668d389b-833e-42dc-90e7-951366e7b5cf-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:03.840208 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:03.840180    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668d389b-833e-42dc-90e7-951366e7b5cf-maas-api-xks-ca" pod "668d389b-833e-42dc-90e7-951366e7b5cf" (UID: "668d389b-833e-42dc-90e7-951366e7b5cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:53:03.840353 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:03.840310    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668d389b-833e-42dc-90e7-951366e7b5cf-kube-api-access-mxzts" pod "668d389b-833e-42dc-90e7-951366e7b5cf" (UID: "668d389b-833e-42dc-90e7-951366e7b5cf"). InnerVolumeSpecName "kube-api-access-mxzts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:03.940211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:03.940172    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxzts\" (UniqueName: \"kubernetes.io/projected/668d389b-833e-42dc-90e7-951366e7b5cf-kube-api-access-mxzts\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:03.940211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:03.940204    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668d389b-833e-42dc-90e7-951366e7b5cf-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:04.592826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:04.592796    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"
Oct 30 01:53:04.593313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:04.592816    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr" event={"ID":"668d389b-833e-42dc-90e7-951366e7b5cf","Type":"ContainerDied","Data":"5fcbfd70b4d1baf03b258096d4778656464994ef918400985434d793cd502c2a"}
Oct 30 01:53:04.593313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:04.592874    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fcbfd70b4d1baf03b258096d4778656464994ef918400985434d793cd502c2a"
Oct 30 01:53:04.773446 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:04.773417    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"
Oct 30 01:53:04.852509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:04.852428    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd9467ca-1998-4000-87e2-e64b3bb98c3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd9467ca-1998-4000-87e2-e64b3bb98c3c-maas-api-openshift-ca\") pod \"bd9467ca-1998-4000-87e2-e64b3bb98c3c\" (UID: \"bd9467ca-1998-4000-87e2-e64b3bb98c3c\") "
Oct 30 01:53:04.852509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:04.852468    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd9467ca-1998-4000-87e2-e64b3bb98c3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd9467ca-1998-4000-87e2-e64b3bb98c3c-maas-api-xks-ca\") pod \"bd9467ca-1998-4000-87e2-e64b3bb98c3c\" (UID: \"bd9467ca-1998-4000-87e2-e64b3bb98c3c\") "
Oct 30 01:53:04.852747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:04.852510    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9467ca-1998-4000-87e2-e64b3bb98c3c-kube-api-access-kcgdq\" (UniqueName: \"kubernetes.io/projected/bd9467ca-1998-4000-87e2-e64b3bb98c3c-kube-api-access-kcgdq\") pod \"bd9467ca-1998-4000-87e2-e64b3bb98c3c\" (UID: \"bd9467ca-1998-4000-87e2-e64b3bb98c3c\") "
Oct 30 01:53:04.852892 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:04.852868    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd9467ca-1998-4000-87e2-e64b3bb98c3c-maas-api-openshift-ca" pod "bd9467ca-1998-4000-87e2-e64b3bb98c3c" (UID: "bd9467ca-1998-4000-87e2-e64b3bb98c3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:04.853456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:04.853232    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd9467ca-1998-4000-87e2-e64b3bb98c3c-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:04.854847 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:04.854824    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9467ca-1998-4000-87e2-e64b3bb98c3c-kube-api-access-kcgdq" pod "bd9467ca-1998-4000-87e2-e64b3bb98c3c" (UID: "bd9467ca-1998-4000-87e2-e64b3bb98c3c"). InnerVolumeSpecName "kube-api-access-kcgdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:04.855054 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:04.855038    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd9467ca-1998-4000-87e2-e64b3bb98c3c-maas-api-xks-ca" pod "bd9467ca-1998-4000-87e2-e64b3bb98c3c" (UID: "bd9467ca-1998-4000-87e2-e64b3bb98c3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:53:04.954963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:04.954929    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd9467ca-1998-4000-87e2-e64b3bb98c3c-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:04.954963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:04.954959    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcgdq\" (UniqueName: \"kubernetes.io/projected/bd9467ca-1998-4000-87e2-e64b3bb98c3c-kube-api-access-kcgdq\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:05.078186 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:05.078135    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"]
Oct 30 01:53:05.088181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:05.088132    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-df4db1-c7p7h"]
Oct 30 01:53:05.428478 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:05.428437    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ec74f1-5a05-48eb-967d-c478b39ec884" path="/var/lib/kubelet/pods/c1ec74f1-5a05-48eb-967d-c478b39ec884/volumes"
Oct 30 01:53:05.601962 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:05.601933    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"
Oct 30 01:53:05.602397 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:05.602013    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9" event={"ID":"bd9467ca-1998-4000-87e2-e64b3bb98c3c","Type":"ContainerDied","Data":"7d861108ddd194d8dd39d02194fe44930ca0c97d2caa2e2bb310b837501cf447"}
Oct 30 01:53:05.602397 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:05.602069    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d861108ddd194d8dd39d02194fe44930ca0c97d2caa2e2bb310b837501cf447"
Oct 30 01:53:07.554269 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:53:07.554236    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:53:11.211475 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:11.211440    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"]
Oct 30 01:53:11.246203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:11.246167    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"]
Oct 30 01:53:12.011225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:12.011185    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"]
Oct 30 01:53:12.043313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:12.043283    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"]
Oct 30 01:53:14.189691 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.189649    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"]
Oct 30 01:53:14.190547 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.190525    4425 cpu_manager.go:414] "containerMap: removing container" podUID="668d389b-833e-42dc-90e7-951366e7b5cf" containerName="revoke-keys"
Oct 30 01:53:14.190547 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.190547    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668d389b-833e-42dc-90e7-951366e7b5cf" containerName="revoke-keys"
Oct 30 01:53:14.190723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.190614    4425 cpu_manager.go:414] "containerMap: removing container" podUID="bd9467ca-1998-4000-87e2-e64b3bb98c3c" containerName="revoke-keys"
Oct 30 01:53:14.190723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.190622    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9467ca-1998-4000-87e2-e64b3bb98c3c" containerName="revoke-keys"
Oct 30 01:53:14.190828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.190779    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9467ca-1998-4000-87e2-e64b3bb98c3c" containerName="revoke-keys"
Oct 30 01:53:14.190828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.190799    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="668d389b-833e-42dc-90e7-951366e7b5cf" containerName="revoke-keys"
Oct 30 01:53:14.194465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.194443    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"
Oct 30 01:53:14.210841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.210819    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:53:14.210931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.210844    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:53:14.222902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.222878    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"]
Oct 30 01:53:14.277291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.277248    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6\" (UID: \"582a1d8e-713d-4bd6-a44e-2d7f2714cb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"
Oct 30 01:53:14.277489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.277325    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6\" (UID: \"582a1d8e-713d-4bd6-a44e-2d7f2714cb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"
Oct 30 01:53:14.277578 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.277503    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6p4s\" (UniqueName: \"kubernetes.io/projected/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-kube-api-access-s6p4s\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6\" (UID: \"582a1d8e-713d-4bd6-a44e-2d7f2714cb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"
Oct 30 01:53:14.378949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.378905    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6\" (UID: \"582a1d8e-713d-4bd6-a44e-2d7f2714cb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"
Oct 30 01:53:14.379192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.379003    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6\" (UID: \"582a1d8e-713d-4bd6-a44e-2d7f2714cb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"
Oct 30 01:53:14.379192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.379078    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6p4s\" (UniqueName: \"kubernetes.io/projected/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-kube-api-access-s6p4s\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6\" (UID: \"582a1d8e-713d-4bd6-a44e-2d7f2714cb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"
Oct 30 01:53:14.379737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.379707    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6\" (UID: \"582a1d8e-713d-4bd6-a44e-2d7f2714cb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"
Oct 30 01:53:14.381546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.381521    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6\" (UID: \"582a1d8e-713d-4bd6-a44e-2d7f2714cb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"
Oct 30 01:53:14.400317 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.400289    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6p4s\" (UniqueName: \"kubernetes.io/projected/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-kube-api-access-s6p4s\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6\" (UID: \"582a1d8e-713d-4bd6-a44e-2d7f2714cb05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"
Oct 30 01:53:14.508522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.508447    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"
Oct 30 01:53:14.819977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:14.819948    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"]
Oct 30 01:53:14.820692 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:53:14.820669    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582a1d8e_713d_4bd6_a44e_2d7f2714cb05.slice/crio-908b9dbab9f1288afc9f199df5ee7f58e23d2b25ca141071c30b52dd72577786 WatchSource:0}: Error finding container 908b9dbab9f1288afc9f199df5ee7f58e23d2b25ca141071c30b52dd72577786: Status 404 returned error can't find the container with id 908b9dbab9f1288afc9f199df5ee7f58e23d2b25ca141071c30b52dd72577786
Oct 30 01:53:15.696509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:15.696472    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6" event={"ID":"582a1d8e-713d-4bd6-a44e-2d7f2714cb05","Type":"ContainerStarted","Data":"1ffac334a160b035e22cf36e1ad62ee1eb49f23ae582b7d7ed94841809eecb01"}
Oct 30 01:53:15.696918 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:15.696538    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6" event={"ID":"582a1d8e-713d-4bd6-a44e-2d7f2714cb05","Type":"ContainerStarted","Data":"908b9dbab9f1288afc9f199df5ee7f58e23d2b25ca141071c30b52dd72577786"}
Oct 30 01:53:15.732677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:15.732631    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6" podStartSLOduration=1.732618336 podStartE2EDuration="1.732618336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:15.731645237 +0000 UTC m=+4304.481300544" watchObservedRunningTime="2026-10-30 01:53:15.732618336 +0000 UTC m=+4304.482273644"
Oct 30 01:53:16.212254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.212202    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp" podUID="ef141a5d-f63e-44da-a7a4-4f8b2e2c874a" containerName="payload-processing" containerID="cri-o://ce116253e383798668390c16effdedc19d62fe7780f655d608976ed92f38a1e8" gracePeriod=25
Oct 30 01:53:16.246847 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.246813    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz" podUID="a4625d33-6aa0-47dc-a9dc-cad1bad7cf94" containerName="payload-pre-processing" containerID="cri-o://c7453a7af4dd021222d46d0260e0fd3dac834c650bb924fc0739f03c3d43fa46" gracePeriod=25
Oct 30 01:53:16.276843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.276811    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9" podUID="f4e5e67f-7ccd-4744-bf03-dfae00c0f982" containerName="main" containerID="cri-o://b394eb5f14c68327f4094cf48f0121fde97af75ce83c6c50780ac3ef8450b61e" gracePeriod=45
Oct 30 01:53:16.558319 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.558294    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:53:16.705309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.705276    4425 generic.go:356] "Generic (PLEG): container finished" podID="582a1d8e-713d-4bd6-a44e-2d7f2714cb05" containerID="1ffac334a160b035e22cf36e1ad62ee1eb49f23ae582b7d7ed94841809eecb01" exitCode=0
Oct 30 01:53:16.705711 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.705363    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6" event={"ID":"582a1d8e-713d-4bd6-a44e-2d7f2714cb05","Type":"ContainerDied","Data":"1ffac334a160b035e22cf36e1ad62ee1eb49f23ae582b7d7ed94841809eecb01"}
Oct 30 01:53:16.706940 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.706915    4425 generic.go:356] "Generic (PLEG): container finished" podID="f4e5e67f-7ccd-4744-bf03-dfae00c0f982" containerID="b394eb5f14c68327f4094cf48f0121fde97af75ce83c6c50780ac3ef8450b61e" exitCode=0
Oct 30 01:53:16.707073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.706959    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"
Oct 30 01:53:16.707073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.706990    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9" event={"ID":"f4e5e67f-7ccd-4744-bf03-dfae00c0f982","Type":"ContainerDied","Data":"b394eb5f14c68327f4094cf48f0121fde97af75ce83c6c50780ac3ef8450b61e"}
Oct 30 01:53:16.707073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.707037    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9" event={"ID":"f4e5e67f-7ccd-4744-bf03-dfae00c0f982","Type":"ContainerDied","Data":"e006a666fde4f8c8040fd298df9d7fc2341ec5bc82f588fd35e5eb5848b341da"}
Oct 30 01:53:16.707073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.707062    4425 scope.go:122] "RemoveContainer" containerID="b394eb5f14c68327f4094cf48f0121fde97af75ce83c6c50780ac3ef8450b61e"
Oct 30 01:53:16.707298 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.707191    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-kube-api-access-dj9v8\" (UniqueName: \"kubernetes.io/projected/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-kube-api-access-dj9v8\") pod \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") "
Oct 30 01:53:16.707358 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.707340    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-model-cache\") pod \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") "
Oct 30 01:53:16.707423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.707394    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-dshm\") pod \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") "
Oct 30 01:53:16.707482 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.707442    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-tls-certs\") pod \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") "
Oct 30 01:53:16.707538 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.707489    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-home\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-home\") pod \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") "
Oct 30 01:53:16.707629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.707614    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-tmp-dir\") pod \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\" (UID: \"f4e5e67f-7ccd-4744-bf03-dfae00c0f982\") "
Oct 30 01:53:16.708531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.708306    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-tmp-dir" pod "f4e5e67f-7ccd-4744-bf03-dfae00c0f982" (UID: "f4e5e67f-7ccd-4744-bf03-dfae00c0f982"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:16.708531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.708343    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-home" pod "f4e5e67f-7ccd-4744-bf03-dfae00c0f982" (UID: "f4e5e67f-7ccd-4744-bf03-dfae00c0f982"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:16.708531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.708378    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-model-cache" pod "f4e5e67f-7ccd-4744-bf03-dfae00c0f982" (UID: "f4e5e67f-7ccd-4744-bf03-dfae00c0f982"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:16.709729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.709707    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-dshm" pod "f4e5e67f-7ccd-4744-bf03-dfae00c0f982" (UID: "f4e5e67f-7ccd-4744-bf03-dfae00c0f982"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:16.709877 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.709861    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-kube-api-access-dj9v8" pod "f4e5e67f-7ccd-4744-bf03-dfae00c0f982" (UID: "f4e5e67f-7ccd-4744-bf03-dfae00c0f982"). InnerVolumeSpecName "kube-api-access-dj9v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:16.710104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.710083    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-tls-certs" pod "f4e5e67f-7ccd-4744-bf03-dfae00c0f982" (UID: "f4e5e67f-7ccd-4744-bf03-dfae00c0f982"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:53:16.737584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.737545    4425 scope.go:122] "RemoveContainer" containerID="b394eb5f14c68327f4094cf48f0121fde97af75ce83c6c50780ac3ef8450b61e"
Oct 30 01:53:16.737861 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:53:16.737840    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b394eb5f14c68327f4094cf48f0121fde97af75ce83c6c50780ac3ef8450b61e\": container with ID starting with b394eb5f14c68327f4094cf48f0121fde97af75ce83c6c50780ac3ef8450b61e not found: ID does not exist" containerID="b394eb5f14c68327f4094cf48f0121fde97af75ce83c6c50780ac3ef8450b61e"
Oct 30 01:53:16.737973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.737864    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b394eb5f14c68327f4094cf48f0121fde97af75ce83c6c50780ac3ef8450b61e"} err="failed to get container status \"b394eb5f14c68327f4094cf48f0121fde97af75ce83c6c50780ac3ef8450b61e\": rpc error: code = NotFound desc = could not find container \"b394eb5f14c68327f4094cf48f0121fde97af75ce83c6c50780ac3ef8450b61e\": container with ID starting with b394eb5f14c68327f4094cf48f0121fde97af75ce83c6c50780ac3ef8450b61e not found: ID does not exist"
Oct 30 01:53:16.809044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.808978    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:16.809044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.809006    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj9v8\" (UniqueName: \"kubernetes.io/projected/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-kube-api-access-dj9v8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:16.809044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.809016    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:16.809044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.809024    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:16.809044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.809032    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:16.809044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:16.809038    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4e5e67f-7ccd-4744-bf03-dfae00c0f982-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:17.012248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:17.012129    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2" podUID="1a04bb9d-f7a6-4333-af09-e078a2cab36e" containerName="payload-processing" containerID="cri-o://f114039bdcddfe298519874b4810d2cd8165459981831655ad939987eb27dafa" gracePeriod=25
Oct 30 01:53:17.044488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:17.044459    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg" podUID="0e087042-3c62-41a2-8f9e-3134d594906b" containerName="payload-pre-processing" containerID="cri-o://497a2e97b3f5ac0e17f75c6766891b5bbff429d66e906fc29b8594fed7765047" gracePeriod=25
Oct 30 01:53:17.061843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:17.061792    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"]
Oct 30 01:53:17.071214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:17.071192    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c666c2/sub-test-model-c666c2-kserve-fbc6c5489-4t9z9"]
Oct 30 01:53:17.423585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:17.423517    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4e5e67f-7ccd-4744-bf03-dfae00c0f982" path="/var/lib/kubelet/pods/f4e5e67f-7ccd-4744-bf03-dfae00c0f982/volumes"
Oct 30 01:53:17.853183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:17.832449    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"]
Oct 30 01:53:17.853183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:17.833438    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz" podUID="3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce" containerName="maas-api" containerID="cri-o://5c5ac4cef5bcf288c556bd7fdb594f4c70dad7d4e99d8aa98ac23d556264afc9" gracePeriod=30
Oct 30 01:53:17.965778 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:17.965751    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"
Oct 30 01:53:18.042370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.042338    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-maas-api-openshift-ca\") pod \"582a1d8e-713d-4bd6-a44e-2d7f2714cb05\" (UID: \"582a1d8e-713d-4bd6-a44e-2d7f2714cb05\") "
Oct 30 01:53:18.042534 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.042390    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-maas-api-xks-ca\") pod \"582a1d8e-713d-4bd6-a44e-2d7f2714cb05\" (UID: \"582a1d8e-713d-4bd6-a44e-2d7f2714cb05\") "
Oct 30 01:53:18.042534 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.042459    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-kube-api-access-s6p4s\" (UniqueName: \"kubernetes.io/projected/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-kube-api-access-s6p4s\") pod \"582a1d8e-713d-4bd6-a44e-2d7f2714cb05\" (UID: \"582a1d8e-713d-4bd6-a44e-2d7f2714cb05\") "
Oct 30 01:53:18.043081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.043055    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-maas-api-openshift-ca" pod "582a1d8e-713d-4bd6-a44e-2d7f2714cb05" (UID: "582a1d8e-713d-4bd6-a44e-2d7f2714cb05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:18.060643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.058996    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-kube-api-access-s6p4s" pod "582a1d8e-713d-4bd6-a44e-2d7f2714cb05" (UID: "582a1d8e-713d-4bd6-a44e-2d7f2714cb05"). InnerVolumeSpecName "kube-api-access-s6p4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:18.061833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.061809    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-maas-api-xks-ca" pod "582a1d8e-713d-4bd6-a44e-2d7f2714cb05" (UID: "582a1d8e-713d-4bd6-a44e-2d7f2714cb05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:53:18.144015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.143975    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:18.144015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.144016    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:18.144282 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.144029    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6p4s\" (UniqueName: \"kubernetes.io/projected/582a1d8e-713d-4bd6-a44e-2d7f2714cb05-kube-api-access-s6p4s\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:18.174949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.174919    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:53:18.244843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.244595    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-maas-api-tls\") pod \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\" (UID: \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\") "
Oct 30 01:53:18.244843 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.244714    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-openshift-service-ca\") pod \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\" (UID: \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\") "
Oct 30 01:53:18.245069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.244853    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-kube-api-access-25bj7\" (UniqueName: \"kubernetes.io/projected/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-kube-api-access-25bj7\") pod \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\" (UID: \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\") "
Oct 30 01:53:18.245069 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.244920    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-metrics-certs\") pod \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\" (UID: \"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce\") "
Oct 30 01:53:18.245500 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.245471    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-openshift-service-ca" pod "3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce" (UID: "3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:18.247854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.247820    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-metrics-certs" pod "3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce" (UID: "3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:53:18.248237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.248209    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-kube-api-access-25bj7" pod "3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce" (UID: "3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce"). InnerVolumeSpecName "kube-api-access-25bj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:18.249065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.249040    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-maas-api-tls" pod "3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce" (UID: "3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:53:18.346615 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.346576    4425 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-openshift-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:18.346615 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.346611    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25bj7\" (UniqueName: \"kubernetes.io/projected/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-kube-api-access-25bj7\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:18.346615 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.346621    4425 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-metrics-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:18.346933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.346630    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce-maas-api-tls\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:18.575269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.575233    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"]
Oct 30 01:53:18.579551 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.579524    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1cb829-f7f2f"]
Oct 30 01:53:18.727560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.727525    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"
Oct 30 01:53:18.727560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.727543    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6" event={"ID":"582a1d8e-713d-4bd6-a44e-2d7f2714cb05","Type":"ContainerDied","Data":"908b9dbab9f1288afc9f199df5ee7f58e23d2b25ca141071c30b52dd72577786"}
Oct 30 01:53:18.727853 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.727605    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="908b9dbab9f1288afc9f199df5ee7f58e23d2b25ca141071c30b52dd72577786"
Oct 30 01:53:18.729136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.729110    4425 generic.go:356] "Generic (PLEG): container finished" podID="3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce" containerID="5c5ac4cef5bcf288c556bd7fdb594f4c70dad7d4e99d8aa98ac23d556264afc9" exitCode=0
Oct 30 01:53:18.729312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.729136    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz" event={"ID":"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce","Type":"ContainerDied","Data":"5c5ac4cef5bcf288c556bd7fdb594f4c70dad7d4e99d8aa98ac23d556264afc9"}
Oct 30 01:53:18.729312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.729190    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"
Oct 30 01:53:18.729312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.729217    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz" event={"ID":"3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce","Type":"ContainerDied","Data":"e7d904676f50adad3b7378855e8ad89dd7aa68a226f304684f20561be8dfd6ba"}
Oct 30 01:53:18.729312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.729242    4425 scope.go:122] "RemoveContainer" containerID="5c5ac4cef5bcf288c556bd7fdb594f4c70dad7d4e99d8aa98ac23d556264afc9"
Oct 30 01:53:18.741124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.741108    4425 scope.go:122] "RemoveContainer" containerID="5c5ac4cef5bcf288c556bd7fdb594f4c70dad7d4e99d8aa98ac23d556264afc9"
Oct 30 01:53:18.741412 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:53:18.741393    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c5ac4cef5bcf288c556bd7fdb594f4c70dad7d4e99d8aa98ac23d556264afc9\": container with ID starting with 5c5ac4cef5bcf288c556bd7fdb594f4c70dad7d4e99d8aa98ac23d556264afc9 not found: ID does not exist" containerID="5c5ac4cef5bcf288c556bd7fdb594f4c70dad7d4e99d8aa98ac23d556264afc9"
Oct 30 01:53:18.741474 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.741420    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5ac4cef5bcf288c556bd7fdb594f4c70dad7d4e99d8aa98ac23d556264afc9"} err="failed to get container status \"5c5ac4cef5bcf288c556bd7fdb594f4c70dad7d4e99d8aa98ac23d556264afc9\": rpc error: code = NotFound desc = could not find container \"5c5ac4cef5bcf288c556bd7fdb594f4c70dad7d4e99d8aa98ac23d556264afc9\": container with ID starting with 5c5ac4cef5bcf288c556bd7fdb594f4c70dad7d4e99d8aa98ac23d556264afc9 not found: ID does not exist"
Oct 30 01:53:18.775683 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.775658    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"]
Oct 30 01:53:18.784502 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:18.784481    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1cb829-5d4c98794c-kcvrz"]
Oct 30 01:53:19.423841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:19.423811    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06fcef97-63bc-454d-a993-5dc46557cf81" path="/var/lib/kubelet/pods/06fcef97-63bc-454d-a993-5dc46557cf81/volumes"
Oct 30 01:53:19.424308 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:19.424293    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce" path="/var/lib/kubelet/pods/3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce/volumes"
Oct 30 01:53:19.565278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:19.565239    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2" podUID="1a04bb9d-f7a6-4333-af09-e078a2cab36e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:53:19.565804 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:19.565781    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg" podUID="0e087042-3c62-41a2-8f9e-3134d594906b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:53:21.027887 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:21.027852    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp" podUID="ef141a5d-f63e-44da-a7a4-4f8b2e2c874a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:53:21.027887 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:21.027874    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz" podUID="a4625d33-6aa0-47dc-a9dc-cad1bad7cf94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:53:24.633619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:24.633582    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"]
Oct 30 01:53:24.680022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:24.679989    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"]
Oct 30 01:53:27.484729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.484692    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"]
Oct 30 01:53:27.485354 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.485338    4425 cpu_manager.go:414] "containerMap: removing container" podUID="582a1d8e-713d-4bd6-a44e-2d7f2714cb05" containerName="revoke-keys"
Oct 30 01:53:27.485418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.485355    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="582a1d8e-713d-4bd6-a44e-2d7f2714cb05" containerName="revoke-keys"
Oct 30 01:53:27.485418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.485370    4425 cpu_manager.go:414] "containerMap: removing container" podUID="f4e5e67f-7ccd-4744-bf03-dfae00c0f982" containerName="main"
Oct 30 01:53:27.485418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.485375    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4e5e67f-7ccd-4744-bf03-dfae00c0f982" containerName="main"
Oct 30 01:53:27.485418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.485406    4425 cpu_manager.go:414] "containerMap: removing container" podUID="3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce" containerName="maas-api"
Oct 30 01:53:27.485418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.485412    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce" containerName="maas-api"
Oct 30 01:53:27.485579 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.485506    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dcc68ce-dd94-4b15-ac6a-fa179b89a3ce" containerName="maas-api"
Oct 30 01:53:27.485579 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.485517    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="582a1d8e-713d-4bd6-a44e-2d7f2714cb05" containerName="revoke-keys"
Oct 30 01:53:27.485579 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.485523    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4e5e67f-7ccd-4744-bf03-dfae00c0f982" containerName="main"
Oct 30 01:53:27.490527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.490511    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"
Oct 30 01:53:27.508090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.508066    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"]
Oct 30 01:53:27.513889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.513870    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:53:27.513889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.513890    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:53:27.567584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.567550    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d19fbfba-256d-4484-9bc7-a865a77d08ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz\" (UID: \"d19fbfba-256d-4484-9bc7-a865a77d08ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"
Oct 30 01:53:27.567724 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.567667    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d19fbfba-256d-4484-9bc7-a865a77d08ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz\" (UID: \"d19fbfba-256d-4484-9bc7-a865a77d08ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"
Oct 30 01:53:27.567802 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.567789    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xdz9\" (UniqueName: \"kubernetes.io/projected/d19fbfba-256d-4484-9bc7-a865a77d08ed-kube-api-access-2xdz9\") pod \"maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz\" (UID: \"d19fbfba-256d-4484-9bc7-a865a77d08ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"
Oct 30 01:53:27.669212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.669182    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d19fbfba-256d-4484-9bc7-a865a77d08ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz\" (UID: \"d19fbfba-256d-4484-9bc7-a865a77d08ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"
Oct 30 01:53:27.669356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.669272    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d19fbfba-256d-4484-9bc7-a865a77d08ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz\" (UID: \"d19fbfba-256d-4484-9bc7-a865a77d08ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"
Oct 30 01:53:27.669415 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.669364    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xdz9\" (UniqueName: \"kubernetes.io/projected/d19fbfba-256d-4484-9bc7-a865a77d08ed-kube-api-access-2xdz9\") pod \"maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz\" (UID: \"d19fbfba-256d-4484-9bc7-a865a77d08ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"
Oct 30 01:53:27.669954 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.669925    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d19fbfba-256d-4484-9bc7-a865a77d08ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz\" (UID: \"d19fbfba-256d-4484-9bc7-a865a77d08ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"
Oct 30 01:53:27.671531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.671508    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d19fbfba-256d-4484-9bc7-a865a77d08ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz\" (UID: \"d19fbfba-256d-4484-9bc7-a865a77d08ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"
Oct 30 01:53:27.689779 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.689757    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xdz9\" (UniqueName: \"kubernetes.io/projected/d19fbfba-256d-4484-9bc7-a865a77d08ed-kube-api-access-2xdz9\") pod \"maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz\" (UID: \"d19fbfba-256d-4484-9bc7-a865a77d08ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"
Oct 30 01:53:27.804131 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:27.804105    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"
Oct 30 01:53:28.106201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:28.106167    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"]
Oct 30 01:53:28.106542 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:53:28.106517    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19fbfba_256d_4484_9bc7_a865a77d08ed.slice/crio-241c92680bec838e5c28c69445544227e095f09fac269d9d07b86a675b0f52d1 WatchSource:0}: Error finding container 241c92680bec838e5c28c69445544227e095f09fac269d9d07b86a675b0f52d1: Status 404 returned error can't find the container with id 241c92680bec838e5c28c69445544227e095f09fac269d9d07b86a675b0f52d1
Oct 30 01:53:28.805435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:28.805395    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz" event={"ID":"d19fbfba-256d-4484-9bc7-a865a77d08ed","Type":"ContainerStarted","Data":"cfd59c7c23760d4255e0492fdb58593cad48b0d20c763eb6d4675d82fa0cdf9d"}
Oct 30 01:53:28.805825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:28.805485    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz" event={"ID":"d19fbfba-256d-4484-9bc7-a865a77d08ed","Type":"ContainerStarted","Data":"241c92680bec838e5c28c69445544227e095f09fac269d9d07b86a675b0f52d1"}
Oct 30 01:53:28.845669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:28.845617    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz" podStartSLOduration=1.8456003939999999 podStartE2EDuration="1.845600394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:28.844536773 +0000 UTC m=+4317.594192080" watchObservedRunningTime="2026-10-30 01:53:28.845600394 +0000 UTC m=+4317.595255702"
Oct 30 01:53:28.853352 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:28.853327    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:53:29.565787 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:29.565749    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2" podUID="1a04bb9d-f7a6-4333-af09-e078a2cab36e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:53:29.566031 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:29.566002    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg" podUID="0e087042-3c62-41a2-8f9e-3134d594906b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:53:29.634185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:29.634129    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp" podUID="6003b30d-107b-4075-aa52-8b00fd6f3d2c" containerName="payload-processing" containerID="cri-o://a4d6078b3c08d80f0225529396916cfb9b5104f10a9f7f4ffa86231c00237221" gracePeriod=25
Oct 30 01:53:29.681011 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:29.680967    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm" podUID="60123056-76c2-4e39-a284-d0a9ae34e1e8" containerName="payload-pre-processing" containerID="cri-o://217d289d19ec55b868645ca6a88f16dfb6043a62b23a650abead30a3420fb881" gracePeriod=25
Oct 30 01:53:29.814240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:29.814205    4425 generic.go:356] "Generic (PLEG): container finished" podID="d19fbfba-256d-4484-9bc7-a865a77d08ed" containerID="cfd59c7c23760d4255e0492fdb58593cad48b0d20c763eb6d4675d82fa0cdf9d" exitCode=0
Oct 30 01:53:29.814630 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:29.814290    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz" event={"ID":"d19fbfba-256d-4484-9bc7-a865a77d08ed","Type":"ContainerDied","Data":"cfd59c7c23760d4255e0492fdb58593cad48b0d20c763eb6d4675d82fa0cdf9d"}
Oct 30 01:53:30.966377 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:30.966349    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"
Oct 30 01:53:31.028504 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:31.028467    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz" podUID="a4625d33-6aa0-47dc-a9dc-cad1bad7cf94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:53:31.028648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:31.028529    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp" podUID="ef141a5d-f63e-44da-a7a4-4f8b2e2c874a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:53:31.113249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:31.113216    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d19fbfba-256d-4484-9bc7-a865a77d08ed-kube-api-access-2xdz9\" (UniqueName: \"kubernetes.io/projected/d19fbfba-256d-4484-9bc7-a865a77d08ed-kube-api-access-2xdz9\") pod \"d19fbfba-256d-4484-9bc7-a865a77d08ed\" (UID: \"d19fbfba-256d-4484-9bc7-a865a77d08ed\") "
Oct 30 01:53:31.113394 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:31.113307    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d19fbfba-256d-4484-9bc7-a865a77d08ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d19fbfba-256d-4484-9bc7-a865a77d08ed-maas-api-xks-ca\") pod \"d19fbfba-256d-4484-9bc7-a865a77d08ed\" (UID: \"d19fbfba-256d-4484-9bc7-a865a77d08ed\") "
Oct 30 01:53:31.113513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:31.113496    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d19fbfba-256d-4484-9bc7-a865a77d08ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d19fbfba-256d-4484-9bc7-a865a77d08ed-maas-api-openshift-ca\") pod \"d19fbfba-256d-4484-9bc7-a865a77d08ed\" (UID: \"d19fbfba-256d-4484-9bc7-a865a77d08ed\") "
Oct 30 01:53:31.113915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:31.113884    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d19fbfba-256d-4484-9bc7-a865a77d08ed-maas-api-openshift-ca" pod "d19fbfba-256d-4484-9bc7-a865a77d08ed" (UID: "d19fbfba-256d-4484-9bc7-a865a77d08ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:31.114058 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:31.113997    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d19fbfba-256d-4484-9bc7-a865a77d08ed-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:31.115431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:31.115406    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19fbfba-256d-4484-9bc7-a865a77d08ed-kube-api-access-2xdz9" pod "d19fbfba-256d-4484-9bc7-a865a77d08ed" (UID: "d19fbfba-256d-4484-9bc7-a865a77d08ed"). InnerVolumeSpecName "kube-api-access-2xdz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:31.115510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:31.115417    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d19fbfba-256d-4484-9bc7-a865a77d08ed-maas-api-xks-ca" pod "d19fbfba-256d-4484-9bc7-a865a77d08ed" (UID: "d19fbfba-256d-4484-9bc7-a865a77d08ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:53:31.215453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:31.215367    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xdz9\" (UniqueName: \"kubernetes.io/projected/d19fbfba-256d-4484-9bc7-a865a77d08ed-kube-api-access-2xdz9\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:31.215453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:31.215406    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d19fbfba-256d-4484-9bc7-a865a77d08ed-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:31.829635 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:31.829604    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"
Oct 30 01:53:31.829813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:31.829646    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz" event={"ID":"d19fbfba-256d-4484-9bc7-a865a77d08ed","Type":"ContainerDied","Data":"241c92680bec838e5c28c69445544227e095f09fac269d9d07b86a675b0f52d1"}
Oct 30 01:53:31.829813 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:31.829694    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="241c92680bec838e5c28c69445544227e095f09fac269d9d07b86a675b0f52d1"
Oct 30 01:53:36.517389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.517360    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"
Oct 30 01:53:36.550730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.550703    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"
Oct 30 01:53:36.683254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.683221    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4625d33-6aa0-47dc-a9dc-cad1bad7cf94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4625d33-6aa0-47dc-a9dc-cad1bad7cf94-plugins-config-volume\") pod \"a4625d33-6aa0-47dc-a9dc-cad1bad7cf94\" (UID: \"a4625d33-6aa0-47dc-a9dc-cad1bad7cf94\") "
Oct 30 01:53:36.683433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.683386    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4625d33-6aa0-47dc-a9dc-cad1bad7cf94-kube-api-access-mwn8f\" (UniqueName: \"kubernetes.io/projected/a4625d33-6aa0-47dc-a9dc-cad1bad7cf94-kube-api-access-mwn8f\") pod \"a4625d33-6aa0-47dc-a9dc-cad1bad7cf94\" (UID: \"a4625d33-6aa0-47dc-a9dc-cad1bad7cf94\") "
Oct 30 01:53:36.683483 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.683447    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef141a5d-f63e-44da-a7a4-4f8b2e2c874a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef141a5d-f63e-44da-a7a4-4f8b2e2c874a-plugins-config-volume\") pod \"ef141a5d-f63e-44da-a7a4-4f8b2e2c874a\" (UID: \"ef141a5d-f63e-44da-a7a4-4f8b2e2c874a\") "
Oct 30 01:53:36.683561 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.683546    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef141a5d-f63e-44da-a7a4-4f8b2e2c874a-kube-api-access-ltc4f\" (UniqueName: \"kubernetes.io/projected/ef141a5d-f63e-44da-a7a4-4f8b2e2c874a-kube-api-access-ltc4f\") pod \"ef141a5d-f63e-44da-a7a4-4f8b2e2c874a\" (UID: \"ef141a5d-f63e-44da-a7a4-4f8b2e2c874a\") "
Oct 30 01:53:36.683798 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.683651    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4625d33-6aa0-47dc-a9dc-cad1bad7cf94-plugins-config-volume" pod "a4625d33-6aa0-47dc-a9dc-cad1bad7cf94" (UID: "a4625d33-6aa0-47dc-a9dc-cad1bad7cf94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:36.683888 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.683852    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef141a5d-f63e-44da-a7a4-4f8b2e2c874a-plugins-config-volume" pod "ef141a5d-f63e-44da-a7a4-4f8b2e2c874a" (UID: "ef141a5d-f63e-44da-a7a4-4f8b2e2c874a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:36.684085 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.684068    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4625d33-6aa0-47dc-a9dc-cad1bad7cf94-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:36.684183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.684087    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef141a5d-f63e-44da-a7a4-4f8b2e2c874a-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:36.685663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.685640    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef141a5d-f63e-44da-a7a4-4f8b2e2c874a-kube-api-access-ltc4f" pod "ef141a5d-f63e-44da-a7a4-4f8b2e2c874a" (UID: "ef141a5d-f63e-44da-a7a4-4f8b2e2c874a"). InnerVolumeSpecName "kube-api-access-ltc4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:36.685772 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.685670    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4625d33-6aa0-47dc-a9dc-cad1bad7cf94-kube-api-access-mwn8f" pod "a4625d33-6aa0-47dc-a9dc-cad1bad7cf94" (UID: "a4625d33-6aa0-47dc-a9dc-cad1bad7cf94"). InnerVolumeSpecName "kube-api-access-mwn8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:36.712935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.712905    4425 scope.go:122] "RemoveContainer" containerID="fbcd456fe2f7ac850db046f8a1fb507dd249521bcd2d34a7a3c4a5cc3e50a6f2"
Oct 30 01:53:36.723569 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.723550    4425 scope.go:122] "RemoveContainer" containerID="ce116253e383798668390c16effdedc19d62fe7780f655d608976ed92f38a1e8"
Oct 30 01:53:36.732914 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.732888    4425 scope.go:122] "RemoveContainer" containerID="c7453a7af4dd021222d46d0260e0fd3dac834c650bb924fc0739f03c3d43fa46"
Oct 30 01:53:36.785219 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.785195    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltc4f\" (UniqueName: \"kubernetes.io/projected/ef141a5d-f63e-44da-a7a4-4f8b2e2c874a-kube-api-access-ltc4f\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:36.785219 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.785220    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwn8f\" (UniqueName: \"kubernetes.io/projected/a4625d33-6aa0-47dc-a9dc-cad1bad7cf94-kube-api-access-mwn8f\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:36.865889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.865851    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp" event={"ID":"ef141a5d-f63e-44da-a7a4-4f8b2e2c874a","Type":"ContainerDied","Data":"ce116253e383798668390c16effdedc19d62fe7780f655d608976ed92f38a1e8"}
Oct 30 01:53:36.865889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.865884    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"
Oct 30 01:53:36.866084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.865905    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp" event={"ID":"ef141a5d-f63e-44da-a7a4-4f8b2e2c874a","Type":"ContainerDied","Data":"368cfda66087c23d58179cf3595222cff001e6d0b45a9fd39b21b76ccce3ff57"}
Oct 30 01:53:36.866084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.865923    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz" event={"ID":"a4625d33-6aa0-47dc-a9dc-cad1bad7cf94","Type":"ContainerDied","Data":"c7453a7af4dd021222d46d0260e0fd3dac834c650bb924fc0739f03c3d43fa46"}
Oct 30 01:53:36.866084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.865941    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz" event={"ID":"a4625d33-6aa0-47dc-a9dc-cad1bad7cf94","Type":"ContainerDied","Data":"cad7d877d7bd968c0d81e0592ca6f6fccda92740302a5d5a3b06dab2caef8727"}
Oct 30 01:53:36.866084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.865886    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"
Oct 30 01:53:36.980508 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.980428    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"]
Oct 30 01:53:36.986450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:36.986422    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dd80cb-6d84795cbc-dm6nz"]
Oct 30 01:53:37.018128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.018096    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"]
Oct 30 01:53:37.024104 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.024071    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dd80cb-5dbf76f984-qk9vp"]
Oct 30 01:53:37.315689 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.315663    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"
Oct 30 01:53:37.351947 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.351917    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"
Oct 30 01:53:37.425032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.424995    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4625d33-6aa0-47dc-a9dc-cad1bad7cf94" path="/var/lib/kubelet/pods/a4625d33-6aa0-47dc-a9dc-cad1bad7cf94/volumes"
Oct 30 01:53:37.425605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.425581    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef141a5d-f63e-44da-a7a4-4f8b2e2c874a" path="/var/lib/kubelet/pods/ef141a5d-f63e-44da-a7a4-4f8b2e2c874a/volumes"
Oct 30 01:53:37.494534 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.494466    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a04bb9d-f7a6-4333-af09-e078a2cab36e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a04bb9d-f7a6-4333-af09-e078a2cab36e-plugins-config-volume\") pod \"1a04bb9d-f7a6-4333-af09-e078a2cab36e\" (UID: \"1a04bb9d-f7a6-4333-af09-e078a2cab36e\") "
Oct 30 01:53:37.494534 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.494515    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e087042-3c62-41a2-8f9e-3134d594906b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e087042-3c62-41a2-8f9e-3134d594906b-plugins-config-volume\") pod \"0e087042-3c62-41a2-8f9e-3134d594906b\" (UID: \"0e087042-3c62-41a2-8f9e-3134d594906b\") "
Oct 30 01:53:37.494713 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.494547    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a04bb9d-f7a6-4333-af09-e078a2cab36e-kube-api-access-hf5v8\" (UniqueName: \"kubernetes.io/projected/1a04bb9d-f7a6-4333-af09-e078a2cab36e-kube-api-access-hf5v8\") pod \"1a04bb9d-f7a6-4333-af09-e078a2cab36e\" (UID: \"1a04bb9d-f7a6-4333-af09-e078a2cab36e\") "
Oct 30 01:53:37.494713 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.494672    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e087042-3c62-41a2-8f9e-3134d594906b-kube-api-access-pgpz5\" (UniqueName: \"kubernetes.io/projected/0e087042-3c62-41a2-8f9e-3134d594906b-kube-api-access-pgpz5\") pod \"0e087042-3c62-41a2-8f9e-3134d594906b\" (UID: \"0e087042-3c62-41a2-8f9e-3134d594906b\") "
Oct 30 01:53:37.494862 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.494832    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a04bb9d-f7a6-4333-af09-e078a2cab36e-plugins-config-volume" pod "1a04bb9d-f7a6-4333-af09-e078a2cab36e" (UID: "1a04bb9d-f7a6-4333-af09-e078a2cab36e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:37.494963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.494937    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e087042-3c62-41a2-8f9e-3134d594906b-plugins-config-volume" pod "0e087042-3c62-41a2-8f9e-3134d594906b" (UID: "0e087042-3c62-41a2-8f9e-3134d594906b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:37.495038 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.495018    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a04bb9d-f7a6-4333-af09-e078a2cab36e-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:37.495038 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.495036    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e087042-3c62-41a2-8f9e-3134d594906b-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:37.496715 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.496695    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a04bb9d-f7a6-4333-af09-e078a2cab36e-kube-api-access-hf5v8" pod "1a04bb9d-f7a6-4333-af09-e078a2cab36e" (UID: "1a04bb9d-f7a6-4333-af09-e078a2cab36e"). InnerVolumeSpecName "kube-api-access-hf5v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:37.496797 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.496779    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e087042-3c62-41a2-8f9e-3134d594906b-kube-api-access-pgpz5" pod "0e087042-3c62-41a2-8f9e-3134d594906b" (UID: "0e087042-3c62-41a2-8f9e-3134d594906b"). InnerVolumeSpecName "kube-api-access-pgpz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:37.554499 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:53:37.554468    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:53:37.596604 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.596573    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf5v8\" (UniqueName: \"kubernetes.io/projected/1a04bb9d-f7a6-4333-af09-e078a2cab36e-kube-api-access-hf5v8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:37.596794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.596771    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgpz5\" (UniqueName: \"kubernetes.io/projected/0e087042-3c62-41a2-8f9e-3134d594906b-kube-api-access-pgpz5\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:37.677417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.677373    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp" podUID="6003b30d-107b-4075-aa52-8b00fd6f3d2c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:53:37.677417 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.677407    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm" podUID="60123056-76c2-4e39-a284-d0a9ae34e1e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:53:37.874792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.874761    4425 generic.go:356] "Generic (PLEG): container finished" podID="0e087042-3c62-41a2-8f9e-3134d594906b" containerID="497a2e97b3f5ac0e17f75c6766891b5bbff429d66e906fc29b8594fed7765047" exitCode=0
Oct 30 01:53:37.874990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.874806    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"
Oct 30 01:53:37.874990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.874842    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg" event={"ID":"0e087042-3c62-41a2-8f9e-3134d594906b","Type":"ContainerDied","Data":"497a2e97b3f5ac0e17f75c6766891b5bbff429d66e906fc29b8594fed7765047"}
Oct 30 01:53:37.874990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.874904    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg" event={"ID":"0e087042-3c62-41a2-8f9e-3134d594906b","Type":"ContainerDied","Data":"ed12ed603ea9ffa83b3b2b2af3f77e523a8d6bce8383d24c47270cfc19104345"}
Oct 30 01:53:37.874990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.874934    4425 scope.go:122] "RemoveContainer" containerID="497a2e97b3f5ac0e17f75c6766891b5bbff429d66e906fc29b8594fed7765047"
Oct 30 01:53:37.876463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.876442    4425 generic.go:356] "Generic (PLEG): container finished" podID="1a04bb9d-f7a6-4333-af09-e078a2cab36e" containerID="f114039bdcddfe298519874b4810d2cd8165459981831655ad939987eb27dafa" exitCode=0
Oct 30 01:53:37.876536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.876482    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"
Oct 30 01:53:37.876536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.876493    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2" event={"ID":"1a04bb9d-f7a6-4333-af09-e078a2cab36e","Type":"ContainerDied","Data":"f114039bdcddfe298519874b4810d2cd8165459981831655ad939987eb27dafa"}
Oct 30 01:53:37.876619 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.876532    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2" event={"ID":"1a04bb9d-f7a6-4333-af09-e078a2cab36e","Type":"ContainerDied","Data":"d0e0dc0c005e0d6d451ee5d5a78faed6490a0ccf792ead026296f853469f3dc8"}
Oct 30 01:53:37.889522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.889506    4425 scope.go:122] "RemoveContainer" containerID="497a2e97b3f5ac0e17f75c6766891b5bbff429d66e906fc29b8594fed7765047"
Oct 30 01:53:37.889875 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:53:37.889846    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"497a2e97b3f5ac0e17f75c6766891b5bbff429d66e906fc29b8594fed7765047\": container with ID starting with 497a2e97b3f5ac0e17f75c6766891b5bbff429d66e906fc29b8594fed7765047 not found: ID does not exist" containerID="497a2e97b3f5ac0e17f75c6766891b5bbff429d66e906fc29b8594fed7765047"
Oct 30 01:53:37.889983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.889886    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"497a2e97b3f5ac0e17f75c6766891b5bbff429d66e906fc29b8594fed7765047"} err="failed to get container status \"497a2e97b3f5ac0e17f75c6766891b5bbff429d66e906fc29b8594fed7765047\": rpc error: code = NotFound desc = could not find container \"497a2e97b3f5ac0e17f75c6766891b5bbff429d66e906fc29b8594fed7765047\": container with ID starting with 497a2e97b3f5ac0e17f75c6766891b5bbff429d66e906fc29b8594fed7765047 not found: ID does not exist"
Oct 30 01:53:37.889983 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.889908    4425 scope.go:122] "RemoveContainer" containerID="f114039bdcddfe298519874b4810d2cd8165459981831655ad939987eb27dafa"
Oct 30 01:53:37.899466 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.899451    4425 scope.go:122] "RemoveContainer" containerID="f114039bdcddfe298519874b4810d2cd8165459981831655ad939987eb27dafa"
Oct 30 01:53:37.899721 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:53:37.899704    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f114039bdcddfe298519874b4810d2cd8165459981831655ad939987eb27dafa\": container with ID starting with f114039bdcddfe298519874b4810d2cd8165459981831655ad939987eb27dafa not found: ID does not exist" containerID="f114039bdcddfe298519874b4810d2cd8165459981831655ad939987eb27dafa"
Oct 30 01:53:37.899762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.899729    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f114039bdcddfe298519874b4810d2cd8165459981831655ad939987eb27dafa"} err="failed to get container status \"f114039bdcddfe298519874b4810d2cd8165459981831655ad939987eb27dafa\": rpc error: code = NotFound desc = could not find container \"f114039bdcddfe298519874b4810d2cd8165459981831655ad939987eb27dafa\": container with ID starting with f114039bdcddfe298519874b4810d2cd8165459981831655ad939987eb27dafa not found: ID does not exist"
Oct 30 01:53:37.939620 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.939598    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"]
Oct 30 01:53:37.949260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.949240    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a47b9a-54b95bb464-zgrk2"]
Oct 30 01:53:37.975083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.975055    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"]
Oct 30 01:53:37.981029 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:37.981007    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a47b9a-7b564f667-xqzfg"]
Oct 30 01:53:38.014676 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:38.014648    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"]
Oct 30 01:53:38.047551 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:38.047519    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"]
Oct 30 01:53:39.084652 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:39.084612    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"]
Oct 30 01:53:39.085141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:39.084837    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n" podUID="4c009b06-2c6a-4909-81b8-f245378e9c4f" containerName="istio-proxy" containerID="cri-o://587b617e80e27d95ac15731bf1871492165d1c3b42a1bdfa6bdf7f1a0cca99eb" gracePeriod=30
Oct 30 01:53:39.157408 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:39.157354    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"]
Oct 30 01:53:39.161574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:39.161538    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-dd80cb-p6bwr"]
Oct 30 01:53:39.423693 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:39.423617    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e087042-3c62-41a2-8f9e-3134d594906b" path="/var/lib/kubelet/pods/0e087042-3c62-41a2-8f9e-3134d594906b/volumes"
Oct 30 01:53:39.424042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:39.424029    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a04bb9d-f7a6-4333-af09-e078a2cab36e" path="/var/lib/kubelet/pods/1a04bb9d-f7a6-4333-af09-e078a2cab36e/volumes"
Oct 30 01:53:39.424461 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:39.424447    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="668d389b-833e-42dc-90e7-951366e7b5cf" path="/var/lib/kubelet/pods/668d389b-833e-42dc-90e7-951366e7b5cf/volumes"
Oct 30 01:53:40.602522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:40.602486    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"]
Oct 30 01:53:40.602891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:40.602723    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88" podUID="692eafe6-0180-4aff-b472-c9d94ce846ea" containerName="maas-api" containerID="cri-o://1a4a9fc0742bec53726fbb7169564e181d028e30ee1eda1a9a9706d0dc00921d" gracePeriod=30
Oct 30 01:53:40.905465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:40.905430    4425 generic.go:356] "Generic (PLEG): container finished" podID="692eafe6-0180-4aff-b472-c9d94ce846ea" containerID="1a4a9fc0742bec53726fbb7169564e181d028e30ee1eda1a9a9706d0dc00921d" exitCode=0
Oct 30 01:53:40.905648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:40.905627    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88" event={"ID":"692eafe6-0180-4aff-b472-c9d94ce846ea","Type":"ContainerDied","Data":"1a4a9fc0742bec53726fbb7169564e181d028e30ee1eda1a9a9706d0dc00921d"}
Oct 30 01:53:40.905745 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:40.905681    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88" event={"ID":"692eafe6-0180-4aff-b472-c9d94ce846ea","Type":"ContainerDied","Data":"49244357d0f629330a1867e2236686c9abc05fa93606e90891222b34a7a7b596"}
Oct 30 01:53:40.905745 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:40.905705    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49244357d0f629330a1867e2236686c9abc05fa93606e90891222b34a7a7b596"
Oct 30 01:53:40.908391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:40.908365    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:53:40.953259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:40.953231    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/692eafe6-0180-4aff-b472-c9d94ce846ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/692eafe6-0180-4aff-b472-c9d94ce846ea-maas-api-tls\") pod \"692eafe6-0180-4aff-b472-c9d94ce846ea\" (UID: \"692eafe6-0180-4aff-b472-c9d94ce846ea\") "
Oct 30 01:53:40.953446 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:40.953374    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/692eafe6-0180-4aff-b472-c9d94ce846ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/692eafe6-0180-4aff-b472-c9d94ce846ea-openshift-service-ca\") pod \"692eafe6-0180-4aff-b472-c9d94ce846ea\" (UID: \"692eafe6-0180-4aff-b472-c9d94ce846ea\") "
Oct 30 01:53:40.953446 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:40.953416    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692eafe6-0180-4aff-b472-c9d94ce846ea-kube-api-access-b7c75\" (UniqueName: \"kubernetes.io/projected/692eafe6-0180-4aff-b472-c9d94ce846ea-kube-api-access-b7c75\") pod \"692eafe6-0180-4aff-b472-c9d94ce846ea\" (UID: \"692eafe6-0180-4aff-b472-c9d94ce846ea\") "
Oct 30 01:53:40.953567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:40.953529    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/692eafe6-0180-4aff-b472-c9d94ce846ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/692eafe6-0180-4aff-b472-c9d94ce846ea-metrics-certs\") pod \"692eafe6-0180-4aff-b472-c9d94ce846ea\" (UID: \"692eafe6-0180-4aff-b472-c9d94ce846ea\") "
Oct 30 01:53:40.954058 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:40.953985    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692eafe6-0180-4aff-b472-c9d94ce846ea-openshift-service-ca" pod "692eafe6-0180-4aff-b472-c9d94ce846ea" (UID: "692eafe6-0180-4aff-b472-c9d94ce846ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:40.954198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:40.954101    4425 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/692eafe6-0180-4aff-b472-c9d94ce846ea-openshift-service-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:40.961202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:40.958531    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692eafe6-0180-4aff-b472-c9d94ce846ea-maas-api-tls" pod "692eafe6-0180-4aff-b472-c9d94ce846ea" (UID: "692eafe6-0180-4aff-b472-c9d94ce846ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:53:40.961202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:40.958573    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692eafe6-0180-4aff-b472-c9d94ce846ea-kube-api-access-b7c75" pod "692eafe6-0180-4aff-b472-c9d94ce846ea" (UID: "692eafe6-0180-4aff-b472-c9d94ce846ea"). InnerVolumeSpecName "kube-api-access-b7c75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:40.961202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:40.958620    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692eafe6-0180-4aff-b472-c9d94ce846ea-metrics-certs" pod "692eafe6-0180-4aff-b472-c9d94ce846ea" (UID: "692eafe6-0180-4aff-b472-c9d94ce846ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:53:41.055685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:41.055653    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7c75\" (UniqueName: \"kubernetes.io/projected/692eafe6-0180-4aff-b472-c9d94ce846ea-kube-api-access-b7c75\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:41.055685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:41.055682    4425 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/692eafe6-0180-4aff-b472-c9d94ce846ea-metrics-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:41.055685 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:41.055691    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/692eafe6-0180-4aff-b472-c9d94ce846ea-maas-api-tls\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:41.340382 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:41.340351    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"]
Oct 30 01:53:41.343958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:41.343931    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c48628-n4mtz"]
Oct 30 01:53:41.424040 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:41.424003    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d19fbfba-256d-4484-9bc7-a865a77d08ed" path="/var/lib/kubelet/pods/d19fbfba-256d-4484-9bc7-a865a77d08ed/volumes"
Oct 30 01:53:41.911938 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:41.911911    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"
Oct 30 01:53:41.958020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:41.957969    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"]
Oct 30 01:53:41.967395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:41.967367    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c48628-648b654d8-k6b88"]
Oct 30 01:53:43.015322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.015277    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4" podUID="473e2e7d-4f24-449d-b508-cb9f05e78c8b" containerName="payload-processing" containerID="cri-o://87cde047e3cf906e7d56c3162af49301bc6cd0a2b1148018ba124e22bd998c2a" gracePeriod=25
Oct 30 01:53:43.048144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.048117    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj" podUID="23a09e9b-a4fa-4ef9-8d76-b75e98f68a74" containerName="payload-pre-processing" containerID="cri-o://45936c75d450850f91b4c2b146e767ba5340965f4c92fc3e213c16158c44ce12" gracePeriod=25
Oct 30 01:53:43.286180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.286144    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"
Oct 30 01:53:43.381306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.381270    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/473e2e7d-4f24-449d-b508-cb9f05e78c8b-kube-api-access-xq4nj\" (UniqueName: \"kubernetes.io/projected/473e2e7d-4f24-449d-b508-cb9f05e78c8b-kube-api-access-xq4nj\") pod \"473e2e7d-4f24-449d-b508-cb9f05e78c8b\" (UID: \"473e2e7d-4f24-449d-b508-cb9f05e78c8b\") "
Oct 30 01:53:43.381509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.381438    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/473e2e7d-4f24-449d-b508-cb9f05e78c8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473e2e7d-4f24-449d-b508-cb9f05e78c8b-plugins-config-volume\") pod \"473e2e7d-4f24-449d-b508-cb9f05e78c8b\" (UID: \"473e2e7d-4f24-449d-b508-cb9f05e78c8b\") "
Oct 30 01:53:43.381833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.381811    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473e2e7d-4f24-449d-b508-cb9f05e78c8b-plugins-config-volume" pod "473e2e7d-4f24-449d-b508-cb9f05e78c8b" (UID: "473e2e7d-4f24-449d-b508-cb9f05e78c8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:43.383270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.383247    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473e2e7d-4f24-449d-b508-cb9f05e78c8b-kube-api-access-xq4nj" pod "473e2e7d-4f24-449d-b508-cb9f05e78c8b" (UID: "473e2e7d-4f24-449d-b508-cb9f05e78c8b"). InnerVolumeSpecName "kube-api-access-xq4nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:43.423736 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.423708    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="692eafe6-0180-4aff-b472-c9d94ce846ea" path="/var/lib/kubelet/pods/692eafe6-0180-4aff-b472-c9d94ce846ea/volumes"
Oct 30 01:53:43.483315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.483286    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/473e2e7d-4f24-449d-b508-cb9f05e78c8b-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:43.483315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.483312    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq4nj\" (UniqueName: \"kubernetes.io/projected/473e2e7d-4f24-449d-b508-cb9f05e78c8b-kube-api-access-xq4nj\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:43.926014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.925980    4425 generic.go:356] "Generic (PLEG): container finished" podID="473e2e7d-4f24-449d-b508-cb9f05e78c8b" containerID="87cde047e3cf906e7d56c3162af49301bc6cd0a2b1148018ba124e22bd998c2a" exitCode=0
Oct 30 01:53:43.926228 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.926025    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"
Oct 30 01:53:43.926228 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.926071    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4" event={"ID":"473e2e7d-4f24-449d-b508-cb9f05e78c8b","Type":"ContainerDied","Data":"87cde047e3cf906e7d56c3162af49301bc6cd0a2b1148018ba124e22bd998c2a"}
Oct 30 01:53:43.926228 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.926117    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4" event={"ID":"473e2e7d-4f24-449d-b508-cb9f05e78c8b","Type":"ContainerDied","Data":"af5a0f0a361fcbf87a5e0b3139a1720c1efe0f66332095069b3916bdd5a62752"}
Oct 30 01:53:43.926228 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.926139    4425 scope.go:122] "RemoveContainer" containerID="87cde047e3cf906e7d56c3162af49301bc6cd0a2b1148018ba124e22bd998c2a"
Oct 30 01:53:43.936881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.936861    4425 scope.go:122] "RemoveContainer" containerID="87cde047e3cf906e7d56c3162af49301bc6cd0a2b1148018ba124e22bd998c2a"
Oct 30 01:53:43.937151 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:53:43.937130    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87cde047e3cf906e7d56c3162af49301bc6cd0a2b1148018ba124e22bd998c2a\": container with ID starting with 87cde047e3cf906e7d56c3162af49301bc6cd0a2b1148018ba124e22bd998c2a not found: ID does not exist" containerID="87cde047e3cf906e7d56c3162af49301bc6cd0a2b1148018ba124e22bd998c2a"
Oct 30 01:53:43.937264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.937174    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87cde047e3cf906e7d56c3162af49301bc6cd0a2b1148018ba124e22bd998c2a"} err="failed to get container status \"87cde047e3cf906e7d56c3162af49301bc6cd0a2b1148018ba124e22bd998c2a\": rpc error: code = NotFound desc = could not find container \"87cde047e3cf906e7d56c3162af49301bc6cd0a2b1148018ba124e22bd998c2a\": container with ID starting with 87cde047e3cf906e7d56c3162af49301bc6cd0a2b1148018ba124e22bd998c2a not found: ID does not exist"
Oct 30 01:53:43.976316 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.976290    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"]
Oct 30 01:53:43.987213 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:43.987190    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c48628-7b6595bb68-zdrr4"]
Oct 30 01:53:44.362852 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.362825    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:53:44.393980 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.393950    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-credential-socket\") pod \"4c009b06-2c6a-4909-81b8-f245378e9c4f\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") "
Oct 30 01:53:44.394149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.393998    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c009b06-2c6a-4909-81b8-f245378e9c4f-kube-api-access-tn6k2\" (UniqueName: \"kubernetes.io/projected/4c009b06-2c6a-4909-81b8-f245378e9c4f-kube-api-access-tn6k2\") pod \"4c009b06-2c6a-4909-81b8-f245378e9c4f\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") "
Oct 30 01:53:44.394149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.394046    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-workload-socket\") pod \"4c009b06-2c6a-4909-81b8-f245378e9c4f\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") "
Oct 30 01:53:44.394149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.394098    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-token\" (UniqueName: \"kubernetes.io/projected/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-token\") pod \"4c009b06-2c6a-4909-81b8-f245378e9c4f\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") "
Oct 30 01:53:44.394356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.394208    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-workload-certs\") pod \"4c009b06-2c6a-4909-81b8-f245378e9c4f\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") "
Oct 30 01:53:44.394356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.394264    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-credential-socket" pod "4c009b06-2c6a-4909-81b8-f245378e9c4f" (UID: "4c009b06-2c6a-4909-81b8-f245378e9c4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.394356 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.394271    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-envoy\") pod \"4c009b06-2c6a-4909-81b8-f245378e9c4f\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") "
Oct 30 01:53:44.394509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.394373    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-workload-socket" pod "4c009b06-2c6a-4909-81b8-f245378e9c4f" (UID: "4c009b06-2c6a-4909-81b8-f245378e9c4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.394509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.394421    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-data\") pod \"4c009b06-2c6a-4909-81b8-f245378e9c4f\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") "
Oct 30 01:53:44.394509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.394492    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-podinfo\") pod \"4c009b06-2c6a-4909-81b8-f245378e9c4f\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") "
Oct 30 01:53:44.394509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.394495    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-workload-certs" pod "4c009b06-2c6a-4909-81b8-f245378e9c4f" (UID: "4c009b06-2c6a-4909-81b8-f245378e9c4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.394717 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.394530    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c009b06-2c6a-4909-81b8-f245378e9c4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c009b06-2c6a-4909-81b8-f245378e9c4f-istiod-ca-cert\") pod \"4c009b06-2c6a-4909-81b8-f245378e9c4f\" (UID: \"4c009b06-2c6a-4909-81b8-f245378e9c4f\") "
Oct 30 01:53:44.394717 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.394700    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-data" pod "4c009b06-2c6a-4909-81b8-f245378e9c4f" (UID: "4c009b06-2c6a-4909-81b8-f245378e9c4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.394936 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.394914    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c009b06-2c6a-4909-81b8-f245378e9c4f-istiod-ca-cert" pod "4c009b06-2c6a-4909-81b8-f245378e9c4f" (UID: "4c009b06-2c6a-4909-81b8-f245378e9c4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:44.395174 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.395142    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.395434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.395302    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.395434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.395329    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c009b06-2c6a-4909-81b8-f245378e9c4f-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.395434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.395344    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.395434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.395361    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.396700 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.396671    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-envoy" pod "4c009b06-2c6a-4909-81b8-f245378e9c4f" (UID: "4c009b06-2c6a-4909-81b8-f245378e9c4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:53:44.396801 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.396736    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c009b06-2c6a-4909-81b8-f245378e9c4f-kube-api-access-tn6k2" pod "4c009b06-2c6a-4909-81b8-f245378e9c4f" (UID: "4c009b06-2c6a-4909-81b8-f245378e9c4f"). InnerVolumeSpecName "kube-api-access-tn6k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:44.396851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.396814    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-podinfo" pod "4c009b06-2c6a-4909-81b8-f245378e9c4f" (UID: "4c009b06-2c6a-4909-81b8-f245378e9c4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:53:44.396851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.396823    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-token" pod "4c009b06-2c6a-4909-81b8-f245378e9c4f" (UID: "4c009b06-2c6a-4909-81b8-f245378e9c4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:44.496214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.496114    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.496214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.496144    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn6k2\" (UniqueName: \"kubernetes.io/projected/4c009b06-2c6a-4909-81b8-f245378e9c4f-kube-api-access-tn6k2\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.496214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.496166    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.496214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.496179    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c009b06-2c6a-4909-81b8-f245378e9c4f-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:44.935547 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.935506    4425 generic.go:356] "Generic (PLEG): container finished" podID="4c009b06-2c6a-4909-81b8-f245378e9c4f" containerID="587b617e80e27d95ac15731bf1871492165d1c3b42a1bdfa6bdf7f1a0cca99eb" exitCode=0
Oct 30 01:53:44.935739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.935576    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"
Oct 30 01:53:44.935739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.935585    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n" event={"ID":"4c009b06-2c6a-4909-81b8-f245378e9c4f","Type":"ContainerDied","Data":"587b617e80e27d95ac15731bf1871492165d1c3b42a1bdfa6bdf7f1a0cca99eb"}
Oct 30 01:53:44.935739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.935670    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n" event={"ID":"4c009b06-2c6a-4909-81b8-f245378e9c4f","Type":"ContainerDied","Data":"eccf5613fdd008bc123f554424cfcb28fb6481c4024dab5148daeb1560e1d806"}
Oct 30 01:53:44.935739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.935702    4425 scope.go:122] "RemoveContainer" containerID="587b617e80e27d95ac15731bf1871492165d1c3b42a1bdfa6bdf7f1a0cca99eb"
Oct 30 01:53:44.948130 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.948108    4425 scope.go:122] "RemoveContainer" containerID="587b617e80e27d95ac15731bf1871492165d1c3b42a1bdfa6bdf7f1a0cca99eb"
Oct 30 01:53:44.948450 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:53:44.948429    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"587b617e80e27d95ac15731bf1871492165d1c3b42a1bdfa6bdf7f1a0cca99eb\": container with ID starting with 587b617e80e27d95ac15731bf1871492165d1c3b42a1bdfa6bdf7f1a0cca99eb not found: ID does not exist" containerID="587b617e80e27d95ac15731bf1871492165d1c3b42a1bdfa6bdf7f1a0cca99eb"
Oct 30 01:53:44.948562 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.948459    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"587b617e80e27d95ac15731bf1871492165d1c3b42a1bdfa6bdf7f1a0cca99eb"} err="failed to get container status \"587b617e80e27d95ac15731bf1871492165d1c3b42a1bdfa6bdf7f1a0cca99eb\": rpc error: code = NotFound desc = could not find container \"587b617e80e27d95ac15731bf1871492165d1c3b42a1bdfa6bdf7f1a0cca99eb\": container with ID starting with 587b617e80e27d95ac15731bf1871492165d1c3b42a1bdfa6bdf7f1a0cca99eb not found: ID does not exist"
Oct 30 01:53:44.993535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:44.993504    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"]
Oct 30 01:53:45.002568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:45.002539    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-ggj4n"]
Oct 30 01:53:45.424141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:45.424110    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="473e2e7d-4f24-449d-b508-cb9f05e78c8b" path="/var/lib/kubelet/pods/473e2e7d-4f24-449d-b508-cb9f05e78c8b/volumes"
Oct 30 01:53:45.424602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:45.424587    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c009b06-2c6a-4909-81b8-f245378e9c4f" path="/var/lib/kubelet/pods/4c009b06-2c6a-4909-81b8-f245378e9c4f/volumes"
Oct 30 01:53:47.264269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.264237    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"]
Oct 30 01:53:47.272395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.272368    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a47b9a-wxnh9"]
Oct 30 01:53:47.424190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.424141    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd9467ca-1998-4000-87e2-e64b3bb98c3c" path="/var/lib/kubelet/pods/bd9467ca-1998-4000-87e2-e64b3bb98c3c/volumes"
Oct 30 01:53:47.424509 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.424494    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"]
Oct 30 01:53:47.425048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425036    4425 cpu_manager.go:414] "containerMap: removing container" podUID="0e087042-3c62-41a2-8f9e-3134d594906b" containerName="payload-pre-processing"
Oct 30 01:53:47.425084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425050    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e087042-3c62-41a2-8f9e-3134d594906b" containerName="payload-pre-processing"
Oct 30 01:53:47.425084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425062    4425 cpu_manager.go:414] "containerMap: removing container" podUID="1a04bb9d-f7a6-4333-af09-e078a2cab36e" containerName="payload-processing"
Oct 30 01:53:47.425084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425067    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a04bb9d-f7a6-4333-af09-e078a2cab36e" containerName="payload-processing"
Oct 30 01:53:47.425084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425075    4425 cpu_manager.go:414] "containerMap: removing container" podUID="ef141a5d-f63e-44da-a7a4-4f8b2e2c874a" containerName="payload-processing"
Oct 30 01:53:47.425084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425080    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef141a5d-f63e-44da-a7a4-4f8b2e2c874a" containerName="payload-processing"
Oct 30 01:53:47.425269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425111    4425 cpu_manager.go:414] "containerMap: removing container" podUID="692eafe6-0180-4aff-b472-c9d94ce846ea" containerName="maas-api"
Oct 30 01:53:47.425269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425116    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="692eafe6-0180-4aff-b472-c9d94ce846ea" containerName="maas-api"
Oct 30 01:53:47.425269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425141    4425 cpu_manager.go:414] "containerMap: removing container" podUID="473e2e7d-4f24-449d-b508-cb9f05e78c8b" containerName="payload-processing"
Oct 30 01:53:47.425269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425146    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="473e2e7d-4f24-449d-b508-cb9f05e78c8b" containerName="payload-processing"
Oct 30 01:53:47.425269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425198    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d19fbfba-256d-4484-9bc7-a865a77d08ed" containerName="revoke-keys"
Oct 30 01:53:47.425269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425205    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d19fbfba-256d-4484-9bc7-a865a77d08ed" containerName="revoke-keys"
Oct 30 01:53:47.425269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425213    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a4625d33-6aa0-47dc-a9dc-cad1bad7cf94" containerName="payload-pre-processing"
Oct 30 01:53:47.425269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425218    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4625d33-6aa0-47dc-a9dc-cad1bad7cf94" containerName="payload-pre-processing"
Oct 30 01:53:47.425269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425226    4425 cpu_manager.go:414] "containerMap: removing container" podUID="4c009b06-2c6a-4909-81b8-f245378e9c4f" containerName="istio-proxy"
Oct 30 01:53:47.425269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425230    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c009b06-2c6a-4909-81b8-f245378e9c4f" containerName="istio-proxy"
Oct 30 01:53:47.425577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425320    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef141a5d-f63e-44da-a7a4-4f8b2e2c874a" containerName="payload-processing"
Oct 30 01:53:47.425577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425331    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="692eafe6-0180-4aff-b472-c9d94ce846ea" containerName="maas-api"
Oct 30 01:53:47.425577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425340    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="473e2e7d-4f24-449d-b508-cb9f05e78c8b" containerName="payload-processing"
Oct 30 01:53:47.425577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425350    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d19fbfba-256d-4484-9bc7-a865a77d08ed" containerName="revoke-keys"
Oct 30 01:53:47.425577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425355    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4625d33-6aa0-47dc-a9dc-cad1bad7cf94" containerName="payload-pre-processing"
Oct 30 01:53:47.425577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425361    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c009b06-2c6a-4909-81b8-f245378e9c4f" containerName="istio-proxy"
Oct 30 01:53:47.425577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425366    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e087042-3c62-41a2-8f9e-3134d594906b" containerName="payload-pre-processing"
Oct 30 01:53:47.425577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.425372    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a04bb9d-f7a6-4333-af09-e078a2cab36e" containerName="payload-processing"
Oct 30 01:53:47.430807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.430787    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"
Oct 30 01:53:47.446790 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.446764    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:53:47.446904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.446841    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:53:47.449534 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.449512    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"]
Oct 30 01:53:47.463389 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.463370    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:53:47.463710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.463693    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:53:47.533955 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.533873    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d481ab61-efd5-4ed6-a9e9-690a210d811b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg\" (UID: \"d481ab61-efd5-4ed6-a9e9-690a210d811b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"
Oct 30 01:53:47.534103 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.534027    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d481ab61-efd5-4ed6-a9e9-690a210d811b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg\" (UID: \"d481ab61-efd5-4ed6-a9e9-690a210d811b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"
Oct 30 01:53:47.534186 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.534130    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxxpt\" (UniqueName: \"kubernetes.io/projected/d481ab61-efd5-4ed6-a9e9-690a210d811b-kube-api-access-rxxpt\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg\" (UID: \"d481ab61-efd5-4ed6-a9e9-690a210d811b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"
Oct 30 01:53:47.635335 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.635306    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d481ab61-efd5-4ed6-a9e9-690a210d811b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg\" (UID: \"d481ab61-efd5-4ed6-a9e9-690a210d811b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"
Oct 30 01:53:47.635529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.635369    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxxpt\" (UniqueName: \"kubernetes.io/projected/d481ab61-efd5-4ed6-a9e9-690a210d811b-kube-api-access-rxxpt\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg\" (UID: \"d481ab61-efd5-4ed6-a9e9-690a210d811b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"
Oct 30 01:53:47.635529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.635431    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d481ab61-efd5-4ed6-a9e9-690a210d811b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg\" (UID: \"d481ab61-efd5-4ed6-a9e9-690a210d811b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"
Oct 30 01:53:47.635943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.635919    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d481ab61-efd5-4ed6-a9e9-690a210d811b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg\" (UID: \"d481ab61-efd5-4ed6-a9e9-690a210d811b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"
Oct 30 01:53:47.637701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.637676    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d481ab61-efd5-4ed6-a9e9-690a210d811b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg\" (UID: \"d481ab61-efd5-4ed6-a9e9-690a210d811b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"
Oct 30 01:53:47.658809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.658781    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxxpt\" (UniqueName: \"kubernetes.io/projected/d481ab61-efd5-4ed6-a9e9-690a210d811b-kube-api-access-rxxpt\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg\" (UID: \"d481ab61-efd5-4ed6-a9e9-690a210d811b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"
Oct 30 01:53:47.677222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.677149    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp" podUID="6003b30d-107b-4075-aa52-8b00fd6f3d2c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:53:47.677354 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.677188    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm" podUID="60123056-76c2-4e39-a284-d0a9ae34e1e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:53:47.745024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:47.744993    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"
Oct 30 01:53:48.048083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:48.048046    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"]
Oct 30 01:53:48.049763 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:53:48.049700    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd481ab61_efd5_4ed6_a9e9_690a210d811b.slice/crio-34977c7c778fd0e110bcb247938c6949c6db4358c7fc990a2b22b46e281f1362 WatchSource:0}: Error finding container 34977c7c778fd0e110bcb247938c6949c6db4358c7fc990a2b22b46e281f1362: Status 404 returned error can't find the container with id 34977c7c778fd0e110bcb247938c6949c6db4358c7fc990a2b22b46e281f1362
Oct 30 01:53:48.971079 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:48.971050    4425 generic.go:356] "Generic (PLEG): container finished" podID="d481ab61-efd5-4ed6-a9e9-690a210d811b" containerID="a635f26ced70f8f1bfc09ee8cdf6115e276e13ca82d180dde718bdc9227d087c" exitCode=0
Oct 30 01:53:48.971462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:48.971132    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg" event={"ID":"d481ab61-efd5-4ed6-a9e9-690a210d811b","Type":"ContainerDied","Data":"a635f26ced70f8f1bfc09ee8cdf6115e276e13ca82d180dde718bdc9227d087c"}
Oct 30 01:53:48.971462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:48.971204    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg" event={"ID":"d481ab61-efd5-4ed6-a9e9-690a210d811b","Type":"ContainerStarted","Data":"34977c7c778fd0e110bcb247938c6949c6db4358c7fc990a2b22b46e281f1362"}
Oct 30 01:53:49.914228 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:49.908379    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"]
Oct 30 01:53:49.914228 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:49.913947    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"
Oct 30 01:53:49.933659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:49.933622    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"]
Oct 30 01:53:49.945385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:49.945324    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"
Oct 30 01:53:49.964624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:49.964600    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6\" (UID: \"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"
Oct 30 01:53:49.964782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:49.964672    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6\" (UID: \"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"
Oct 30 01:53:49.964853 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:49.964824    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7n5v\" (UniqueName: \"kubernetes.io/projected/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-kube-api-access-g7n5v\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6\" (UID: \"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"
Oct 30 01:53:49.980400 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:49.980371    4425 generic.go:356] "Generic (PLEG): container finished" podID="6003b30d-107b-4075-aa52-8b00fd6f3d2c" containerID="a4d6078b3c08d80f0225529396916cfb9b5104f10a9f7f4ffa86231c00237221" exitCode=0
Oct 30 01:53:49.980780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:49.980410    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"
Oct 30 01:53:49.980780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:49.980452    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp" event={"ID":"6003b30d-107b-4075-aa52-8b00fd6f3d2c","Type":"ContainerDied","Data":"a4d6078b3c08d80f0225529396916cfb9b5104f10a9f7f4ffa86231c00237221"}
Oct 30 01:53:49.980780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:49.980512    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp" event={"ID":"6003b30d-107b-4075-aa52-8b00fd6f3d2c","Type":"ContainerDied","Data":"3117e7ee54fdf7e13100ce768d9876abfcc4ce0345b5ca5c128de09659c0a82a"}
Oct 30 01:53:49.980780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:49.980528    4425 scope.go:122] "RemoveContainer" containerID="a4d6078b3c08d80f0225529396916cfb9b5104f10a9f7f4ffa86231c00237221"
Oct 30 01:53:49.981390 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:49.981374    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"
Oct 30 01:53:49.981874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:49.981854    4425 generic.go:356] "Generic (PLEG): container finished" podID="60123056-76c2-4e39-a284-d0a9ae34e1e8" containerID="217d289d19ec55b868645ca6a88f16dfb6043a62b23a650abead30a3420fb881" exitCode=0
Oct 30 01:53:49.981967 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:49.981928    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm" event={"ID":"60123056-76c2-4e39-a284-d0a9ae34e1e8","Type":"ContainerDied","Data":"217d289d19ec55b868645ca6a88f16dfb6043a62b23a650abead30a3420fb881"}
Oct 30 01:53:49.981967 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:49.981965    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm" event={"ID":"60123056-76c2-4e39-a284-d0a9ae34e1e8","Type":"ContainerDied","Data":"3a4e6471a917cba0bba95720160cea6600b8811a784d3deefca34581271e658d"}
Oct 30 01:53:50.000969 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.000945    4425 scope.go:122] "RemoveContainer" containerID="a4d6078b3c08d80f0225529396916cfb9b5104f10a9f7f4ffa86231c00237221"
Oct 30 01:53:50.001361 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:53:50.001336    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4d6078b3c08d80f0225529396916cfb9b5104f10a9f7f4ffa86231c00237221\": container with ID starting with a4d6078b3c08d80f0225529396916cfb9b5104f10a9f7f4ffa86231c00237221 not found: ID does not exist" containerID="a4d6078b3c08d80f0225529396916cfb9b5104f10a9f7f4ffa86231c00237221"
Oct 30 01:53:50.001434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.001374    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d6078b3c08d80f0225529396916cfb9b5104f10a9f7f4ffa86231c00237221"} err="failed to get container status \"a4d6078b3c08d80f0225529396916cfb9b5104f10a9f7f4ffa86231c00237221\": rpc error: code = NotFound desc = could not find container \"a4d6078b3c08d80f0225529396916cfb9b5104f10a9f7f4ffa86231c00237221\": container with ID starting with a4d6078b3c08d80f0225529396916cfb9b5104f10a9f7f4ffa86231c00237221 not found: ID does not exist"
Oct 30 01:53:50.001434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.001398    4425 scope.go:122] "RemoveContainer" containerID="217d289d19ec55b868645ca6a88f16dfb6043a62b23a650abead30a3420fb881"
Oct 30 01:53:50.013102 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.013084    4425 scope.go:122] "RemoveContainer" containerID="217d289d19ec55b868645ca6a88f16dfb6043a62b23a650abead30a3420fb881"
Oct 30 01:53:50.013434 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:53:50.013417    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"217d289d19ec55b868645ca6a88f16dfb6043a62b23a650abead30a3420fb881\": container with ID starting with 217d289d19ec55b868645ca6a88f16dfb6043a62b23a650abead30a3420fb881 not found: ID does not exist" containerID="217d289d19ec55b868645ca6a88f16dfb6043a62b23a650abead30a3420fb881"
Oct 30 01:53:50.013521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.013504    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217d289d19ec55b868645ca6a88f16dfb6043a62b23a650abead30a3420fb881"} err="failed to get container status \"217d289d19ec55b868645ca6a88f16dfb6043a62b23a650abead30a3420fb881\": rpc error: code = NotFound desc = could not find container \"217d289d19ec55b868645ca6a88f16dfb6043a62b23a650abead30a3420fb881\": container with ID starting with 217d289d19ec55b868645ca6a88f16dfb6043a62b23a650abead30a3420fb881 not found: ID does not exist"
Oct 30 01:53:50.066053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.066021    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60123056-76c2-4e39-a284-d0a9ae34e1e8-kube-api-access-j827d\" (UniqueName: \"kubernetes.io/projected/60123056-76c2-4e39-a284-d0a9ae34e1e8-kube-api-access-j827d\") pod \"60123056-76c2-4e39-a284-d0a9ae34e1e8\" (UID: \"60123056-76c2-4e39-a284-d0a9ae34e1e8\") "
Oct 30 01:53:50.066226 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.066073    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60123056-76c2-4e39-a284-d0a9ae34e1e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60123056-76c2-4e39-a284-d0a9ae34e1e8-plugins-config-volume\") pod \"60123056-76c2-4e39-a284-d0a9ae34e1e8\" (UID: \"60123056-76c2-4e39-a284-d0a9ae34e1e8\") "
Oct 30 01:53:50.066312 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.066262    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6003b30d-107b-4075-aa52-8b00fd6f3d2c-kube-api-access-7szdv\" (UniqueName: \"kubernetes.io/projected/6003b30d-107b-4075-aa52-8b00fd6f3d2c-kube-api-access-7szdv\") pod \"6003b30d-107b-4075-aa52-8b00fd6f3d2c\" (UID: \"6003b30d-107b-4075-aa52-8b00fd6f3d2c\") "
Oct 30 01:53:50.066398 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.066374    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6003b30d-107b-4075-aa52-8b00fd6f3d2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6003b30d-107b-4075-aa52-8b00fd6f3d2c-plugins-config-volume\") pod \"6003b30d-107b-4075-aa52-8b00fd6f3d2c\" (UID: \"6003b30d-107b-4075-aa52-8b00fd6f3d2c\") "
Oct 30 01:53:50.067362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.066670    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7n5v\" (UniqueName: \"kubernetes.io/projected/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-kube-api-access-g7n5v\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6\" (UID: \"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"
Oct 30 01:53:50.067362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.066744    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60123056-76c2-4e39-a284-d0a9ae34e1e8-plugins-config-volume" pod "60123056-76c2-4e39-a284-d0a9ae34e1e8" (UID: "60123056-76c2-4e39-a284-d0a9ae34e1e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:50.067362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.066850    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6\" (UID: \"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"
Oct 30 01:53:50.067362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.066916    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6\" (UID: \"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"
Oct 30 01:53:50.067362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.067013    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60123056-76c2-4e39-a284-d0a9ae34e1e8-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:50.067708 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.067569    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6\" (UID: \"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"
Oct 30 01:53:50.068203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.068181    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6003b30d-107b-4075-aa52-8b00fd6f3d2c-plugins-config-volume" pod "6003b30d-107b-4075-aa52-8b00fd6f3d2c" (UID: "6003b30d-107b-4075-aa52-8b00fd6f3d2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:50.070060 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.070040    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6\" (UID: \"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"
Oct 30 01:53:50.070220 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.070165    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6003b30d-107b-4075-aa52-8b00fd6f3d2c-kube-api-access-7szdv" pod "6003b30d-107b-4075-aa52-8b00fd6f3d2c" (UID: "6003b30d-107b-4075-aa52-8b00fd6f3d2c"). InnerVolumeSpecName "kube-api-access-7szdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:50.070318 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.070183    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60123056-76c2-4e39-a284-d0a9ae34e1e8-kube-api-access-j827d" pod "60123056-76c2-4e39-a284-d0a9ae34e1e8" (UID: "60123056-76c2-4e39-a284-d0a9ae34e1e8"). InnerVolumeSpecName "kube-api-access-j827d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:50.089469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.089441    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7n5v\" (UniqueName: \"kubernetes.io/projected/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-kube-api-access-g7n5v\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6\" (UID: \"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"
Oct 30 01:53:50.140633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.140608    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"
Oct 30 01:53:50.168380 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.168328    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j827d\" (UniqueName: \"kubernetes.io/projected/60123056-76c2-4e39-a284-d0a9ae34e1e8-kube-api-access-j827d\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:50.168380 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.168351    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7szdv\" (UniqueName: \"kubernetes.io/projected/6003b30d-107b-4075-aa52-8b00fd6f3d2c-kube-api-access-7szdv\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:50.168380 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.168361    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6003b30d-107b-4075-aa52-8b00fd6f3d2c-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:50.257881 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.257848    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"
Oct 30 01:53:50.269021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.269001    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d481ab61-efd5-4ed6-a9e9-690a210d811b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d481ab61-efd5-4ed6-a9e9-690a210d811b-maas-api-xks-ca\") pod \"d481ab61-efd5-4ed6-a9e9-690a210d811b\" (UID: \"d481ab61-efd5-4ed6-a9e9-690a210d811b\") "
Oct 30 01:53:50.269142 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.269037    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d481ab61-efd5-4ed6-a9e9-690a210d811b-kube-api-access-rxxpt\" (UniqueName: \"kubernetes.io/projected/d481ab61-efd5-4ed6-a9e9-690a210d811b-kube-api-access-rxxpt\") pod \"d481ab61-efd5-4ed6-a9e9-690a210d811b\" (UID: \"d481ab61-efd5-4ed6-a9e9-690a210d811b\") "
Oct 30 01:53:50.269254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.269141    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d481ab61-efd5-4ed6-a9e9-690a210d811b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d481ab61-efd5-4ed6-a9e9-690a210d811b-maas-api-openshift-ca\") pod \"d481ab61-efd5-4ed6-a9e9-690a210d811b\" (UID: \"d481ab61-efd5-4ed6-a9e9-690a210d811b\") "
Oct 30 01:53:50.269879 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.269844    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d481ab61-efd5-4ed6-a9e9-690a210d811b-maas-api-openshift-ca" pod "d481ab61-efd5-4ed6-a9e9-690a210d811b" (UID: "d481ab61-efd5-4ed6-a9e9-690a210d811b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:50.274203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.272969    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d481ab61-efd5-4ed6-a9e9-690a210d811b-kube-api-access-rxxpt" pod "d481ab61-efd5-4ed6-a9e9-690a210d811b" (UID: "d481ab61-efd5-4ed6-a9e9-690a210d811b"). InnerVolumeSpecName "kube-api-access-rxxpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:50.280759 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.280721    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d481ab61-efd5-4ed6-a9e9-690a210d811b-maas-api-xks-ca" pod "d481ab61-efd5-4ed6-a9e9-690a210d811b" (UID: "d481ab61-efd5-4ed6-a9e9-690a210d811b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:53:50.323928 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.323196    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"]
Oct 30 01:53:50.331130 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.331102    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bdf6ca-69fc85944d-pdwxp"]
Oct 30 01:53:50.371910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.371880    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d481ab61-efd5-4ed6-a9e9-690a210d811b-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:50.371910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.371909    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxxpt\" (UniqueName: \"kubernetes.io/projected/d481ab61-efd5-4ed6-a9e9-690a210d811b-kube-api-access-rxxpt\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:50.372178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.371918    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d481ab61-efd5-4ed6-a9e9-690a210d811b-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:50.571459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.571431    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"]
Oct 30 01:53:50.572350 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:53:50.572329    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ecd6b8_4c3a_4b60_88cc_190c50de0cd8.slice/crio-d3142bf5f1ff7e9963cadab7d00b761c4b2cc4992cf67bdaab4c41a763b475b2 WatchSource:0}: Error finding container d3142bf5f1ff7e9963cadab7d00b761c4b2cc4992cf67bdaab4c41a763b475b2: Status 404 returned error can't find the container with id d3142bf5f1ff7e9963cadab7d00b761c4b2cc4992cf67bdaab4c41a763b475b2
Oct 30 01:53:50.990780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.990754    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"
Oct 30 01:53:50.991256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.990755    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg" event={"ID":"d481ab61-efd5-4ed6-a9e9-690a210d811b","Type":"ContainerDied","Data":"34977c7c778fd0e110bcb247938c6949c6db4358c7fc990a2b22b46e281f1362"}
Oct 30 01:53:50.991256 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.990897    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34977c7c778fd0e110bcb247938c6949c6db4358c7fc990a2b22b46e281f1362"
Oct 30 01:53:50.992952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.992925    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"
Oct 30 01:53:50.994876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.994846    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6" event={"ID":"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8","Type":"ContainerStarted","Data":"8d39eee47ade30fadaab4a2190584a072055ce7fee4b6c0b15a76ca0ac29be23"}
Oct 30 01:53:50.994998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:50.994894    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6" event={"ID":"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8","Type":"ContainerStarted","Data":"d3142bf5f1ff7e9963cadab7d00b761c4b2cc4992cf67bdaab4c41a763b475b2"}
Oct 30 01:53:51.043494 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:51.043448    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6" podStartSLOduration=2.043434278 podStartE2EDuration="2.043434278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:51.039914216 +0000 UTC m=+4339.789569523" watchObservedRunningTime="2026-10-30 01:53:51.043434278 +0000 UTC m=+4339.793089648"
Oct 30 01:53:51.072164 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:51.072135    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"]
Oct 30 01:53:51.078512 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:51.078489    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bdf6ca-5d9dc5b799-qlzqm"]
Oct 30 01:53:51.423814 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:51.423773    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6003b30d-107b-4075-aa52-8b00fd6f3d2c" path="/var/lib/kubelet/pods/6003b30d-107b-4075-aa52-8b00fd6f3d2c/volumes"
Oct 30 01:53:51.424280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:51.424256    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60123056-76c2-4e39-a284-d0a9ae34e1e8" path="/var/lib/kubelet/pods/60123056-76c2-4e39-a284-d0a9ae34e1e8/volumes"
Oct 30 01:53:52.003437 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:52.003403    4425 generic.go:356] "Generic (PLEG): container finished" podID="a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8" containerID="8d39eee47ade30fadaab4a2190584a072055ce7fee4b6c0b15a76ca0ac29be23" exitCode=0
Oct 30 01:53:52.003835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:52.003482    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6" event={"ID":"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8","Type":"ContainerDied","Data":"8d39eee47ade30fadaab4a2190584a072055ce7fee4b6c0b15a76ca0ac29be23"}
Oct 30 01:53:52.388261 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:52.388224    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj" podUID="23a09e9b-a4fa-4ef9-8d76-b75e98f68a74" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:53:53.183282 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:53.183257    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"
Oct 30 01:53:53.304530 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:53.304446    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-maas-api-openshift-ca\") pod \"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8\" (UID: \"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8\") "
Oct 30 01:53:53.304530 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:53.304495    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-kube-api-access-g7n5v\" (UniqueName: \"kubernetes.io/projected/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-kube-api-access-g7n5v\") pod \"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8\" (UID: \"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8\") "
Oct 30 01:53:53.304767 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:53.304674    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-maas-api-xks-ca\") pod \"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8\" (UID: \"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8\") "
Oct 30 01:53:53.304900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:53.304875    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-maas-api-openshift-ca" pod "a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8" (UID: "a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:53.305053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:53.305036    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:53.306641 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:53.306616    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-kube-api-access-g7n5v" pod "a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8" (UID: "a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8"). InnerVolumeSpecName "kube-api-access-g7n5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:53.306770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:53.306667    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-maas-api-xks-ca" pod "a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8" (UID: "a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:53:53.406549 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:53.406522    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:53.406549 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:53.406547    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7n5v\" (UniqueName: \"kubernetes.io/projected/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8-kube-api-access-g7n5v\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:54.021164 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:54.021121    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"
Oct 30 01:53:54.021326 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:54.021120    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6" event={"ID":"a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8","Type":"ContainerDied","Data":"d3142bf5f1ff7e9963cadab7d00b761c4b2cc4992cf67bdaab4c41a763b475b2"}
Oct 30 01:53:54.021326 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:54.021272    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3142bf5f1ff7e9963cadab7d00b761c4b2cc4992cf67bdaab4c41a763b475b2"
Oct 30 01:53:55.391946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.391907    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"]
Oct 30 01:53:55.392819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.392798    4425 cpu_manager.go:414] "containerMap: removing container" podUID="6003b30d-107b-4075-aa52-8b00fd6f3d2c" containerName="payload-processing"
Oct 30 01:53:55.392900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.392822    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6003b30d-107b-4075-aa52-8b00fd6f3d2c" containerName="payload-processing"
Oct 30 01:53:55.392900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.392876    4425 cpu_manager.go:414] "containerMap: removing container" podUID="60123056-76c2-4e39-a284-d0a9ae34e1e8" containerName="payload-pre-processing"
Oct 30 01:53:55.392900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.392885    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60123056-76c2-4e39-a284-d0a9ae34e1e8" containerName="payload-pre-processing"
Oct 30 01:53:55.392900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.392898    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d481ab61-efd5-4ed6-a9e9-690a210d811b" containerName="revoke-keys"
Oct 30 01:53:55.393107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.392906    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d481ab61-efd5-4ed6-a9e9-690a210d811b" containerName="revoke-keys"
Oct 30 01:53:55.393107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.392937    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8" containerName="revoke-keys"
Oct 30 01:53:55.393107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.392945    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8" containerName="revoke-keys"
Oct 30 01:53:55.393107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.393073    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="60123056-76c2-4e39-a284-d0a9ae34e1e8" containerName="payload-pre-processing"
Oct 30 01:53:55.393107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.393085    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d481ab61-efd5-4ed6-a9e9-690a210d811b" containerName="revoke-keys"
Oct 30 01:53:55.393107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.393099    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8" containerName="revoke-keys"
Oct 30 01:53:55.393422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.393110    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="6003b30d-107b-4075-aa52-8b00fd6f3d2c" containerName="payload-processing"
Oct 30 01:53:55.396633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.396613    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"
Oct 30 01:53:55.421360 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.421331    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:53:55.421518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.421370    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:53:55.421518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.421405    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:53:55.421518 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.421457    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:53:55.427702 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.427675    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"]
Oct 30 01:53:55.533311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.533274    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf\" (UID: \"a6146564-9202-4acd-a3cc-8d6d4c1efbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"
Oct 30 01:53:55.533508 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.533408    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf\" (UID: \"a6146564-9202-4acd-a3cc-8d6d4c1efbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"
Oct 30 01:53:55.533508 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.533450    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hw99\" (UniqueName: \"kubernetes.io/projected/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-kube-api-access-9hw99\") pod \"maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf\" (UID: \"a6146564-9202-4acd-a3cc-8d6d4c1efbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"
Oct 30 01:53:55.634680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.634641    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf\" (UID: \"a6146564-9202-4acd-a3cc-8d6d4c1efbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"
Oct 30 01:53:55.634865 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.634764    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf\" (UID: \"a6146564-9202-4acd-a3cc-8d6d4c1efbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"
Oct 30 01:53:55.634865 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.634823    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hw99\" (UniqueName: \"kubernetes.io/projected/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-kube-api-access-9hw99\") pod \"maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf\" (UID: \"a6146564-9202-4acd-a3cc-8d6d4c1efbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"
Oct 30 01:53:55.635407 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.635380    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf\" (UID: \"a6146564-9202-4acd-a3cc-8d6d4c1efbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"
Oct 30 01:53:55.636942 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.636919    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf\" (UID: \"a6146564-9202-4acd-a3cc-8d6d4c1efbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"
Oct 30 01:53:55.656062 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.656006    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hw99\" (UniqueName: \"kubernetes.io/projected/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-kube-api-access-9hw99\") pod \"maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf\" (UID: \"a6146564-9202-4acd-a3cc-8d6d4c1efbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"
Oct 30 01:53:55.712350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:55.712327    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"
Oct 30 01:53:56.019132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:56.019106    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"]
Oct 30 01:53:56.020459 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:53:56.020435    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6146564_9202_4acd_a3cc_8d6d4c1efbf6.slice/crio-1e430f376833cf1392c16e5228942ae45f5c7b2f98aba1e70156304f5b31c995 WatchSource:0}: Error finding container 1e430f376833cf1392c16e5228942ae45f5c7b2f98aba1e70156304f5b31c995: Status 404 returned error can't find the container with id 1e430f376833cf1392c16e5228942ae45f5c7b2f98aba1e70156304f5b31c995
Oct 30 01:53:56.050513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:56.050480    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf" event={"ID":"a6146564-9202-4acd-a3cc-8d6d4c1efbf6","Type":"ContainerStarted","Data":"1e430f376833cf1392c16e5228942ae45f5c7b2f98aba1e70156304f5b31c995"}
Oct 30 01:53:56.974707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:56.974675    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"]
Oct 30 01:53:56.984047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:56.984016    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bdf6ca-jgkg6"]
Oct 30 01:53:57.060763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:57.060726    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf" event={"ID":"a6146564-9202-4acd-a3cc-8d6d4c1efbf6","Type":"ContainerStarted","Data":"5cdd9cf0657b9e2aa4ec1962756a5ae6080a27ad9ef95ee3c0f23057807332b6"}
Oct 30 01:53:57.098073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:57.098020    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf" podStartSLOduration=2.098007327 podStartE2EDuration="2.098007327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:57.095684107 +0000 UTC m=+4345.845339415" watchObservedRunningTime="2026-10-30 01:53:57.098007327 +0000 UTC m=+4345.847662688"
Oct 30 01:53:57.423987 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:57.423958    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="582a1d8e-713d-4bd6-a44e-2d7f2714cb05" path="/var/lib/kubelet/pods/582a1d8e-713d-4bd6-a44e-2d7f2714cb05/volumes"
Oct 30 01:53:57.820196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:57.820165    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"]
Oct 30 01:53:57.857146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:57.857115    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"]
Oct 30 01:53:58.074004 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:58.073810    4425 generic.go:356] "Generic (PLEG): container finished" podID="a6146564-9202-4acd-a3cc-8d6d4c1efbf6" containerID="5cdd9cf0657b9e2aa4ec1962756a5ae6080a27ad9ef95ee3c0f23057807332b6" exitCode=0
Oct 30 01:53:58.074004 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:58.073903    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf" event={"ID":"a6146564-9202-4acd-a3cc-8d6d4c1efbf6","Type":"ContainerDied","Data":"5cdd9cf0657b9e2aa4ec1962756a5ae6080a27ad9ef95ee3c0f23057807332b6"}
Oct 30 01:53:59.250102 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:59.250076    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"
Oct 30 01:53:59.381006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:59.380919    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-maas-api-xks-ca\") pod \"a6146564-9202-4acd-a3cc-8d6d4c1efbf6\" (UID: \"a6146564-9202-4acd-a3cc-8d6d4c1efbf6\") "
Oct 30 01:53:59.381006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:59.380974    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-kube-api-access-9hw99\" (UniqueName: \"kubernetes.io/projected/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-kube-api-access-9hw99\") pod \"a6146564-9202-4acd-a3cc-8d6d4c1efbf6\" (UID: \"a6146564-9202-4acd-a3cc-8d6d4c1efbf6\") "
Oct 30 01:53:59.381270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:59.381082    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-maas-api-openshift-ca\") pod \"a6146564-9202-4acd-a3cc-8d6d4c1efbf6\" (UID: \"a6146564-9202-4acd-a3cc-8d6d4c1efbf6\") "
Oct 30 01:53:59.381432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:59.381401    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-maas-api-openshift-ca" pod "a6146564-9202-4acd-a3cc-8d6d4c1efbf6" (UID: "a6146564-9202-4acd-a3cc-8d6d4c1efbf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:53:59.381660 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:59.381638    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:59.383063 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:59.383039    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-maas-api-xks-ca" pod "a6146564-9202-4acd-a3cc-8d6d4c1efbf6" (UID: "a6146564-9202-4acd-a3cc-8d6d4c1efbf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:53:59.383263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:59.383240    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-kube-api-access-9hw99" pod "a6146564-9202-4acd-a3cc-8d6d4c1efbf6" (UID: "a6146564-9202-4acd-a3cc-8d6d4c1efbf6"). InnerVolumeSpecName "kube-api-access-9hw99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:53:59.483136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:59.483108    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:53:59.483136 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:53:59.483133    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hw99\" (UniqueName: \"kubernetes.io/projected/a6146564-9202-4acd-a3cc-8d6d4c1efbf6-kube-api-access-9hw99\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:00.089896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:00.089855    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf" event={"ID":"a6146564-9202-4acd-a3cc-8d6d4c1efbf6","Type":"ContainerDied","Data":"1e430f376833cf1392c16e5228942ae45f5c7b2f98aba1e70156304f5b31c995"}
Oct 30 01:54:00.090056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:00.089913    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e430f376833cf1392c16e5228942ae45f5c7b2f98aba1e70156304f5b31c995"
Oct 30 01:54:00.090056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:00.089868    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"
Oct 30 01:54:00.345602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:00.345515    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"]
Oct 30 01:54:02.388266 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:02.388223    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj" podUID="23a09e9b-a4fa-4ef9-8d76-b75e98f68a74" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:02.821461 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:02.821419    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl" podUID="d36566e9-c18e-4026-b593-467e3491e95a" containerName="payload-processing" containerID="cri-o://081cd735a10af86c406dbe3edfd6575895f2b2247ce78d6a63233522af925a42" gracePeriod=25
Oct 30 01:54:02.858345 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:02.858307    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl" podUID="aaf97e80-fd90-4db8-84a9-872db881b7c7" containerName="payload-pre-processing" containerID="cri-o://4215f23058a3c9b739702d1e19f7dab71b8d0e39e2d76dd72a2db762b9ada5d0" gracePeriod=25
Oct 30 01:54:03.341825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:03.341799    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"
Oct 30 01:54:03.530220 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:03.530190    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a09e9b-a4fa-4ef9-8d76-b75e98f68a74-kube-api-access-5qv2d\" (UniqueName: \"kubernetes.io/projected/23a09e9b-a4fa-4ef9-8d76-b75e98f68a74-kube-api-access-5qv2d\") pod \"23a09e9b-a4fa-4ef9-8d76-b75e98f68a74\" (UID: \"23a09e9b-a4fa-4ef9-8d76-b75e98f68a74\") "
Oct 30 01:54:03.530678 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:03.530277    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a09e9b-a4fa-4ef9-8d76-b75e98f68a74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a09e9b-a4fa-4ef9-8d76-b75e98f68a74-plugins-config-volume\") pod \"23a09e9b-a4fa-4ef9-8d76-b75e98f68a74\" (UID: \"23a09e9b-a4fa-4ef9-8d76-b75e98f68a74\") "
Oct 30 01:54:03.530678 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:03.530666    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a09e9b-a4fa-4ef9-8d76-b75e98f68a74-plugins-config-volume" pod "23a09e9b-a4fa-4ef9-8d76-b75e98f68a74" (UID: "23a09e9b-a4fa-4ef9-8d76-b75e98f68a74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:03.532241 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:03.532219    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a09e9b-a4fa-4ef9-8d76-b75e98f68a74-kube-api-access-5qv2d" pod "23a09e9b-a4fa-4ef9-8d76-b75e98f68a74" (UID: "23a09e9b-a4fa-4ef9-8d76-b75e98f68a74"). InnerVolumeSpecName "kube-api-access-5qv2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:03.631529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:03.631495    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qv2d\" (UniqueName: \"kubernetes.io/projected/23a09e9b-a4fa-4ef9-8d76-b75e98f68a74-kube-api-access-5qv2d\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:03.631529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:03.631522    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a09e9b-a4fa-4ef9-8d76-b75e98f68a74-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:04.133804 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:04.133767    4425 generic.go:356] "Generic (PLEG): container finished" podID="23a09e9b-a4fa-4ef9-8d76-b75e98f68a74" containerID="45936c75d450850f91b4c2b146e767ba5340965f4c92fc3e213c16158c44ce12" exitCode=0
Oct 30 01:54:04.134015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:04.133822    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"
Oct 30 01:54:04.134015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:04.133858    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj" event={"ID":"23a09e9b-a4fa-4ef9-8d76-b75e98f68a74","Type":"ContainerDied","Data":"45936c75d450850f91b4c2b146e767ba5340965f4c92fc3e213c16158c44ce12"}
Oct 30 01:54:04.134015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:04.133922    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj" event={"ID":"23a09e9b-a4fa-4ef9-8d76-b75e98f68a74","Type":"ContainerDied","Data":"8ddd526f3aa01e562d88f0caaded8a6c8c0dcbd51e2a1f9f838db56610192745"}
Oct 30 01:54:04.134015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:04.133951    4425 scope.go:122] "RemoveContainer" containerID="45936c75d450850f91b4c2b146e767ba5340965f4c92fc3e213c16158c44ce12"
Oct 30 01:54:04.145968 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:04.145950    4425 scope.go:122] "RemoveContainer" containerID="45936c75d450850f91b4c2b146e767ba5340965f4c92fc3e213c16158c44ce12"
Oct 30 01:54:04.146271 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:54:04.146253    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45936c75d450850f91b4c2b146e767ba5340965f4c92fc3e213c16158c44ce12\": container with ID starting with 45936c75d450850f91b4c2b146e767ba5340965f4c92fc3e213c16158c44ce12 not found: ID does not exist" containerID="45936c75d450850f91b4c2b146e767ba5340965f4c92fc3e213c16158c44ce12"
Oct 30 01:54:04.146337 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:04.146280    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45936c75d450850f91b4c2b146e767ba5340965f4c92fc3e213c16158c44ce12"} err="failed to get container status \"45936c75d450850f91b4c2b146e767ba5340965f4c92fc3e213c16158c44ce12\": rpc error: code = NotFound desc = could not find container \"45936c75d450850f91b4c2b146e767ba5340965f4c92fc3e213c16158c44ce12\": container with ID starting with 45936c75d450850f91b4c2b146e767ba5340965f4c92fc3e213c16158c44ce12 not found: ID does not exist"
Oct 30 01:54:04.183432 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:04.183409    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"]
Oct 30 01:54:04.215754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:04.215730    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c48628-8448444687-vfbkj"]
Oct 30 01:54:05.345854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:05.345812    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n" podUID="18c8b78c-9e58-4f50-90c6-16684bfe274a" containerName="payload-pre-processing" containerID="cri-o://f116d49e73054fd253d98977a9e07d6549e4724014e38060be92be1ef212f5f1" gracePeriod=25
Oct 30 01:54:05.423305 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:05.423278    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a09e9b-a4fa-4ef9-8d76-b75e98f68a74" path="/var/lib/kubelet/pods/23a09e9b-a4fa-4ef9-8d76-b75e98f68a74/volumes"
Oct 30 01:54:05.799841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:05.799806    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"]
Oct 30 01:54:05.830038 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:05.830008    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"]
Oct 30 01:54:06.090042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:06.089959    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n" podUID="18c8b78c-9e58-4f50-90c6-16684bfe274a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:07.096490 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.096456    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"]
Oct 30 01:54:07.097024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.097010    4425 cpu_manager.go:414] "containerMap: removing container" podUID="23a09e9b-a4fa-4ef9-8d76-b75e98f68a74" containerName="payload-pre-processing"
Oct 30 01:54:07.097075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.097025    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a09e9b-a4fa-4ef9-8d76-b75e98f68a74" containerName="payload-pre-processing"
Oct 30 01:54:07.097075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.097065    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a6146564-9202-4acd-a3cc-8d6d4c1efbf6" containerName="revoke-keys"
Oct 30 01:54:07.097075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.097070    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6146564-9202-4acd-a3cc-8d6d4c1efbf6" containerName="revoke-keys"
Oct 30 01:54:07.097205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.097184    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6146564-9202-4acd-a3cc-8d6d4c1efbf6" containerName="revoke-keys"
Oct 30 01:54:07.097205 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.097197    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a09e9b-a4fa-4ef9-8d76-b75e98f68a74" containerName="payload-pre-processing"
Oct 30 01:54:07.101953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.101937    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"
Oct 30 01:54:07.121272 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.121231    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"]
Oct 30 01:54:07.126085 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.126059    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:54:07.126259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.126148    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:54:07.163092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.163058    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnkpf\" (UniqueName: \"kubernetes.io/projected/a7046f8f-887d-42d9-9a30-6e0f369ccf29-kube-api-access-pnkpf\") pod \"maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj\" (UID: \"a7046f8f-887d-42d9-9a30-6e0f369ccf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"
Oct 30 01:54:07.163265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.163122    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7046f8f-887d-42d9-9a30-6e0f369ccf29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj\" (UID: \"a7046f8f-887d-42d9-9a30-6e0f369ccf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"
Oct 30 01:54:07.163265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.163194    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7046f8f-887d-42d9-9a30-6e0f369ccf29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj\" (UID: \"a7046f8f-887d-42d9-9a30-6e0f369ccf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"
Oct 30 01:54:07.199220 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.199196    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:54:07.199392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.199377    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:54:07.263908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.263872    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnkpf\" (UniqueName: \"kubernetes.io/projected/a7046f8f-887d-42d9-9a30-6e0f369ccf29-kube-api-access-pnkpf\") pod \"maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj\" (UID: \"a7046f8f-887d-42d9-9a30-6e0f369ccf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"
Oct 30 01:54:07.264086 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.263944    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7046f8f-887d-42d9-9a30-6e0f369ccf29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj\" (UID: \"a7046f8f-887d-42d9-9a30-6e0f369ccf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"
Oct 30 01:54:07.264086 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.263969    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7046f8f-887d-42d9-9a30-6e0f369ccf29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj\" (UID: \"a7046f8f-887d-42d9-9a30-6e0f369ccf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"
Oct 30 01:54:07.264626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.264604    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7046f8f-887d-42d9-9a30-6e0f369ccf29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj\" (UID: \"a7046f8f-887d-42d9-9a30-6e0f369ccf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"
Oct 30 01:54:07.266395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.266372    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7046f8f-887d-42d9-9a30-6e0f369ccf29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj\" (UID: \"a7046f8f-887d-42d9-9a30-6e0f369ccf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"
Oct 30 01:54:07.284517 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.284481    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnkpf\" (UniqueName: \"kubernetes.io/projected/a7046f8f-887d-42d9-9a30-6e0f369ccf29-kube-api-access-pnkpf\") pod \"maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj\" (UID: \"a7046f8f-887d-42d9-9a30-6e0f369ccf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"
Oct 30 01:54:07.417409 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.417317    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"
Oct 30 01:54:07.554510 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:54:07.554477    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:54:07.716579 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.716545    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"]
Oct 30 01:54:07.716751 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:54:07.716725    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7046f8f_887d_42d9_9a30_6e0f369ccf29.slice/crio-fb01ab4879293f1cb4dfecc549150652240d931f6a26e0e0cb7749d8d903f986 WatchSource:0}: Error finding container fb01ab4879293f1cb4dfecc549150652240d931f6a26e0e0cb7749d8d903f986: Status 404 returned error can't find the container with id fb01ab4879293f1cb4dfecc549150652240d931f6a26e0e0cb7749d8d903f986
Oct 30 01:54:07.777980 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:07.777946    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl" podUID="aaf97e80-fd90-4db8-84a9-872db881b7c7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:08.166116 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:08.166078    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj" event={"ID":"a7046f8f-887d-42d9-9a30-6e0f369ccf29","Type":"ContainerStarted","Data":"c681867b378c27544a2cd1f7d68adc0e65f6fe2c32ff09851f92bac285b7b490"}
Oct 30 01:54:08.166499 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:08.166181    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj" event={"ID":"a7046f8f-887d-42d9-9a30-6e0f369ccf29","Type":"ContainerStarted","Data":"fb01ab4879293f1cb4dfecc549150652240d931f6a26e0e0cb7749d8d903f986"}
Oct 30 01:54:08.200863 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:08.200814    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj" podStartSLOduration=1.200803568 podStartE2EDuration="1.200803568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:08.200489559 +0000 UTC m=+4356.950144889" watchObservedRunningTime="2026-10-30 01:54:08.200803568 +0000 UTC m=+4356.950458875"
Oct 30 01:54:08.784742 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:08.784707    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl" podUID="d36566e9-c18e-4026-b593-467e3491e95a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:09.176241 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:09.176210    4425 generic.go:356] "Generic (PLEG): container finished" podID="a7046f8f-887d-42d9-9a30-6e0f369ccf29" containerID="c681867b378c27544a2cd1f7d68adc0e65f6fe2c32ff09851f92bac285b7b490" exitCode=0
Oct 30 01:54:09.176641 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:09.176274    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj" event={"ID":"a7046f8f-887d-42d9-9a30-6e0f369ccf29","Type":"ContainerDied","Data":"c681867b378c27544a2cd1f7d68adc0e65f6fe2c32ff09851f92bac285b7b490"}
Oct 30 01:54:10.342457 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:10.342428    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"
Oct 30 01:54:10.398169 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:10.398134    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7046f8f-887d-42d9-9a30-6e0f369ccf29-kube-api-access-pnkpf\" (UniqueName: \"kubernetes.io/projected/a7046f8f-887d-42d9-9a30-6e0f369ccf29-kube-api-access-pnkpf\") pod \"a7046f8f-887d-42d9-9a30-6e0f369ccf29\" (UID: \"a7046f8f-887d-42d9-9a30-6e0f369ccf29\") "
Oct 30 01:54:10.398347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:10.398206    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7046f8f-887d-42d9-9a30-6e0f369ccf29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7046f8f-887d-42d9-9a30-6e0f369ccf29-maas-api-openshift-ca\") pod \"a7046f8f-887d-42d9-9a30-6e0f369ccf29\" (UID: \"a7046f8f-887d-42d9-9a30-6e0f369ccf29\") "
Oct 30 01:54:10.398347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:10.398285    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7046f8f-887d-42d9-9a30-6e0f369ccf29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7046f8f-887d-42d9-9a30-6e0f369ccf29-maas-api-xks-ca\") pod \"a7046f8f-887d-42d9-9a30-6e0f369ccf29\" (UID: \"a7046f8f-887d-42d9-9a30-6e0f369ccf29\") "
Oct 30 01:54:10.398627 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:10.398587    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7046f8f-887d-42d9-9a30-6e0f369ccf29-maas-api-openshift-ca" pod "a7046f8f-887d-42d9-9a30-6e0f369ccf29" (UID: "a7046f8f-887d-42d9-9a30-6e0f369ccf29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:10.398750 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:10.398736    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7046f8f-887d-42d9-9a30-6e0f369ccf29-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:10.400475 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:10.400452    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7046f8f-887d-42d9-9a30-6e0f369ccf29-kube-api-access-pnkpf" pod "a7046f8f-887d-42d9-9a30-6e0f369ccf29" (UID: "a7046f8f-887d-42d9-9a30-6e0f369ccf29"). InnerVolumeSpecName "kube-api-access-pnkpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:10.400676 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:10.400661    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7046f8f-887d-42d9-9a30-6e0f369ccf29-maas-api-xks-ca" pod "a7046f8f-887d-42d9-9a30-6e0f369ccf29" (UID: "a7046f8f-887d-42d9-9a30-6e0f369ccf29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:54:10.499963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:10.499875    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7046f8f-887d-42d9-9a30-6e0f369ccf29-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:10.499963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:10.499908    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnkpf\" (UniqueName: \"kubernetes.io/projected/a7046f8f-887d-42d9-9a30-6e0f369ccf29-kube-api-access-pnkpf\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:10.800713 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:10.800664    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg" podUID="045aea57-0daf-4b8d-96b7-38073146d269" containerName="payload-processing" containerID="cri-o://e4fc60add2e63e88d9bef7b9db6cb1717987420d951212ef50d3c16483b2c3a7" gracePeriod=25
Oct 30 01:54:10.830247 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:10.830205    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz" podUID="0d78846e-a31f-4911-8c8f-4d0a93748883" containerName="payload-pre-processing" containerID="cri-o://2c927f7a0cf34a53ae8717e0de68fea2f4a1c04a13c3ff46781a2a5353dcb880" gracePeriod=25
Oct 30 01:54:11.190265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:11.190176    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz" podUID="0d78846e-a31f-4911-8c8f-4d0a93748883" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:11.192570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:11.192548    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"
Oct 30 01:54:11.192716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:11.192583    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj" event={"ID":"a7046f8f-887d-42d9-9a30-6e0f369ccf29","Type":"ContainerDied","Data":"fb01ab4879293f1cb4dfecc549150652240d931f6a26e0e0cb7749d8d903f986"}
Oct 30 01:54:11.192716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:11.192625    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb01ab4879293f1cb4dfecc549150652240d931f6a26e0e0cb7749d8d903f986"
Oct 30 01:54:11.857920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:11.857883    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"]
Oct 30 01:54:12.196297 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:12.196213    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg" podUID="045aea57-0daf-4b8d-96b7-38073146d269" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:16.090612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:16.090536    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n" podUID="18c8b78c-9e58-4f50-90c6-16684bfe274a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:17.778388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:17.778346    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl" podUID="aaf97e80-fd90-4db8-84a9-872db881b7c7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:18.785502 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:18.785467    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl" podUID="d36566e9-c18e-4026-b593-467e3491e95a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:19.747224 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:19.747175    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"]
Oct 30 01:54:19.756170 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:19.756131    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0e1a98-dhkdg"]
Oct 30 01:54:21.189781 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:21.189737    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz" podUID="0d78846e-a31f-4911-8c8f-4d0a93748883" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:21.424739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:21.424701    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d481ab61-efd5-4ed6-a9e9-690a210d811b" path="/var/lib/kubelet/pods/d481ab61-efd5-4ed6-a9e9-690a210d811b/volumes"
Oct 30 01:54:22.007301 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:22.007271    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"]
Oct 30 01:54:22.016908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:22.016880    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-35042a-czhs6"]
Oct 30 01:54:22.196150 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:22.196110    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg" podUID="045aea57-0daf-4b8d-96b7-38073146d269" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:23.129514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.129487    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"
Oct 30 01:54:23.151721 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.151699    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"
Oct 30 01:54:23.244609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.244580    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d36566e9-c18e-4026-b593-467e3491e95a-kube-api-access-47slg\" (UniqueName: \"kubernetes.io/projected/d36566e9-c18e-4026-b593-467e3491e95a-kube-api-access-47slg\") pod \"d36566e9-c18e-4026-b593-467e3491e95a\" (UID: \"d36566e9-c18e-4026-b593-467e3491e95a\") "
Oct 30 01:54:23.245074 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.244669    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d36566e9-c18e-4026-b593-467e3491e95a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36566e9-c18e-4026-b593-467e3491e95a-plugins-config-volume\") pod \"d36566e9-c18e-4026-b593-467e3491e95a\" (UID: \"d36566e9-c18e-4026-b593-467e3491e95a\") "
Oct 30 01:54:23.245074 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.244689    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf97e80-fd90-4db8-84a9-872db881b7c7-kube-api-access-bpxxk\" (UniqueName: \"kubernetes.io/projected/aaf97e80-fd90-4db8-84a9-872db881b7c7-kube-api-access-bpxxk\") pod \"aaf97e80-fd90-4db8-84a9-872db881b7c7\" (UID: \"aaf97e80-fd90-4db8-84a9-872db881b7c7\") "
Oct 30 01:54:23.245074 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.244796    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf97e80-fd90-4db8-84a9-872db881b7c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf97e80-fd90-4db8-84a9-872db881b7c7-plugins-config-volume\") pod \"aaf97e80-fd90-4db8-84a9-872db881b7c7\" (UID: \"aaf97e80-fd90-4db8-84a9-872db881b7c7\") "
Oct 30 01:54:23.245425 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.245094    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36566e9-c18e-4026-b593-467e3491e95a-plugins-config-volume" pod "d36566e9-c18e-4026-b593-467e3491e95a" (UID: "d36566e9-c18e-4026-b593-467e3491e95a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:23.245425 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.245209    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf97e80-fd90-4db8-84a9-872db881b7c7-plugins-config-volume" pod "aaf97e80-fd90-4db8-84a9-872db881b7c7" (UID: "aaf97e80-fd90-4db8-84a9-872db881b7c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:23.246878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.246858    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf97e80-fd90-4db8-84a9-872db881b7c7-kube-api-access-bpxxk" pod "aaf97e80-fd90-4db8-84a9-872db881b7c7" (UID: "aaf97e80-fd90-4db8-84a9-872db881b7c7"). InnerVolumeSpecName "kube-api-access-bpxxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:23.246947 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.246872    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36566e9-c18e-4026-b593-467e3491e95a-kube-api-access-47slg" pod "d36566e9-c18e-4026-b593-467e3491e95a" (UID: "d36566e9-c18e-4026-b593-467e3491e95a"). InnerVolumeSpecName "kube-api-access-47slg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:23.279516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.279487    4425 generic.go:356] "Generic (PLEG): container finished" podID="aaf97e80-fd90-4db8-84a9-872db881b7c7" containerID="4215f23058a3c9b739702d1e19f7dab71b8d0e39e2d76dd72a2db762b9ada5d0" exitCode=0
Oct 30 01:54:23.279643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.279523    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"
Oct 30 01:54:23.279643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.279565    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl" event={"ID":"aaf97e80-fd90-4db8-84a9-872db881b7c7","Type":"ContainerDied","Data":"4215f23058a3c9b739702d1e19f7dab71b8d0e39e2d76dd72a2db762b9ada5d0"}
Oct 30 01:54:23.279763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.279652    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl" event={"ID":"aaf97e80-fd90-4db8-84a9-872db881b7c7","Type":"ContainerDied","Data":"52f874fc487a54b9a8cc72a6a6200fd157e47c9f9387c7ef0cce38799adafa72"}
Oct 30 01:54:23.279763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.279681    4425 scope.go:122] "RemoveContainer" containerID="4215f23058a3c9b739702d1e19f7dab71b8d0e39e2d76dd72a2db762b9ada5d0"
Oct 30 01:54:23.281071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.281050    4425 generic.go:356] "Generic (PLEG): container finished" podID="d36566e9-c18e-4026-b593-467e3491e95a" containerID="081cd735a10af86c406dbe3edfd6575895f2b2247ce78d6a63233522af925a42" exitCode=0
Oct 30 01:54:23.281185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.281075    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl" event={"ID":"d36566e9-c18e-4026-b593-467e3491e95a","Type":"ContainerDied","Data":"081cd735a10af86c406dbe3edfd6575895f2b2247ce78d6a63233522af925a42"}
Oct 30 01:54:23.281185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.281106    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"
Oct 30 01:54:23.281185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.281117    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl" event={"ID":"d36566e9-c18e-4026-b593-467e3491e95a","Type":"ContainerDied","Data":"b56fed4ede55e3c54fdb6c45d0f0a24231dc4d0bea86320b0477682234341faf"}
Oct 30 01:54:23.290864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.290781    4425 scope.go:122] "RemoveContainer" containerID="4215f23058a3c9b739702d1e19f7dab71b8d0e39e2d76dd72a2db762b9ada5d0"
Oct 30 01:54:23.291132 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:54:23.291108    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4215f23058a3c9b739702d1e19f7dab71b8d0e39e2d76dd72a2db762b9ada5d0\": container with ID starting with 4215f23058a3c9b739702d1e19f7dab71b8d0e39e2d76dd72a2db762b9ada5d0 not found: ID does not exist" containerID="4215f23058a3c9b739702d1e19f7dab71b8d0e39e2d76dd72a2db762b9ada5d0"
Oct 30 01:54:23.291248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.291208    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4215f23058a3c9b739702d1e19f7dab71b8d0e39e2d76dd72a2db762b9ada5d0"} err="failed to get container status \"4215f23058a3c9b739702d1e19f7dab71b8d0e39e2d76dd72a2db762b9ada5d0\": rpc error: code = NotFound desc = could not find container \"4215f23058a3c9b739702d1e19f7dab71b8d0e39e2d76dd72a2db762b9ada5d0\": container with ID starting with 4215f23058a3c9b739702d1e19f7dab71b8d0e39e2d76dd72a2db762b9ada5d0 not found: ID does not exist"
Oct 30 01:54:23.291248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.291237    4425 scope.go:122] "RemoveContainer" containerID="081cd735a10af86c406dbe3edfd6575895f2b2247ce78d6a63233522af925a42"
Oct 30 01:54:23.302560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.302540    4425 scope.go:122] "RemoveContainer" containerID="081cd735a10af86c406dbe3edfd6575895f2b2247ce78d6a63233522af925a42"
Oct 30 01:54:23.302859 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:54:23.302839    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081cd735a10af86c406dbe3edfd6575895f2b2247ce78d6a63233522af925a42\": container with ID starting with 081cd735a10af86c406dbe3edfd6575895f2b2247ce78d6a63233522af925a42 not found: ID does not exist" containerID="081cd735a10af86c406dbe3edfd6575895f2b2247ce78d6a63233522af925a42"
Oct 30 01:54:23.302942 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.302864    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081cd735a10af86c406dbe3edfd6575895f2b2247ce78d6a63233522af925a42"} err="failed to get container status \"081cd735a10af86c406dbe3edfd6575895f2b2247ce78d6a63233522af925a42\": rpc error: code = NotFound desc = could not find container \"081cd735a10af86c406dbe3edfd6575895f2b2247ce78d6a63233522af925a42\": container with ID starting with 081cd735a10af86c406dbe3edfd6575895f2b2247ce78d6a63233522af925a42 not found: ID does not exist"
Oct 30 01:54:23.326687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.326667    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"]
Oct 30 01:54:23.330303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.330285    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0e1a98-67f8ffd95d-spvkl"]
Oct 30 01:54:23.346212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.346188    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47slg\" (UniqueName: \"kubernetes.io/projected/d36566e9-c18e-4026-b593-467e3491e95a-kube-api-access-47slg\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:23.346212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.346212    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36566e9-c18e-4026-b593-467e3491e95a-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:23.346395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.346224    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpxxk\" (UniqueName: \"kubernetes.io/projected/aaf97e80-fd90-4db8-84a9-872db881b7c7-kube-api-access-bpxxk\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:23.346395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.346237    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf97e80-fd90-4db8-84a9-872db881b7c7-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:23.360381 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.360319    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"]
Oct 30 01:54:23.363402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.363380    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0e1a98-7fcb95fc6c-dhkfl"]
Oct 30 01:54:23.424074 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.424040    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8" path="/var/lib/kubelet/pods/a4ecd6b8-4c3a-4b60-88cc-190c50de0cd8/volumes"
Oct 30 01:54:23.424520 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.424504    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf97e80-fd90-4db8-84a9-872db881b7c7" path="/var/lib/kubelet/pods/aaf97e80-fd90-4db8-84a9-872db881b7c7/volumes"
Oct 30 01:54:23.424910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:23.424895    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d36566e9-c18e-4026-b593-467e3491e95a" path="/var/lib/kubelet/pods/d36566e9-c18e-4026-b593-467e3491e95a/volumes"
Oct 30 01:54:24.346459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:24.346425    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"]
Oct 30 01:54:24.355353 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:24.355327    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-ae1820-bqrtf"]
Oct 30 01:54:25.421977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:25.421947    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6146564-9202-4acd-a3cc-8d6d4c1efbf6" path="/var/lib/kubelet/pods/a6146564-9202-4acd-a3cc-8d6d4c1efbf6/volumes"
Oct 30 01:54:25.609631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:25.609580    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"
Oct 30 01:54:25.672083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:25.672059    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18c8b78c-9e58-4f50-90c6-16684bfe274a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c8b78c-9e58-4f50-90c6-16684bfe274a-plugins-config-volume\") pod \"18c8b78c-9e58-4f50-90c6-16684bfe274a\" (UID: \"18c8b78c-9e58-4f50-90c6-16684bfe274a\") "
Oct 30 01:54:25.672248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:25.672177    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c8b78c-9e58-4f50-90c6-16684bfe274a-kube-api-access-xxm9v\" (UniqueName: \"kubernetes.io/projected/18c8b78c-9e58-4f50-90c6-16684bfe274a-kube-api-access-xxm9v\") pod \"18c8b78c-9e58-4f50-90c6-16684bfe274a\" (UID: \"18c8b78c-9e58-4f50-90c6-16684bfe274a\") "
Oct 30 01:54:25.672536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:25.672507    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c8b78c-9e58-4f50-90c6-16684bfe274a-plugins-config-volume" pod "18c8b78c-9e58-4f50-90c6-16684bfe274a" (UID: "18c8b78c-9e58-4f50-90c6-16684bfe274a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:25.672751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:25.672735    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18c8b78c-9e58-4f50-90c6-16684bfe274a-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:25.674425 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:25.674393    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c8b78c-9e58-4f50-90c6-16684bfe274a-kube-api-access-xxm9v" pod "18c8b78c-9e58-4f50-90c6-16684bfe274a" (UID: "18c8b78c-9e58-4f50-90c6-16684bfe274a"). InnerVolumeSpecName "kube-api-access-xxm9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:25.774112 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:25.774082    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxm9v\" (UniqueName: \"kubernetes.io/projected/18c8b78c-9e58-4f50-90c6-16684bfe274a-kube-api-access-xxm9v\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:26.305656 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:26.305623    4425 generic.go:356] "Generic (PLEG): container finished" podID="18c8b78c-9e58-4f50-90c6-16684bfe274a" containerID="f116d49e73054fd253d98977a9e07d6549e4724014e38060be92be1ef212f5f1" exitCode=0
Oct 30 01:54:26.305882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:26.305666    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"
Oct 30 01:54:26.305882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:26.305706    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n" event={"ID":"18c8b78c-9e58-4f50-90c6-16684bfe274a","Type":"ContainerDied","Data":"f116d49e73054fd253d98977a9e07d6549e4724014e38060be92be1ef212f5f1"}
Oct 30 01:54:26.305882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:26.305762    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n" event={"ID":"18c8b78c-9e58-4f50-90c6-16684bfe274a","Type":"ContainerDied","Data":"c1e9a8b152fba62b9fe29b114601bfa0ff3f26b4d6fa0541ee8b1668d48a7e4c"}
Oct 30 01:54:26.305882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:26.305792    4425 scope.go:122] "RemoveContainer" containerID="f116d49e73054fd253d98977a9e07d6549e4724014e38060be92be1ef212f5f1"
Oct 30 01:54:26.316204 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:26.316188    4425 scope.go:122] "RemoveContainer" containerID="f116d49e73054fd253d98977a9e07d6549e4724014e38060be92be1ef212f5f1"
Oct 30 01:54:26.316470 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:54:26.316449    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f116d49e73054fd253d98977a9e07d6549e4724014e38060be92be1ef212f5f1\": container with ID starting with f116d49e73054fd253d98977a9e07d6549e4724014e38060be92be1ef212f5f1 not found: ID does not exist" containerID="f116d49e73054fd253d98977a9e07d6549e4724014e38060be92be1ef212f5f1"
Oct 30 01:54:26.316522 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:26.316480    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f116d49e73054fd253d98977a9e07d6549e4724014e38060be92be1ef212f5f1"} err="failed to get container status \"f116d49e73054fd253d98977a9e07d6549e4724014e38060be92be1ef212f5f1\": rpc error: code = NotFound desc = could not find container \"f116d49e73054fd253d98977a9e07d6549e4724014e38060be92be1ef212f5f1\": container with ID starting with f116d49e73054fd253d98977a9e07d6549e4724014e38060be92be1ef212f5f1 not found: ID does not exist"
Oct 30 01:54:26.354039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:26.354010    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"]
Oct 30 01:54:26.362920 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:26.362897    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-35042a-5445b75b55-8m76n"]
Oct 30 01:54:26.878122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:26.878080    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw" podUID="4d3d278e-ecd0-46ee-a203-33003899a369" containerName="main" containerID="cri-o://7fd9c865f2a139953b4820d1eae69590fde43ff3c626d9686578af751a1848a8" gracePeriod=45
Oct 30 01:54:27.147746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.147721    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 01:54:27.290086 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.290055    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-model-cache\") pod \"4d3d278e-ecd0-46ee-a203-33003899a369\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") "
Oct 30 01:54:27.290311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.290192    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d3d278e-ecd0-46ee-a203-33003899a369-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3d278e-ecd0-46ee-a203-33003899a369-tls-certs\") pod \"4d3d278e-ecd0-46ee-a203-33003899a369\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") "
Oct 30 01:54:27.290311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.290299    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-tmp-dir\") pod \"4d3d278e-ecd0-46ee-a203-33003899a369\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") "
Oct 30 01:54:27.290437 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.290342    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-dshm\") pod \"4d3d278e-ecd0-46ee-a203-33003899a369\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") "
Oct 30 01:54:27.290437 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.290379    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-model-cache" pod "4d3d278e-ecd0-46ee-a203-33003899a369" (UID: "4d3d278e-ecd0-46ee-a203-33003899a369"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:27.290437 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.290386    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-home\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-home\") pod \"4d3d278e-ecd0-46ee-a203-33003899a369\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") "
Oct 30 01:54:27.290612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.290501    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3d278e-ecd0-46ee-a203-33003899a369-kube-api-access-5pkww\" (UniqueName: \"kubernetes.io/projected/4d3d278e-ecd0-46ee-a203-33003899a369-kube-api-access-5pkww\") pod \"4d3d278e-ecd0-46ee-a203-33003899a369\" (UID: \"4d3d278e-ecd0-46ee-a203-33003899a369\") "
Oct 30 01:54:27.290612 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.290575    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-tmp-dir" pod "4d3d278e-ecd0-46ee-a203-33003899a369" (UID: "4d3d278e-ecd0-46ee-a203-33003899a369"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:27.290815 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.290788    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-home" pod "4d3d278e-ecd0-46ee-a203-33003899a369" (UID: "4d3d278e-ecd0-46ee-a203-33003899a369"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:27.290996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.290978    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.291060 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.291001    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.291060 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.291017    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.292489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.292469    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d3d278e-ecd0-46ee-a203-33003899a369-tls-certs" pod "4d3d278e-ecd0-46ee-a203-33003899a369" (UID: "4d3d278e-ecd0-46ee-a203-33003899a369"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:54:27.292599 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.292550    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3d278e-ecd0-46ee-a203-33003899a369-kube-api-access-5pkww" pod "4d3d278e-ecd0-46ee-a203-33003899a369" (UID: "4d3d278e-ecd0-46ee-a203-33003899a369"). InnerVolumeSpecName "kube-api-access-5pkww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:27.292668 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.292606    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-dshm" pod "4d3d278e-ecd0-46ee-a203-33003899a369" (UID: "4d3d278e-ecd0-46ee-a203-33003899a369"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:54:27.325957 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.318987    4425 generic.go:356] "Generic (PLEG): container finished" podID="4d3d278e-ecd0-46ee-a203-33003899a369" containerID="7fd9c865f2a139953b4820d1eae69590fde43ff3c626d9686578af751a1848a8" exitCode=0
Oct 30 01:54:27.325957 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.319095    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw" event={"ID":"4d3d278e-ecd0-46ee-a203-33003899a369","Type":"ContainerDied","Data":"7fd9c865f2a139953b4820d1eae69590fde43ff3c626d9686578af751a1848a8"}
Oct 30 01:54:27.325957 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.319142    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw" event={"ID":"4d3d278e-ecd0-46ee-a203-33003899a369","Type":"ContainerDied","Data":"dd678f60e3d3bb88413c3e7f17b8b6ceb30d54d58f64d89e0448ebb26eb7fb66"}
Oct 30 01:54:27.325957 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.319183    4425 scope.go:122] "RemoveContainer" containerID="7fd9c865f2a139953b4820d1eae69590fde43ff3c626d9686578af751a1848a8"
Oct 30 01:54:27.325957 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.319370    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"
Oct 30 01:54:27.332326 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.332307    4425 scope.go:122] "RemoveContainer" containerID="7fd9c865f2a139953b4820d1eae69590fde43ff3c626d9686578af751a1848a8"
Oct 30 01:54:27.332617 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:54:27.332596    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd9c865f2a139953b4820d1eae69590fde43ff3c626d9686578af751a1848a8\": container with ID starting with 7fd9c865f2a139953b4820d1eae69590fde43ff3c626d9686578af751a1848a8 not found: ID does not exist" containerID="7fd9c865f2a139953b4820d1eae69590fde43ff3c626d9686578af751a1848a8"
Oct 30 01:54:27.332661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.332630    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd9c865f2a139953b4820d1eae69590fde43ff3c626d9686578af751a1848a8"} err="failed to get container status \"7fd9c865f2a139953b4820d1eae69590fde43ff3c626d9686578af751a1848a8\": rpc error: code = NotFound desc = could not find container \"7fd9c865f2a139953b4820d1eae69590fde43ff3c626d9686578af751a1848a8\": container with ID starting with 7fd9c865f2a139953b4820d1eae69590fde43ff3c626d9686578af751a1848a8 not found: ID does not exist"
Oct 30 01:54:27.369476 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.369437    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"]
Oct 30 01:54:27.374064 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.374041    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-39b241/simulator-w3-39b241-kserve-86496dd68c-r5xpw"]
Oct 30 01:54:27.392103 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.392048    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d3d278e-ecd0-46ee-a203-33003899a369-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.392103 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.392074    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pkww\" (UniqueName: \"kubernetes.io/projected/4d3d278e-ecd0-46ee-a203-33003899a369-kube-api-access-5pkww\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.392103 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.392084    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d3d278e-ecd0-46ee-a203-33003899a369-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:27.424823 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.424789    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18c8b78c-9e58-4f50-90c6-16684bfe274a" path="/var/lib/kubelet/pods/18c8b78c-9e58-4f50-90c6-16684bfe274a/volumes"
Oct 30 01:54:27.425527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.425506    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d3d278e-ecd0-46ee-a203-33003899a369" path="/var/lib/kubelet/pods/4d3d278e-ecd0-46ee-a203-33003899a369/volumes"
Oct 30 01:54:27.798654 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.798618    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d"]
Oct 30 01:54:27.799176 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.799142    4425 cpu_manager.go:414] "containerMap: removing container" podUID="4d3d278e-ecd0-46ee-a203-33003899a369" containerName="main"
Oct 30 01:54:27.799176 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.799167    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3d278e-ecd0-46ee-a203-33003899a369" containerName="main"
Oct 30 01:54:27.799262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.799180    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a7046f8f-887d-42d9-9a30-6e0f369ccf29" containerName="revoke-keys"
Oct 30 01:54:27.799262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.799187    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7046f8f-887d-42d9-9a30-6e0f369ccf29" containerName="revoke-keys"
Oct 30 01:54:27.799262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.799195    4425 cpu_manager.go:414] "containerMap: removing container" podUID="18c8b78c-9e58-4f50-90c6-16684bfe274a" containerName="payload-pre-processing"
Oct 30 01:54:27.799262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.799200    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c8b78c-9e58-4f50-90c6-16684bfe274a" containerName="payload-pre-processing"
Oct 30 01:54:27.799262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.799222    4425 cpu_manager.go:414] "containerMap: removing container" podUID="aaf97e80-fd90-4db8-84a9-872db881b7c7" containerName="payload-pre-processing"
Oct 30 01:54:27.799262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.799227    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf97e80-fd90-4db8-84a9-872db881b7c7" containerName="payload-pre-processing"
Oct 30 01:54:27.799262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.799245    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d36566e9-c18e-4026-b593-467e3491e95a" containerName="payload-processing"
Oct 30 01:54:27.799262 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.799250    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d36566e9-c18e-4026-b593-467e3491e95a" containerName="payload-processing"
Oct 30 01:54:27.799515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.799327    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf97e80-fd90-4db8-84a9-872db881b7c7" containerName="payload-pre-processing"
Oct 30 01:54:27.799515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.799337    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d36566e9-c18e-4026-b593-467e3491e95a" containerName="payload-processing"
Oct 30 01:54:27.799515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.799345    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3d278e-ecd0-46ee-a203-33003899a369" containerName="main"
Oct 30 01:54:27.799515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.799351    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7046f8f-887d-42d9-9a30-6e0f369ccf29" containerName="revoke-keys"
Oct 30 01:54:27.799515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.799357    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c8b78c-9e58-4f50-90c6-16684bfe274a" containerName="payload-pre-processing"
Oct 30 01:54:27.804220 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.804199    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d"
Oct 30 01:54:27.817632 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.817611    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5"]
Oct 30 01:54:27.821935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.821912    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5"
Oct 30 01:54:27.822173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.822140    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:54:27.822402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.822378    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d"]
Oct 30 01:54:27.822679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.822654    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:54:27.825143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.824476    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:54:27.825273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.825110    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:54:27.842953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.842929    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5"]
Oct 30 01:54:27.897723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.897693    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c609f8ad-3635-43a6-966d-27a69e356480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f6db33c-lqvw5\" (UID: \"c609f8ad-3635-43a6-966d-27a69e356480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5"
Oct 30 01:54:27.897723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.897727    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9411c92b-9579-4916-b415-95c0ceb4bb3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f7125d8-2v28d\" (UID: \"9411c92b-9579-4916-b415-95c0ceb4bb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d"
Oct 30 01:54:27.898172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.897744    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjwc6\" (UniqueName: \"kubernetes.io/projected/c609f8ad-3635-43a6-966d-27a69e356480-kube-api-access-sjwc6\") pod \"maas-api-revoke-sub-1f6db33c-lqvw5\" (UID: \"c609f8ad-3635-43a6-966d-27a69e356480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5"
Oct 30 01:54:27.898172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.897873    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb5js\" (UniqueName: \"kubernetes.io/projected/9411c92b-9579-4916-b415-95c0ceb4bb3b-kube-api-access-fb5js\") pod \"maas-api-revoke-sub-8f7125d8-2v28d\" (UID: \"9411c92b-9579-4916-b415-95c0ceb4bb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d"
Oct 30 01:54:27.898172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.897945    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c609f8ad-3635-43a6-966d-27a69e356480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f6db33c-lqvw5\" (UID: \"c609f8ad-3635-43a6-966d-27a69e356480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5"
Oct 30 01:54:27.898172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.898074    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9411c92b-9579-4916-b415-95c0ceb4bb3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f7125d8-2v28d\" (UID: \"9411c92b-9579-4916-b415-95c0ceb4bb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d"
Oct 30 01:54:27.999655 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.999623    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9411c92b-9579-4916-b415-95c0ceb4bb3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f7125d8-2v28d\" (UID: \"9411c92b-9579-4916-b415-95c0ceb4bb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d"
Oct 30 01:54:27.999818 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.999717    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c609f8ad-3635-43a6-966d-27a69e356480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f6db33c-lqvw5\" (UID: \"c609f8ad-3635-43a6-966d-27a69e356480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5"
Oct 30 01:54:27.999818 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.999748    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9411c92b-9579-4916-b415-95c0ceb4bb3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f7125d8-2v28d\" (UID: \"9411c92b-9579-4916-b415-95c0ceb4bb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d"
Oct 30 01:54:27.999818 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.999768    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjwc6\" (UniqueName: \"kubernetes.io/projected/c609f8ad-3635-43a6-966d-27a69e356480-kube-api-access-sjwc6\") pod \"maas-api-revoke-sub-1f6db33c-lqvw5\" (UID: \"c609f8ad-3635-43a6-966d-27a69e356480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5"
Oct 30 01:54:28.000023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.999873    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb5js\" (UniqueName: \"kubernetes.io/projected/9411c92b-9579-4916-b415-95c0ceb4bb3b-kube-api-access-fb5js\") pod \"maas-api-revoke-sub-8f7125d8-2v28d\" (UID: \"9411c92b-9579-4916-b415-95c0ceb4bb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d"
Oct 30 01:54:28.000023 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:27.999942    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c609f8ad-3635-43a6-966d-27a69e356480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f6db33c-lqvw5\" (UID: \"c609f8ad-3635-43a6-966d-27a69e356480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5"
Oct 30 01:54:28.000606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.000577    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9411c92b-9579-4916-b415-95c0ceb4bb3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f7125d8-2v28d\" (UID: \"9411c92b-9579-4916-b415-95c0ceb4bb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d"
Oct 30 01:54:28.000722 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.000691    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c609f8ad-3635-43a6-966d-27a69e356480-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f6db33c-lqvw5\" (UID: \"c609f8ad-3635-43a6-966d-27a69e356480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5"
Oct 30 01:54:28.002072 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.002051    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9411c92b-9579-4916-b415-95c0ceb4bb3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f7125d8-2v28d\" (UID: \"9411c92b-9579-4916-b415-95c0ceb4bb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d"
Oct 30 01:54:28.002245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.002225    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c609f8ad-3635-43a6-966d-27a69e356480-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f6db33c-lqvw5\" (UID: \"c609f8ad-3635-43a6-966d-27a69e356480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5"
Oct 30 01:54:28.021214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.021188    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjwc6\" (UniqueName: \"kubernetes.io/projected/c609f8ad-3635-43a6-966d-27a69e356480-kube-api-access-sjwc6\") pod \"maas-api-revoke-sub-1f6db33c-lqvw5\" (UID: \"c609f8ad-3635-43a6-966d-27a69e356480\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5"
Oct 30 01:54:28.022035 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.022018    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb5js\" (UniqueName: \"kubernetes.io/projected/9411c92b-9579-4916-b415-95c0ceb4bb3b-kube-api-access-fb5js\") pod \"maas-api-revoke-sub-8f7125d8-2v28d\" (UID: \"9411c92b-9579-4916-b415-95c0ceb4bb3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d"
Oct 30 01:54:28.098943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.098877    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"]
Oct 30 01:54:28.104024 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.104007    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"
Oct 30 01:54:28.117845 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.117821    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d"
Oct 30 01:54:28.123673 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.123645    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"]
Oct 30 01:54:28.137350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.137325    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5"
Oct 30 01:54:28.202529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.202480    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf4fr\" (UniqueName: \"kubernetes.io/projected/4e013743-4770-4284-a5d4-2011effd9a1c-kube-api-access-kf4fr\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks\" (UID: \"4e013743-4770-4284-a5d4-2011effd9a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"
Oct 30 01:54:28.202625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.202555    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e013743-4770-4284-a5d4-2011effd9a1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks\" (UID: \"4e013743-4770-4284-a5d4-2011effd9a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"
Oct 30 01:54:28.202625 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.202592    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e013743-4770-4284-a5d4-2011effd9a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks\" (UID: \"4e013743-4770-4284-a5d4-2011effd9a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"
Oct 30 01:54:28.303917 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.303879    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf4fr\" (UniqueName: \"kubernetes.io/projected/4e013743-4770-4284-a5d4-2011effd9a1c-kube-api-access-kf4fr\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks\" (UID: \"4e013743-4770-4284-a5d4-2011effd9a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"
Oct 30 01:54:28.304137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.304047    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e013743-4770-4284-a5d4-2011effd9a1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks\" (UID: \"4e013743-4770-4284-a5d4-2011effd9a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"
Oct 30 01:54:28.304137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.304098    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e013743-4770-4284-a5d4-2011effd9a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks\" (UID: \"4e013743-4770-4284-a5d4-2011effd9a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"
Oct 30 01:54:28.304907 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.304883    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e013743-4770-4284-a5d4-2011effd9a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks\" (UID: \"4e013743-4770-4284-a5d4-2011effd9a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"
Oct 30 01:54:28.306531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.306508    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e013743-4770-4284-a5d4-2011effd9a1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks\" (UID: \"4e013743-4770-4284-a5d4-2011effd9a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"
Oct 30 01:54:28.324727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.324705    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf4fr\" (UniqueName: \"kubernetes.io/projected/4e013743-4770-4284-a5d4-2011effd9a1c-kube-api-access-kf4fr\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks\" (UID: \"4e013743-4770-4284-a5d4-2011effd9a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"
Oct 30 01:54:28.418901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.418840    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"
Oct 30 01:54:28.435567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.435541    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d"]
Oct 30 01:54:28.437447 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:54:28.437421    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9411c92b_9579_4916_b415_95c0ceb4bb3b.slice/crio-a0c91064d629cfabab5983cf6d6128f6a5d87902e3efca6e2a9e842584d5c05b WatchSource:0}: Error finding container a0c91064d629cfabab5983cf6d6128f6a5d87902e3efca6e2a9e842584d5c05b: Status 404 returned error can't find the container with id a0c91064d629cfabab5983cf6d6128f6a5d87902e3efca6e2a9e842584d5c05b
Oct 30 01:54:28.448284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.448258    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5"]
Oct 30 01:54:28.450119 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:54:28.450099    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc609f8ad_3635_43a6_966d_27a69e356480.slice/crio-06654893b1ced18be9e52f99d4898eaaf532aefa3a05a665b37fefb0c0a0e6b3 WatchSource:0}: Error finding container 06654893b1ced18be9e52f99d4898eaaf532aefa3a05a665b37fefb0c0a0e6b3: Status 404 returned error can't find the container with id 06654893b1ced18be9e52f99d4898eaaf532aefa3a05a665b37fefb0c0a0e6b3
Oct 30 01:54:28.731773 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:28.731749    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"]
Oct 30 01:54:28.732656 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:54:28.732633    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e013743_4770_4284_a5d4_2011effd9a1c.slice/crio-57be974585f583ec2ba6229430a310f32dcb0d5240caa0a80f62ec146b536338 WatchSource:0}: Error finding container 57be974585f583ec2ba6229430a310f32dcb0d5240caa0a80f62ec146b536338: Status 404 returned error can't find the container with id 57be974585f583ec2ba6229430a310f32dcb0d5240caa0a80f62ec146b536338
Oct 30 01:54:29.339828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:29.339794    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5" event={"ID":"c609f8ad-3635-43a6-966d-27a69e356480","Type":"ContainerStarted","Data":"54a57c2d8ea3ac1860aafee37311c6ee5d2e8b57dba15646ced4e08bfe44c090"}
Oct 30 01:54:29.340319 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:29.339851    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5" event={"ID":"c609f8ad-3635-43a6-966d-27a69e356480","Type":"ContainerStarted","Data":"06654893b1ced18be9e52f99d4898eaaf532aefa3a05a665b37fefb0c0a0e6b3"}
Oct 30 01:54:29.341751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:29.341713    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks" event={"ID":"4e013743-4770-4284-a5d4-2011effd9a1c","Type":"ContainerStarted","Data":"4d41bec64f220055067bad4ba7d0a255dfe4e28d4c16e3b12ffe28bf9d3e8649"}
Oct 30 01:54:29.341892 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:29.341785    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks" event={"ID":"4e013743-4770-4284-a5d4-2011effd9a1c","Type":"ContainerStarted","Data":"57be974585f583ec2ba6229430a310f32dcb0d5240caa0a80f62ec146b536338"}
Oct 30 01:54:29.343535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:29.343513    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d" event={"ID":"9411c92b-9579-4916-b415-95c0ceb4bb3b","Type":"ContainerStarted","Data":"591ca02d8aada6389111efb998faecbf380e6588da623281cbf150f3446258d0"}
Oct 30 01:54:29.343663 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:29.343560    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d" event={"ID":"9411c92b-9579-4916-b415-95c0ceb4bb3b","Type":"ContainerStarted","Data":"a0c91064d629cfabab5983cf6d6128f6a5d87902e3efca6e2a9e842584d5c05b"}
Oct 30 01:54:29.377554 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:29.377511    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5" podStartSLOduration=2.377497986 podStartE2EDuration="2.377497986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:29.374867552 +0000 UTC m=+4378.124522857" watchObservedRunningTime="2026-10-30 01:54:29.377497986 +0000 UTC m=+4378.127153293"
Oct 30 01:54:29.414757 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:29.414713    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks" podStartSLOduration=1.414702166 podStartE2EDuration="1.414702166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:29.411150558 +0000 UTC m=+4378.160805876" watchObservedRunningTime="2026-10-30 01:54:29.414702166 +0000 UTC m=+4378.164357473"
Oct 30 01:54:29.449110 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:29.449068    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d" podStartSLOduration=2.449055755 podStartE2EDuration="2.449055755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:29.446288107 +0000 UTC m=+4378.195943417" watchObservedRunningTime="2026-10-30 01:54:29.449055755 +0000 UTC m=+4378.198711063"
Oct 30 01:54:30.353124 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:30.353076    4425 generic.go:356] "Generic (PLEG): container finished" podID="c609f8ad-3635-43a6-966d-27a69e356480" containerID="54a57c2d8ea3ac1860aafee37311c6ee5d2e8b57dba15646ced4e08bfe44c090" exitCode=0
Oct 30 01:54:30.353589 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:30.353198    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5" event={"ID":"c609f8ad-3635-43a6-966d-27a69e356480","Type":"ContainerDied","Data":"54a57c2d8ea3ac1860aafee37311c6ee5d2e8b57dba15646ced4e08bfe44c090"}
Oct 30 01:54:30.355188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:30.355143    4425 generic.go:356] "Generic (PLEG): container finished" podID="4e013743-4770-4284-a5d4-2011effd9a1c" containerID="4d41bec64f220055067bad4ba7d0a255dfe4e28d4c16e3b12ffe28bf9d3e8649" exitCode=0
Oct 30 01:54:30.355344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:30.355186    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks" event={"ID":"4e013743-4770-4284-a5d4-2011effd9a1c","Type":"ContainerDied","Data":"4d41bec64f220055067bad4ba7d0a255dfe4e28d4c16e3b12ffe28bf9d3e8649"}
Oct 30 01:54:30.357678 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:30.357658    4425 generic.go:356] "Generic (PLEG): container finished" podID="9411c92b-9579-4916-b415-95c0ceb4bb3b" containerID="591ca02d8aada6389111efb998faecbf380e6588da623281cbf150f3446258d0" exitCode=0
Oct 30 01:54:30.357821 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:30.357685    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d" event={"ID":"9411c92b-9579-4916-b415-95c0ceb4bb3b","Type":"ContainerDied","Data":"591ca02d8aada6389111efb998faecbf380e6588da623281cbf150f3446258d0"}
Oct 30 01:54:30.964199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:30.964145    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"]
Oct 30 01:54:30.970639 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:30.969984    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"
Oct 30 01:54:30.989415 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:30.989381    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"]
Oct 30 01:54:31.040302 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.040253    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a68aab-1017-46ff-857f-f465634b7f02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48\" (UID: \"54a68aab-1017-46ff-857f-f465634b7f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"
Oct 30 01:54:31.040467 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.040416    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6wft\" (UniqueName: \"kubernetes.io/projected/54a68aab-1017-46ff-857f-f465634b7f02-kube-api-access-h6wft\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48\" (UID: \"54a68aab-1017-46ff-857f-f465634b7f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"
Oct 30 01:54:31.040559 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.040540    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a68aab-1017-46ff-857f-f465634b7f02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48\" (UID: \"54a68aab-1017-46ff-857f-f465634b7f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"
Oct 30 01:54:31.135782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.135758    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"
Oct 30 01:54:31.138836 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.138815    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"
Oct 30 01:54:31.141371 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.141345    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a68aab-1017-46ff-857f-f465634b7f02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48\" (UID: \"54a68aab-1017-46ff-857f-f465634b7f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"
Oct 30 01:54:31.141454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.141392    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6wft\" (UniqueName: \"kubernetes.io/projected/54a68aab-1017-46ff-857f-f465634b7f02-kube-api-access-h6wft\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48\" (UID: \"54a68aab-1017-46ff-857f-f465634b7f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"
Oct 30 01:54:31.141503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.141450    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a68aab-1017-46ff-857f-f465634b7f02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48\" (UID: \"54a68aab-1017-46ff-857f-f465634b7f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"
Oct 30 01:54:31.142353 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.142336    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a68aab-1017-46ff-857f-f465634b7f02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48\" (UID: \"54a68aab-1017-46ff-857f-f465634b7f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"
Oct 30 01:54:31.143558 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.143542    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a68aab-1017-46ff-857f-f465634b7f02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48\" (UID: \"54a68aab-1017-46ff-857f-f465634b7f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"
Oct 30 01:54:31.162073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.162055    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6wft\" (UniqueName: \"kubernetes.io/projected/54a68aab-1017-46ff-857f-f465634b7f02-kube-api-access-h6wft\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48\" (UID: \"54a68aab-1017-46ff-857f-f465634b7f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"
Oct 30 01:54:31.242263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.242187    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d78846e-a31f-4911-8c8f-4d0a93748883-kube-api-access-hxhsc\" (UniqueName: \"kubernetes.io/projected/0d78846e-a31f-4911-8c8f-4d0a93748883-kube-api-access-hxhsc\") pod \"0d78846e-a31f-4911-8c8f-4d0a93748883\" (UID: \"0d78846e-a31f-4911-8c8f-4d0a93748883\") "
Oct 30 01:54:31.242405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.242283    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/045aea57-0daf-4b8d-96b7-38073146d269-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045aea57-0daf-4b8d-96b7-38073146d269-plugins-config-volume\") pod \"045aea57-0daf-4b8d-96b7-38073146d269\" (UID: \"045aea57-0daf-4b8d-96b7-38073146d269\") "
Oct 30 01:54:31.242486 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.242412    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d78846e-a31f-4911-8c8f-4d0a93748883-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d78846e-a31f-4911-8c8f-4d0a93748883-plugins-config-volume\") pod \"0d78846e-a31f-4911-8c8f-4d0a93748883\" (UID: \"0d78846e-a31f-4911-8c8f-4d0a93748883\") "
Oct 30 01:54:31.242553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.242485    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045aea57-0daf-4b8d-96b7-38073146d269-kube-api-access-g72s6\" (UniqueName: \"kubernetes.io/projected/045aea57-0daf-4b8d-96b7-38073146d269-kube-api-access-g72s6\") pod \"045aea57-0daf-4b8d-96b7-38073146d269\" (UID: \"045aea57-0daf-4b8d-96b7-38073146d269\") "
Oct 30 01:54:31.242721 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.242698    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045aea57-0daf-4b8d-96b7-38073146d269-plugins-config-volume" pod "045aea57-0daf-4b8d-96b7-38073146d269" (UID: "045aea57-0daf-4b8d-96b7-38073146d269"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:31.243184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.243135    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045aea57-0daf-4b8d-96b7-38073146d269-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:31.243523 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.243483    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d78846e-a31f-4911-8c8f-4d0a93748883-plugins-config-volume" pod "0d78846e-a31f-4911-8c8f-4d0a93748883" (UID: "0d78846e-a31f-4911-8c8f-4d0a93748883"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:31.245058 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.245028    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d78846e-a31f-4911-8c8f-4d0a93748883-kube-api-access-hxhsc" pod "0d78846e-a31f-4911-8c8f-4d0a93748883" (UID: "0d78846e-a31f-4911-8c8f-4d0a93748883"). InnerVolumeSpecName "kube-api-access-hxhsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:31.245228 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.245207    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045aea57-0daf-4b8d-96b7-38073146d269-kube-api-access-g72s6" pod "045aea57-0daf-4b8d-96b7-38073146d269" (UID: "045aea57-0daf-4b8d-96b7-38073146d269"). InnerVolumeSpecName "kube-api-access-g72s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:31.288373 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.288349    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"
Oct 30 01:54:31.345304 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.345272    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d78846e-a31f-4911-8c8f-4d0a93748883-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:31.345438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.345306    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g72s6\" (UniqueName: \"kubernetes.io/projected/045aea57-0daf-4b8d-96b7-38073146d269-kube-api-access-g72s6\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:31.345438 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.345322    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxhsc\" (UniqueName: \"kubernetes.io/projected/0d78846e-a31f-4911-8c8f-4d0a93748883-kube-api-access-hxhsc\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:31.372571 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.372541    4425 generic.go:356] "Generic (PLEG): container finished" podID="0d78846e-a31f-4911-8c8f-4d0a93748883" containerID="2c927f7a0cf34a53ae8717e0de68fea2f4a1c04a13c3ff46781a2a5353dcb880" exitCode=0
Oct 30 01:54:31.372981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.372582    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"
Oct 30 01:54:31.372981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.372623    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz" event={"ID":"0d78846e-a31f-4911-8c8f-4d0a93748883","Type":"ContainerDied","Data":"2c927f7a0cf34a53ae8717e0de68fea2f4a1c04a13c3ff46781a2a5353dcb880"}
Oct 30 01:54:31.372981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.372679    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz" event={"ID":"0d78846e-a31f-4911-8c8f-4d0a93748883","Type":"ContainerDied","Data":"7e3a885fac8827b25fe449f49d96f6625afb2a8d918d563dacb0629091b1ee51"}
Oct 30 01:54:31.372981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.372707    4425 scope.go:122] "RemoveContainer" containerID="2c927f7a0cf34a53ae8717e0de68fea2f4a1c04a13c3ff46781a2a5353dcb880"
Oct 30 01:54:31.379672 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.379647    4425 generic.go:356] "Generic (PLEG): container finished" podID="045aea57-0daf-4b8d-96b7-38073146d269" containerID="e4fc60add2e63e88d9bef7b9db6cb1717987420d951212ef50d3c16483b2c3a7" exitCode=0
Oct 30 01:54:31.379806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.379696    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"
Oct 30 01:54:31.379806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.379735    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg" event={"ID":"045aea57-0daf-4b8d-96b7-38073146d269","Type":"ContainerDied","Data":"e4fc60add2e63e88d9bef7b9db6cb1717987420d951212ef50d3c16483b2c3a7"}
Oct 30 01:54:31.379806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.379775    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg" event={"ID":"045aea57-0daf-4b8d-96b7-38073146d269","Type":"ContainerDied","Data":"7445bd0cf6532f725a2c2d09da7c7012e0fb557f8e922bcbd9d1efa5b5d14ef0"}
Oct 30 01:54:31.406911 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.405228    4425 scope.go:122] "RemoveContainer" containerID="2c927f7a0cf34a53ae8717e0de68fea2f4a1c04a13c3ff46781a2a5353dcb880"
Oct 30 01:54:31.407865 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:54:31.407680    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c927f7a0cf34a53ae8717e0de68fea2f4a1c04a13c3ff46781a2a5353dcb880\": container with ID starting with 2c927f7a0cf34a53ae8717e0de68fea2f4a1c04a13c3ff46781a2a5353dcb880 not found: ID does not exist" containerID="2c927f7a0cf34a53ae8717e0de68fea2f4a1c04a13c3ff46781a2a5353dcb880"
Oct 30 01:54:31.407865 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.407732    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c927f7a0cf34a53ae8717e0de68fea2f4a1c04a13c3ff46781a2a5353dcb880"} err="failed to get container status \"2c927f7a0cf34a53ae8717e0de68fea2f4a1c04a13c3ff46781a2a5353dcb880\": rpc error: code = NotFound desc = could not find container \"2c927f7a0cf34a53ae8717e0de68fea2f4a1c04a13c3ff46781a2a5353dcb880\": container with ID starting with 2c927f7a0cf34a53ae8717e0de68fea2f4a1c04a13c3ff46781a2a5353dcb880 not found: ID does not exist"
Oct 30 01:54:31.407865 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.407760    4425 scope.go:122] "RemoveContainer" containerID="e4fc60add2e63e88d9bef7b9db6cb1717987420d951212ef50d3c16483b2c3a7"
Oct 30 01:54:31.425890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.425866    4425 scope.go:122] "RemoveContainer" containerID="e4fc60add2e63e88d9bef7b9db6cb1717987420d951212ef50d3c16483b2c3a7"
Oct 30 01:54:31.427192 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:54:31.426235    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4fc60add2e63e88d9bef7b9db6cb1717987420d951212ef50d3c16483b2c3a7\": container with ID starting with e4fc60add2e63e88d9bef7b9db6cb1717987420d951212ef50d3c16483b2c3a7 not found: ID does not exist" containerID="e4fc60add2e63e88d9bef7b9db6cb1717987420d951212ef50d3c16483b2c3a7"
Oct 30 01:54:31.427192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.426274    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4fc60add2e63e88d9bef7b9db6cb1717987420d951212ef50d3c16483b2c3a7"} err="failed to get container status \"e4fc60add2e63e88d9bef7b9db6cb1717987420d951212ef50d3c16483b2c3a7\": rpc error: code = NotFound desc = could not find container \"e4fc60add2e63e88d9bef7b9db6cb1717987420d951212ef50d3c16483b2c3a7\": container with ID starting with e4fc60add2e63e88d9bef7b9db6cb1717987420d951212ef50d3c16483b2c3a7 not found: ID does not exist"
Oct 30 01:54:31.456121 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.456091    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"]
Oct 30 01:54:31.460735 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.460707    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ae1820-85757f595-k6hcz"]
Oct 30 01:54:31.494684 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.494442    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"]
Oct 30 01:54:31.499433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.499414    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ae1820-65885bfc4d-4r8tg"]
Oct 30 01:54:31.558844 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.558817    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d"
Oct 30 01:54:31.586306 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.586275    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5"
Oct 30 01:54:31.593716 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.593691    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"]
Oct 30 01:54:31.603134 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:54:31.599454    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a68aab_1017_46ff_857f_f465634b7f02.slice/crio-93dd59cfb247e55e77a6743f519ec0593b2769283f9023cac7fbc17d44662692 WatchSource:0}: Error finding container 93dd59cfb247e55e77a6743f519ec0593b2769283f9023cac7fbc17d44662692: Status 404 returned error can't find the container with id 93dd59cfb247e55e77a6743f519ec0593b2769283f9023cac7fbc17d44662692
Oct 30 01:54:31.641434 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.640678    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"
Oct 30 01:54:31.648123 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.648096    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9411c92b-9579-4916-b415-95c0ceb4bb3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9411c92b-9579-4916-b415-95c0ceb4bb3b-maas-api-openshift-ca\") pod \"9411c92b-9579-4916-b415-95c0ceb4bb3b\" (UID: \"9411c92b-9579-4916-b415-95c0ceb4bb3b\") "
Oct 30 01:54:31.648273 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.648172    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9411c92b-9579-4916-b415-95c0ceb4bb3b-kube-api-access-fb5js\" (UniqueName: \"kubernetes.io/projected/9411c92b-9579-4916-b415-95c0ceb4bb3b-kube-api-access-fb5js\") pod \"9411c92b-9579-4916-b415-95c0ceb4bb3b\" (UID: \"9411c92b-9579-4916-b415-95c0ceb4bb3b\") "
Oct 30 01:54:31.648460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.648435    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9411c92b-9579-4916-b415-95c0ceb4bb3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9411c92b-9579-4916-b415-95c0ceb4bb3b-maas-api-xks-ca\") pod \"9411c92b-9579-4916-b415-95c0ceb4bb3b\" (UID: \"9411c92b-9579-4916-b415-95c0ceb4bb3b\") "
Oct 30 01:54:31.649081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.649055    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9411c92b-9579-4916-b415-95c0ceb4bb3b-maas-api-openshift-ca" pod "9411c92b-9579-4916-b415-95c0ceb4bb3b" (UID: "9411c92b-9579-4916-b415-95c0ceb4bb3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:31.650410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.650367    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9411c92b-9579-4916-b415-95c0ceb4bb3b-kube-api-access-fb5js" pod "9411c92b-9579-4916-b415-95c0ceb4bb3b" (UID: "9411c92b-9579-4916-b415-95c0ceb4bb3b"). InnerVolumeSpecName "kube-api-access-fb5js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:31.650614 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.650587    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9411c92b-9579-4916-b415-95c0ceb4bb3b-maas-api-xks-ca" pod "9411c92b-9579-4916-b415-95c0ceb4bb3b" (UID: "9411c92b-9579-4916-b415-95c0ceb4bb3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:54:31.750333 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.750259    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c609f8ad-3635-43a6-966d-27a69e356480-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c609f8ad-3635-43a6-966d-27a69e356480-maas-api-openshift-ca\") pod \"c609f8ad-3635-43a6-966d-27a69e356480\" (UID: \"c609f8ad-3635-43a6-966d-27a69e356480\") "
Oct 30 01:54:31.750461 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.750354    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e013743-4770-4284-a5d4-2011effd9a1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e013743-4770-4284-a5d4-2011effd9a1c-maas-api-openshift-ca\") pod \"4e013743-4770-4284-a5d4-2011effd9a1c\" (UID: \"4e013743-4770-4284-a5d4-2011effd9a1c\") "
Oct 30 01:54:31.750461 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.750425    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e013743-4770-4284-a5d4-2011effd9a1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e013743-4770-4284-a5d4-2011effd9a1c-maas-api-xks-ca\") pod \"4e013743-4770-4284-a5d4-2011effd9a1c\" (UID: \"4e013743-4770-4284-a5d4-2011effd9a1c\") "
Oct 30 01:54:31.750604 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.750501    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c609f8ad-3635-43a6-966d-27a69e356480-kube-api-access-sjwc6\" (UniqueName: \"kubernetes.io/projected/c609f8ad-3635-43a6-966d-27a69e356480-kube-api-access-sjwc6\") pod \"c609f8ad-3635-43a6-966d-27a69e356480\" (UID: \"c609f8ad-3635-43a6-966d-27a69e356480\") "
Oct 30 01:54:31.750604 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.750558    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c609f8ad-3635-43a6-966d-27a69e356480-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c609f8ad-3635-43a6-966d-27a69e356480-maas-api-xks-ca\") pod \"c609f8ad-3635-43a6-966d-27a69e356480\" (UID: \"c609f8ad-3635-43a6-966d-27a69e356480\") "
Oct 30 01:54:31.750717 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.750602    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c609f8ad-3635-43a6-966d-27a69e356480-maas-api-openshift-ca" pod "c609f8ad-3635-43a6-966d-27a69e356480" (UID: "c609f8ad-3635-43a6-966d-27a69e356480"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:31.750717 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.750675    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e013743-4770-4284-a5d4-2011effd9a1c-maas-api-openshift-ca" pod "4e013743-4770-4284-a5d4-2011effd9a1c" (UID: "4e013743-4770-4284-a5d4-2011effd9a1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:31.750825 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.750738    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e013743-4770-4284-a5d4-2011effd9a1c-kube-api-access-kf4fr\" (UniqueName: \"kubernetes.io/projected/4e013743-4770-4284-a5d4-2011effd9a1c-kube-api-access-kf4fr\") pod \"4e013743-4770-4284-a5d4-2011effd9a1c\" (UID: \"4e013743-4770-4284-a5d4-2011effd9a1c\") "
Oct 30 01:54:31.751174 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.751132    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9411c92b-9579-4916-b415-95c0ceb4bb3b-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:31.751319 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.751185    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb5js\" (UniqueName: \"kubernetes.io/projected/9411c92b-9579-4916-b415-95c0ceb4bb3b-kube-api-access-fb5js\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:31.751319 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.751201    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9411c92b-9579-4916-b415-95c0ceb4bb3b-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:31.751319 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.751212    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c609f8ad-3635-43a6-966d-27a69e356480-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:31.751319 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.751224    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e013743-4770-4284-a5d4-2011effd9a1c-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:31.752728 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.752700    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e013743-4770-4284-a5d4-2011effd9a1c-maas-api-xks-ca" pod "4e013743-4770-4284-a5d4-2011effd9a1c" (UID: "4e013743-4770-4284-a5d4-2011effd9a1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:54:31.752978 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.752952    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c609f8ad-3635-43a6-966d-27a69e356480-maas-api-xks-ca" pod "c609f8ad-3635-43a6-966d-27a69e356480" (UID: "c609f8ad-3635-43a6-966d-27a69e356480"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:54:31.753081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.752976    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c609f8ad-3635-43a6-966d-27a69e356480-kube-api-access-sjwc6" pod "c609f8ad-3635-43a6-966d-27a69e356480" (UID: "c609f8ad-3635-43a6-966d-27a69e356480"). InnerVolumeSpecName "kube-api-access-sjwc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:31.753141 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.753122    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e013743-4770-4284-a5d4-2011effd9a1c-kube-api-access-kf4fr" pod "4e013743-4770-4284-a5d4-2011effd9a1c" (UID: "4e013743-4770-4284-a5d4-2011effd9a1c"). InnerVolumeSpecName "kube-api-access-kf4fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:31.852493 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.852464    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjwc6\" (UniqueName: \"kubernetes.io/projected/c609f8ad-3635-43a6-966d-27a69e356480-kube-api-access-sjwc6\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:31.852493 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.852489    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c609f8ad-3635-43a6-966d-27a69e356480-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:31.852493 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.852500    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf4fr\" (UniqueName: \"kubernetes.io/projected/4e013743-4770-4284-a5d4-2011effd9a1c-kube-api-access-kf4fr\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:31.852760 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:31.852510    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e013743-4770-4284-a5d4-2011effd9a1c-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:32.269636 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:32.269594    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:54:32.388807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:32.388782    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5"
Oct 30 01:54:32.389239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:32.388775    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f6db33c-lqvw5" event={"ID":"c609f8ad-3635-43a6-966d-27a69e356480","Type":"ContainerDied","Data":"06654893b1ced18be9e52f99d4898eaaf532aefa3a05a665b37fefb0c0a0e6b3"}
Oct 30 01:54:32.389239 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:32.388894    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06654893b1ced18be9e52f99d4898eaaf532aefa3a05a665b37fefb0c0a0e6b3"
Oct 30 01:54:32.391439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:32.391420    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"
Oct 30 01:54:32.391580 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:32.391452    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks" event={"ID":"4e013743-4770-4284-a5d4-2011effd9a1c","Type":"ContainerDied","Data":"57be974585f583ec2ba6229430a310f32dcb0d5240caa0a80f62ec146b536338"}
Oct 30 01:54:32.391580 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:32.391494    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57be974585f583ec2ba6229430a310f32dcb0d5240caa0a80f62ec146b536338"
Oct 30 01:54:32.392953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:32.392925    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d"
Oct 30 01:54:32.393048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:32.392950    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f7125d8-2v28d" event={"ID":"9411c92b-9579-4916-b415-95c0ceb4bb3b","Type":"ContainerDied","Data":"a0c91064d629cfabab5983cf6d6128f6a5d87902e3efca6e2a9e842584d5c05b"}
Oct 30 01:54:32.393048 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:32.392996    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0c91064d629cfabab5983cf6d6128f6a5d87902e3efca6e2a9e842584d5c05b"
Oct 30 01:54:32.394752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:32.394727    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48" event={"ID":"54a68aab-1017-46ff-857f-f465634b7f02","Type":"ContainerStarted","Data":"14b20f7ef79c56b30ca93360744eb5b5c5d28bf384d45344c9b8cb5d86aa9528"}
Oct 30 01:54:32.394841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:32.394774    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48" event={"ID":"54a68aab-1017-46ff-857f-f465634b7f02","Type":"ContainerStarted","Data":"93dd59cfb247e55e77a6743f519ec0593b2769283f9023cac7fbc17d44662692"}
Oct 30 01:54:32.432180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:32.432115    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48" podStartSLOduration=2.432103845 podStartE2EDuration="2.432103845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:32.430098821 +0000 UTC m=+4381.179754140" watchObservedRunningTime="2026-10-30 01:54:32.432103845 +0000 UTC m=+4381.181759203"
Oct 30 01:54:33.412582 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.412536    4425 generic.go:356] "Generic (PLEG): container finished" podID="54a68aab-1017-46ff-857f-f465634b7f02" containerID="14b20f7ef79c56b30ca93360744eb5b5c5d28bf384d45344c9b8cb5d86aa9528" exitCode=0
Oct 30 01:54:33.413076 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.412748    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48" event={"ID":"54a68aab-1017-46ff-857f-f465634b7f02","Type":"ContainerDied","Data":"14b20f7ef79c56b30ca93360744eb5b5c5d28bf384d45344c9b8cb5d86aa9528"}
Oct 30 01:54:33.426088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.425477    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="045aea57-0daf-4b8d-96b7-38073146d269" path="/var/lib/kubelet/pods/045aea57-0daf-4b8d-96b7-38073146d269/volumes"
Oct 30 01:54:33.426088 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.425930    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d78846e-a31f-4911-8c8f-4d0a93748883" path="/var/lib/kubelet/pods/0d78846e-a31f-4911-8c8f-4d0a93748883/volumes"
Oct 30 01:54:33.998455 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.998422    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"]
Oct 30 01:54:33.998958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.998941    4425 cpu_manager.go:414] "containerMap: removing container" podUID="4e013743-4770-4284-a5d4-2011effd9a1c" containerName="revoke-keys"
Oct 30 01:54:33.998958 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.998955    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e013743-4770-4284-a5d4-2011effd9a1c" containerName="revoke-keys"
Oct 30 01:54:33.999106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.998973    4425 cpu_manager.go:414] "containerMap: removing container" podUID="9411c92b-9579-4916-b415-95c0ceb4bb3b" containerName="revoke-keys"
Oct 30 01:54:33.999106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.998979    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9411c92b-9579-4916-b415-95c0ceb4bb3b" containerName="revoke-keys"
Oct 30 01:54:33.999106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.998991    4425 cpu_manager.go:414] "containerMap: removing container" podUID="0d78846e-a31f-4911-8c8f-4d0a93748883" containerName="payload-pre-processing"
Oct 30 01:54:33.999106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.998996    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d78846e-a31f-4911-8c8f-4d0a93748883" containerName="payload-pre-processing"
Oct 30 01:54:33.999106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.999005    4425 cpu_manager.go:414] "containerMap: removing container" podUID="045aea57-0daf-4b8d-96b7-38073146d269" containerName="payload-processing"
Oct 30 01:54:33.999106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.999010    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045aea57-0daf-4b8d-96b7-38073146d269" containerName="payload-processing"
Oct 30 01:54:33.999106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.999031    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c609f8ad-3635-43a6-966d-27a69e356480" containerName="revoke-keys"
Oct 30 01:54:33.999106 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.999036    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c609f8ad-3635-43a6-966d-27a69e356480" containerName="revoke-keys"
Oct 30 01:54:33.999392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.999113    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e013743-4770-4284-a5d4-2011effd9a1c" containerName="revoke-keys"
Oct 30 01:54:33.999392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.999124    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="9411c92b-9579-4916-b415-95c0ceb4bb3b" containerName="revoke-keys"
Oct 30 01:54:33.999392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.999130    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d78846e-a31f-4911-8c8f-4d0a93748883" containerName="payload-pre-processing"
Oct 30 01:54:33.999392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.999136    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="045aea57-0daf-4b8d-96b7-38073146d269" containerName="payload-processing"
Oct 30 01:54:33.999392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:33.999145    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c609f8ad-3635-43a6-966d-27a69e356480" containerName="revoke-keys"
Oct 30 01:54:34.002730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.002714    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"
Oct 30 01:54:34.021731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.021708    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"]
Oct 30 01:54:34.176187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.176142    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a5fb72-0291-4566-9101-4b438be4a2b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj\" (UID: \"57a5fb72-0291-4566-9101-4b438be4a2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"
Oct 30 01:54:34.176386 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.176359    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a5fb72-0291-4566-9101-4b438be4a2b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj\" (UID: \"57a5fb72-0291-4566-9101-4b438be4a2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"
Oct 30 01:54:34.176447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.176405    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4kmq\" (UniqueName: \"kubernetes.io/projected/57a5fb72-0291-4566-9101-4b438be4a2b0-kube-api-access-d4kmq\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj\" (UID: \"57a5fb72-0291-4566-9101-4b438be4a2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"
Oct 30 01:54:34.277529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.277499    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a5fb72-0291-4566-9101-4b438be4a2b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj\" (UID: \"57a5fb72-0291-4566-9101-4b438be4a2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"
Oct 30 01:54:34.277687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.277534    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4kmq\" (UniqueName: \"kubernetes.io/projected/57a5fb72-0291-4566-9101-4b438be4a2b0-kube-api-access-d4kmq\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj\" (UID: \"57a5fb72-0291-4566-9101-4b438be4a2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"
Oct 30 01:54:34.277687 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.277610    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a5fb72-0291-4566-9101-4b438be4a2b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj\" (UID: \"57a5fb72-0291-4566-9101-4b438be4a2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"
Oct 30 01:54:34.278122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.278098    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a5fb72-0291-4566-9101-4b438be4a2b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj\" (UID: \"57a5fb72-0291-4566-9101-4b438be4a2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"
Oct 30 01:54:34.279802 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.279780    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a5fb72-0291-4566-9101-4b438be4a2b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj\" (UID: \"57a5fb72-0291-4566-9101-4b438be4a2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"
Oct 30 01:54:34.298492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.298468    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4kmq\" (UniqueName: \"kubernetes.io/projected/57a5fb72-0291-4566-9101-4b438be4a2b0-kube-api-access-d4kmq\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj\" (UID: \"57a5fb72-0291-4566-9101-4b438be4a2b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"
Oct 30 01:54:34.316042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.316014    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"
Oct 30 01:54:34.571968 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.571932    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"
Oct 30 01:54:34.621460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.621433    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"]
Oct 30 01:54:34.683830 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.683806    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a68aab-1017-46ff-857f-f465634b7f02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a68aab-1017-46ff-857f-f465634b7f02-maas-api-openshift-ca\") pod \"54a68aab-1017-46ff-857f-f465634b7f02\" (UID: \"54a68aab-1017-46ff-857f-f465634b7f02\") "
Oct 30 01:54:34.683966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.683863    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a68aab-1017-46ff-857f-f465634b7f02-kube-api-access-h6wft\" (UniqueName: \"kubernetes.io/projected/54a68aab-1017-46ff-857f-f465634b7f02-kube-api-access-h6wft\") pod \"54a68aab-1017-46ff-857f-f465634b7f02\" (UID: \"54a68aab-1017-46ff-857f-f465634b7f02\") "
Oct 30 01:54:34.683966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.683929    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54a68aab-1017-46ff-857f-f465634b7f02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a68aab-1017-46ff-857f-f465634b7f02-maas-api-xks-ca\") pod \"54a68aab-1017-46ff-857f-f465634b7f02\" (UID: \"54a68aab-1017-46ff-857f-f465634b7f02\") "
Oct 30 01:54:34.684142 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.684119    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a68aab-1017-46ff-857f-f465634b7f02-maas-api-openshift-ca" pod "54a68aab-1017-46ff-857f-f465634b7f02" (UID: "54a68aab-1017-46ff-857f-f465634b7f02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:34.684280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.684260    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a68aab-1017-46ff-857f-f465634b7f02-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:34.685796 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.685775    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a68aab-1017-46ff-857f-f465634b7f02-kube-api-access-h6wft" pod "54a68aab-1017-46ff-857f-f465634b7f02" (UID: "54a68aab-1017-46ff-857f-f465634b7f02"). InnerVolumeSpecName "kube-api-access-h6wft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:34.685890 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.685832    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a68aab-1017-46ff-857f-f465634b7f02-maas-api-xks-ca" pod "54a68aab-1017-46ff-857f-f465634b7f02" (UID: "54a68aab-1017-46ff-857f-f465634b7f02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:54:34.785636 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.785613    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6wft\" (UniqueName: \"kubernetes.io/projected/54a68aab-1017-46ff-857f-f465634b7f02-kube-api-access-h6wft\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:34.785636 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:34.785634    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a68aab-1017-46ff-857f-f465634b7f02-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:35.429489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:35.429460    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"
Oct 30 01:54:35.429640 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:35.429463    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48" event={"ID":"54a68aab-1017-46ff-857f-f465634b7f02","Type":"ContainerDied","Data":"93dd59cfb247e55e77a6743f519ec0593b2769283f9023cac7fbc17d44662692"}
Oct 30 01:54:35.429640 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:35.429581    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93dd59cfb247e55e77a6743f519ec0593b2769283f9023cac7fbc17d44662692"
Oct 30 01:54:35.436378 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:35.436348    4425 generic.go:356] "Generic (PLEG): container finished" podID="57a5fb72-0291-4566-9101-4b438be4a2b0" containerID="b40822fb1fc492d7b94e734965ac7311314977f757287387472d0b619c94ae4a" exitCode=0
Oct 30 01:54:35.436479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:35.436405    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj" event={"ID":"57a5fb72-0291-4566-9101-4b438be4a2b0","Type":"ContainerDied","Data":"b40822fb1fc492d7b94e734965ac7311314977f757287387472d0b619c94ae4a"}
Oct 30 01:54:35.436479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:35.436445    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj" event={"ID":"57a5fb72-0291-4566-9101-4b438be4a2b0","Type":"ContainerStarted","Data":"d9c31fc282cf66289989fda430753504e980f4f3b22e1e0f1760998a595229b4"}
Oct 30 01:54:36.581550 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:36.581526    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"
Oct 30 01:54:36.705972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:36.705902    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57a5fb72-0291-4566-9101-4b438be4a2b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a5fb72-0291-4566-9101-4b438be4a2b0-maas-api-xks-ca\") pod \"57a5fb72-0291-4566-9101-4b438be4a2b0\" (UID: \"57a5fb72-0291-4566-9101-4b438be4a2b0\") "
Oct 30 01:54:36.705972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:36.705939    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a5fb72-0291-4566-9101-4b438be4a2b0-kube-api-access-d4kmq\" (UniqueName: \"kubernetes.io/projected/57a5fb72-0291-4566-9101-4b438be4a2b0-kube-api-access-d4kmq\") pod \"57a5fb72-0291-4566-9101-4b438be4a2b0\" (UID: \"57a5fb72-0291-4566-9101-4b438be4a2b0\") "
Oct 30 01:54:36.706231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:36.705977    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a5fb72-0291-4566-9101-4b438be4a2b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a5fb72-0291-4566-9101-4b438be4a2b0-maas-api-openshift-ca\") pod \"57a5fb72-0291-4566-9101-4b438be4a2b0\" (UID: \"57a5fb72-0291-4566-9101-4b438be4a2b0\") "
Oct 30 01:54:36.706439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:36.706415    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a5fb72-0291-4566-9101-4b438be4a2b0-maas-api-openshift-ca" pod "57a5fb72-0291-4566-9101-4b438be4a2b0" (UID: "57a5fb72-0291-4566-9101-4b438be4a2b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:54:36.707990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:36.707961    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a5fb72-0291-4566-9101-4b438be4a2b0-kube-api-access-d4kmq" pod "57a5fb72-0291-4566-9101-4b438be4a2b0" (UID: "57a5fb72-0291-4566-9101-4b438be4a2b0"). InnerVolumeSpecName "kube-api-access-d4kmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:54:36.708071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:36.708004    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a5fb72-0291-4566-9101-4b438be4a2b0-maas-api-xks-ca" pod "57a5fb72-0291-4566-9101-4b438be4a2b0" (UID: "57a5fb72-0291-4566-9101-4b438be4a2b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:54:36.801369 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:36.801341    4425 scope.go:122] "RemoveContainer" containerID="1a4a9fc0742bec53726fbb7169564e181d028e30ee1eda1a9a9706d0dc00921d"
Oct 30 01:54:36.806946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:36.806916    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a5fb72-0291-4566-9101-4b438be4a2b0-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:36.806946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:36.806944    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a5fb72-0291-4566-9101-4b438be4a2b0-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:36.807074 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:36.806960    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4kmq\" (UniqueName: \"kubernetes.io/projected/57a5fb72-0291-4566-9101-4b438be4a2b0-kube-api-access-d4kmq\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:54:36.811707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:36.811689    4425 scope.go:122] "RemoveContainer" containerID="13ebe953f2986aca078f498c53145780d2b4371047a5ceb834cd085a3a2fbfe3"
Oct 30 01:54:37.451573 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:37.451546    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"
Oct 30 01:54:37.451730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:37.451587    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj" event={"ID":"57a5fb72-0291-4566-9101-4b438be4a2b0","Type":"ContainerDied","Data":"d9c31fc282cf66289989fda430753504e980f4f3b22e1e0f1760998a595229b4"}
Oct 30 01:54:37.451730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:37.451640    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9c31fc282cf66289989fda430753504e980f4f3b22e1e0f1760998a595229b4"
Oct 30 01:54:37.555052 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:54:37.555011    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:54:38.526771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:38.526735    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"]
Oct 30 01:54:38.558304 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:38.558272    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"]
Oct 30 01:54:41.455570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:41.455534    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"]
Oct 30 01:54:43.527792 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:43.527742    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7" podUID="f909610e-c454-4ab6-bdf2-d061ade22e78" containerName="payload-processing" containerID="cri-o://e56f1683037c50bec3d8cc9650dfadd3ec083672e4f60703f8052ed26c89f602" gracePeriod=25
Oct 30 01:54:43.559222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:43.559172    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs" podUID="e47bac65-a69e-45e0-8a8c-f239e14c2e67" containerName="payload-pre-processing" containerID="cri-o://55eac7f9375154924459baf1df17d10ddff844820bb3c4ac0d023af0d8158628" gracePeriod=25
Oct 30 01:54:44.400638 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:44.400605    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"]
Oct 30 01:54:44.431999 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:44.431966    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"]
Oct 30 01:54:46.456576 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:46.456531    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s" podUID="50a5c7e3-26ea-42d6-b760-2ac74b5bdda1" containerName="payload-processing" containerID="cri-o://c2c3e91f9224de206a584ce7e15d5b0de34b92d221f300c56f353dd98d17541c" gracePeriod=25
Oct 30 01:54:46.791715 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:46.791677    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s" podUID="50a5c7e3-26ea-42d6-b760-2ac74b5bdda1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:49.401581 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:49.401526    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x" podUID="3ac3ee9c-d077-44ac-8be5-ed5555679412" containerName="payload-processing" containerID="cri-o://53ef72fdb7e25171fd039f365071887731d6e12a9ddf56266e5a8ab2dff03f9f" gracePeriod=25
Oct 30 01:54:49.432222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:49.432187    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g" podUID="4c46fb47-d71a-4831-bf35-2acc1bb4c935" containerName="payload-pre-processing" containerID="cri-o://7441f9119204455033681f3e0b544d35642fdee66bcf70e330c206b9a16bed77" gracePeriod=25
Oct 30 01:54:49.968688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:49.968642    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g" podUID="4c46fb47-d71a-4831-bf35-2acc1bb4c935" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:52.284367 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:52.284333    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"]
Oct 30 01:54:52.294616 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:52.294588    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-39b241-kcpxj"]
Oct 30 01:54:53.419946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:53.419912    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs" podUID="e47bac65-a69e-45e0-8a8c-f239e14c2e67" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:53.419946 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:53.419934    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7" podUID="f909610e-c454-4ab6-bdf2-d061ade22e78" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:53.422980 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:53.422956    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7046f8f-887d-42d9-9a30-6e0f369ccf29" path="/var/lib/kubelet/pods/a7046f8f-887d-42d9-9a30-6e0f369ccf29/volumes"
Oct 30 01:54:56.426182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:56.426125    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"]
Oct 30 01:54:56.435307 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:56.435283    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-1854c4-dkfks"]
Oct 30 01:54:56.791053 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:56.791012    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s" podUID="50a5c7e3-26ea-42d6-b760-2ac74b5bdda1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:57.422237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:57.422199    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e013743-4770-4284-a5d4-2011effd9a1c" path="/var/lib/kubelet/pods/4e013743-4770-4284-a5d4-2011effd9a1c/volumes"
Oct 30 01:54:58.727078 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:58.727040    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"]
Oct 30 01:54:58.736516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:58.736490    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9da49f-6tr48"]
Oct 30 01:54:58.916005 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:58.915956    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x" podUID="3ac3ee9c-d077-44ac-8be5-ed5555679412" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:54:59.253222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:59.253171    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"]
Oct 30 01:54:59.256216 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:59.256190    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8ae41e-t76bj"]
Oct 30 01:54:59.423886 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:59.423855    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a68aab-1017-46ff-857f-f465634b7f02" path="/var/lib/kubelet/pods/54a68aab-1017-46ff-857f-f465634b7f02/volumes"
Oct 30 01:54:59.424352 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:59.424336    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57a5fb72-0291-4566-9101-4b438be4a2b0" path="/var/lib/kubelet/pods/57a5fb72-0291-4566-9101-4b438be4a2b0/volumes"
Oct 30 01:54:59.947492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:59.947451    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:54:59.968877 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:54:59.968827    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g" podUID="4c46fb47-d71a-4831-bf35-2acc1bb4c935" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:55:03.420727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:03.420681    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs" podUID="e47bac65-a69e-45e0-8a8c-f239e14c2e67" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:55:03.421224 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:03.420726    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7" podUID="f909610e-c454-4ab6-bdf2-d061ade22e78" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:55:03.817689 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:03.817655    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"
Oct 30 01:55:03.854885 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:03.854849    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"
Oct 30 01:55:03.912966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:03.912932    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f909610e-c454-4ab6-bdf2-d061ade22e78-kube-api-access-bgt7j\" (UniqueName: \"kubernetes.io/projected/f909610e-c454-4ab6-bdf2-d061ade22e78-kube-api-access-bgt7j\") pod \"f909610e-c454-4ab6-bdf2-d061ade22e78\" (UID: \"f909610e-c454-4ab6-bdf2-d061ade22e78\") "
Oct 30 01:55:03.913214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:03.913009    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f909610e-c454-4ab6-bdf2-d061ade22e78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f909610e-c454-4ab6-bdf2-d061ade22e78-plugins-config-volume\") pod \"f909610e-c454-4ab6-bdf2-d061ade22e78\" (UID: \"f909610e-c454-4ab6-bdf2-d061ade22e78\") "
Oct 30 01:55:03.913541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:03.913508    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f909610e-c454-4ab6-bdf2-d061ade22e78-plugins-config-volume" pod "f909610e-c454-4ab6-bdf2-d061ade22e78" (UID: "f909610e-c454-4ab6-bdf2-d061ade22e78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:03.915819 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:03.915792    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f909610e-c454-4ab6-bdf2-d061ade22e78-kube-api-access-bgt7j" pod "f909610e-c454-4ab6-bdf2-d061ade22e78" (UID: "f909610e-c454-4ab6-bdf2-d061ade22e78"). InnerVolumeSpecName "kube-api-access-bgt7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:04.013956 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.013928    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e47bac65-a69e-45e0-8a8c-f239e14c2e67-kube-api-access-tw7tp\" (UniqueName: \"kubernetes.io/projected/e47bac65-a69e-45e0-8a8c-f239e14c2e67-kube-api-access-tw7tp\") pod \"e47bac65-a69e-45e0-8a8c-f239e14c2e67\" (UID: \"e47bac65-a69e-45e0-8a8c-f239e14c2e67\") "
Oct 30 01:55:04.014111 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.014051    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e47bac65-a69e-45e0-8a8c-f239e14c2e67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47bac65-a69e-45e0-8a8c-f239e14c2e67-plugins-config-volume\") pod \"e47bac65-a69e-45e0-8a8c-f239e14c2e67\" (UID: \"e47bac65-a69e-45e0-8a8c-f239e14c2e67\") "
Oct 30 01:55:04.014402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.014383    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgt7j\" (UniqueName: \"kubernetes.io/projected/f909610e-c454-4ab6-bdf2-d061ade22e78-kube-api-access-bgt7j\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:04.014481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.014402    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e47bac65-a69e-45e0-8a8c-f239e14c2e67-plugins-config-volume" pod "e47bac65-a69e-45e0-8a8c-f239e14c2e67" (UID: "e47bac65-a69e-45e0-8a8c-f239e14c2e67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:04.014481 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.014406    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f909610e-c454-4ab6-bdf2-d061ade22e78-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:04.015872 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.015853    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47bac65-a69e-45e0-8a8c-f239e14c2e67-kube-api-access-tw7tp" pod "e47bac65-a69e-45e0-8a8c-f239e14c2e67" (UID: "e47bac65-a69e-45e0-8a8c-f239e14c2e67"). InnerVolumeSpecName "kube-api-access-tw7tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:04.121354 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.121321    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47bac65-a69e-45e0-8a8c-f239e14c2e67-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:04.121354 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.121355    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw7tp\" (UniqueName: \"kubernetes.io/projected/e47bac65-a69e-45e0-8a8c-f239e14c2e67-kube-api-access-tw7tp\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:04.638433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.638400    4425 generic.go:356] "Generic (PLEG): container finished" podID="e47bac65-a69e-45e0-8a8c-f239e14c2e67" containerID="55eac7f9375154924459baf1df17d10ddff844820bb3c4ac0d023af0d8158628" exitCode=0
Oct 30 01:55:04.638916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.638447    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"
Oct 30 01:55:04.638916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.638444    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs" event={"ID":"e47bac65-a69e-45e0-8a8c-f239e14c2e67","Type":"ContainerDied","Data":"55eac7f9375154924459baf1df17d10ddff844820bb3c4ac0d023af0d8158628"}
Oct 30 01:55:04.638916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.638600    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs" event={"ID":"e47bac65-a69e-45e0-8a8c-f239e14c2e67","Type":"ContainerDied","Data":"3715bcaaf13991d487fe37d7569c210a3d7b1fd3264bc875580af7a3437c9cc9"}
Oct 30 01:55:04.638916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.638639    4425 scope.go:122] "RemoveContainer" containerID="55eac7f9375154924459baf1df17d10ddff844820bb3c4ac0d023af0d8158628"
Oct 30 01:55:04.640020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.639992    4425 generic.go:356] "Generic (PLEG): container finished" podID="f909610e-c454-4ab6-bdf2-d061ade22e78" containerID="e56f1683037c50bec3d8cc9650dfadd3ec083672e4f60703f8052ed26c89f602" exitCode=0
Oct 30 01:55:04.640146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.640032    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"
Oct 30 01:55:04.640146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.640077    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7" event={"ID":"f909610e-c454-4ab6-bdf2-d061ade22e78","Type":"ContainerDied","Data":"e56f1683037c50bec3d8cc9650dfadd3ec083672e4f60703f8052ed26c89f602"}
Oct 30 01:55:04.640146 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.640117    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7" event={"ID":"f909610e-c454-4ab6-bdf2-d061ade22e78","Type":"ContainerDied","Data":"e35c6329a8e9dd9edb1864e4de0ce601d1860a97f36fe02584ad794a1e5a39da"}
Oct 30 01:55:04.650089 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.650068    4425 scope.go:122] "RemoveContainer" containerID="55eac7f9375154924459baf1df17d10ddff844820bb3c4ac0d023af0d8158628"
Oct 30 01:55:04.650537 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:55:04.650381    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55eac7f9375154924459baf1df17d10ddff844820bb3c4ac0d023af0d8158628\": container with ID starting with 55eac7f9375154924459baf1df17d10ddff844820bb3c4ac0d023af0d8158628 not found: ID does not exist" containerID="55eac7f9375154924459baf1df17d10ddff844820bb3c4ac0d023af0d8158628"
Oct 30 01:55:04.650537 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.650413    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55eac7f9375154924459baf1df17d10ddff844820bb3c4ac0d023af0d8158628"} err="failed to get container status \"55eac7f9375154924459baf1df17d10ddff844820bb3c4ac0d023af0d8158628\": rpc error: code = NotFound desc = could not find container \"55eac7f9375154924459baf1df17d10ddff844820bb3c4ac0d023af0d8158628\": container with ID starting with 55eac7f9375154924459baf1df17d10ddff844820bb3c4ac0d023af0d8158628 not found: ID does not exist"
Oct 30 01:55:04.650537 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.650434    4425 scope.go:122] "RemoveContainer" containerID="e56f1683037c50bec3d8cc9650dfadd3ec083672e4f60703f8052ed26c89f602"
Oct 30 01:55:04.662857 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.662838    4425 scope.go:122] "RemoveContainer" containerID="e56f1683037c50bec3d8cc9650dfadd3ec083672e4f60703f8052ed26c89f602"
Oct 30 01:55:04.663206 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:55:04.663172    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56f1683037c50bec3d8cc9650dfadd3ec083672e4f60703f8052ed26c89f602\": container with ID starting with e56f1683037c50bec3d8cc9650dfadd3ec083672e4f60703f8052ed26c89f602 not found: ID does not exist" containerID="e56f1683037c50bec3d8cc9650dfadd3ec083672e4f60703f8052ed26c89f602"
Oct 30 01:55:04.663307 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.663211    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56f1683037c50bec3d8cc9650dfadd3ec083672e4f60703f8052ed26c89f602"} err="failed to get container status \"e56f1683037c50bec3d8cc9650dfadd3ec083672e4f60703f8052ed26c89f602\": rpc error: code = NotFound desc = could not find container \"e56f1683037c50bec3d8cc9650dfadd3ec083672e4f60703f8052ed26c89f602\": container with ID starting with e56f1683037c50bec3d8cc9650dfadd3ec083672e4f60703f8052ed26c89f602 not found: ID does not exist"
Oct 30 01:55:04.698568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.698546    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"]
Oct 30 01:55:04.707633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.707614    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1854c4-b6f5594bb-p7gd7"]
Oct 30 01:55:04.730883 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.730862    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"]
Oct 30 01:55:04.735664 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:04.735636    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1854c4-d88957d6c-g5krs"]
Oct 30 01:55:05.422285 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:05.422256    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e47bac65-a69e-45e0-8a8c-f239e14c2e67" path="/var/lib/kubelet/pods/e47bac65-a69e-45e0-8a8c-f239e14c2e67/volumes"
Oct 30 01:55:05.422665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:05.422652    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f909610e-c454-4ab6-bdf2-d061ade22e78" path="/var/lib/kubelet/pods/f909610e-c454-4ab6-bdf2-d061ade22e78/volumes"
Oct 30 01:55:06.607418 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.607386    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"]
Oct 30 01:55:06.607902 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.607889    4425 cpu_manager.go:414] "containerMap: removing container" podUID="e47bac65-a69e-45e0-8a8c-f239e14c2e67" containerName="payload-pre-processing"
Oct 30 01:55:06.607949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.607902    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47bac65-a69e-45e0-8a8c-f239e14c2e67" containerName="payload-pre-processing"
Oct 30 01:55:06.607949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.607918    4425 cpu_manager.go:414] "containerMap: removing container" podUID="f909610e-c454-4ab6-bdf2-d061ade22e78" containerName="payload-processing"
Oct 30 01:55:06.607949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.607923    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f909610e-c454-4ab6-bdf2-d061ade22e78" containerName="payload-processing"
Oct 30 01:55:06.607949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.607935    4425 cpu_manager.go:414] "containerMap: removing container" podUID="57a5fb72-0291-4566-9101-4b438be4a2b0" containerName="revoke-keys"
Oct 30 01:55:06.607949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.607940    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a5fb72-0291-4566-9101-4b438be4a2b0" containerName="revoke-keys"
Oct 30 01:55:06.608105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.607951    4425 cpu_manager.go:414] "containerMap: removing container" podUID="54a68aab-1017-46ff-857f-f465634b7f02" containerName="revoke-keys"
Oct 30 01:55:06.608105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.607956    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a68aab-1017-46ff-857f-f465634b7f02" containerName="revoke-keys"
Oct 30 01:55:06.608105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.608045    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47bac65-a69e-45e0-8a8c-f239e14c2e67" containerName="payload-pre-processing"
Oct 30 01:55:06.608105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.608064    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f909610e-c454-4ab6-bdf2-d061ade22e78" containerName="payload-processing"
Oct 30 01:55:06.608105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.608071    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a5fb72-0291-4566-9101-4b438be4a2b0" containerName="revoke-keys"
Oct 30 01:55:06.608105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.608078    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a68aab-1017-46ff-857f-f465634b7f02" containerName="revoke-keys"
Oct 30 01:55:06.611452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.611427    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"
Oct 30 01:55:06.630712 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.630686    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"]
Oct 30 01:55:06.636541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.636517    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:55:06.636671 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.636605    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:55:06.636726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.636694    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:55:06.636726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.636610    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:55:06.659294 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.659259    4425 generic.go:356] "Generic (PLEG): container finished" podID="50a5c7e3-26ea-42d6-b760-2ac74b5bdda1" containerID="c2c3e91f9224de206a584ce7e15d5b0de34b92d221f300c56f353dd98d17541c" exitCode=0
Oct 30 01:55:06.659498 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.659479    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s" event={"ID":"50a5c7e3-26ea-42d6-b760-2ac74b5bdda1","Type":"ContainerDied","Data":"c2c3e91f9224de206a584ce7e15d5b0de34b92d221f300c56f353dd98d17541c"}
Oct 30 01:55:06.735113 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.735092    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"
Oct 30 01:55:06.750643 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.750614    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgzvb\" (UniqueName: \"kubernetes.io/projected/264d250a-bc4d-4fd6-a1fa-f0a446892681-kube-api-access-dgzvb\") pod \"maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw\" (UID: \"264d250a-bc4d-4fd6-a1fa-f0a446892681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"
Oct 30 01:55:06.750762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.750739    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/264d250a-bc4d-4fd6-a1fa-f0a446892681-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw\" (UID: \"264d250a-bc4d-4fd6-a1fa-f0a446892681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"
Oct 30 01:55:06.750827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.750813    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/264d250a-bc4d-4fd6-a1fa-f0a446892681-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw\" (UID: \"264d250a-bc4d-4fd6-a1fa-f0a446892681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"
Oct 30 01:55:06.851726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.851692    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a5c7e3-26ea-42d6-b760-2ac74b5bdda1-kube-api-access-fxhdn\" (UniqueName: \"kubernetes.io/projected/50a5c7e3-26ea-42d6-b760-2ac74b5bdda1-kube-api-access-fxhdn\") pod \"50a5c7e3-26ea-42d6-b760-2ac74b5bdda1\" (UID: \"50a5c7e3-26ea-42d6-b760-2ac74b5bdda1\") "
Oct 30 01:55:06.851904 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.851864    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a5c7e3-26ea-42d6-b760-2ac74b5bdda1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a5c7e3-26ea-42d6-b760-2ac74b5bdda1-plugins-config-volume\") pod \"50a5c7e3-26ea-42d6-b760-2ac74b5bdda1\" (UID: \"50a5c7e3-26ea-42d6-b760-2ac74b5bdda1\") "
Oct 30 01:55:06.852125 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.852106    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/264d250a-bc4d-4fd6-a1fa-f0a446892681-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw\" (UID: \"264d250a-bc4d-4fd6-a1fa-f0a446892681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"
Oct 30 01:55:06.852249 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.852230    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a5c7e3-26ea-42d6-b760-2ac74b5bdda1-plugins-config-volume" pod "50a5c7e3-26ea-42d6-b760-2ac74b5bdda1" (UID: "50a5c7e3-26ea-42d6-b760-2ac74b5bdda1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:06.852311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.852244    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzvb\" (UniqueName: \"kubernetes.io/projected/264d250a-bc4d-4fd6-a1fa-f0a446892681-kube-api-access-dgzvb\") pod \"maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw\" (UID: \"264d250a-bc4d-4fd6-a1fa-f0a446892681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"
Oct 30 01:55:06.852380 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.852364    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/264d250a-bc4d-4fd6-a1fa-f0a446892681-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw\" (UID: \"264d250a-bc4d-4fd6-a1fa-f0a446892681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"
Oct 30 01:55:06.852458 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.852436    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a5c7e3-26ea-42d6-b760-2ac74b5bdda1-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:06.853008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.852983    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/264d250a-bc4d-4fd6-a1fa-f0a446892681-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw\" (UID: \"264d250a-bc4d-4fd6-a1fa-f0a446892681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"
Oct 30 01:55:06.854315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.854282    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a5c7e3-26ea-42d6-b760-2ac74b5bdda1-kube-api-access-fxhdn" pod "50a5c7e3-26ea-42d6-b760-2ac74b5bdda1" (UID: "50a5c7e3-26ea-42d6-b760-2ac74b5bdda1"). InnerVolumeSpecName "kube-api-access-fxhdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:06.854441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.854323    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/264d250a-bc4d-4fd6-a1fa-f0a446892681-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw\" (UID: \"264d250a-bc4d-4fd6-a1fa-f0a446892681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"
Oct 30 01:55:06.873285 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.873264    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgzvb\" (UniqueName: \"kubernetes.io/projected/264d250a-bc4d-4fd6-a1fa-f0a446892681-kube-api-access-dgzvb\") pod \"maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw\" (UID: \"264d250a-bc4d-4fd6-a1fa-f0a446892681\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"
Oct 30 01:55:06.925225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.925206    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"
Oct 30 01:55:06.953972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:06.953949    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxhdn\" (UniqueName: \"kubernetes.io/projected/50a5c7e3-26ea-42d6-b760-2ac74b5bdda1-kube-api-access-fxhdn\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:07.223148 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:07.223115    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"]
Oct 30 01:55:07.225050 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:55:07.225026    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264d250a_bc4d_4fd6_a1fa_f0a446892681.slice/crio-3cfb5cb48eda52571d572ecf4481fed6204072e4ba4cf8f3c62121473eebce7f WatchSource:0}: Error finding container 3cfb5cb48eda52571d572ecf4481fed6204072e4ba4cf8f3c62121473eebce7f: Status 404 returned error can't find the container with id 3cfb5cb48eda52571d572ecf4481fed6204072e4ba4cf8f3c62121473eebce7f
Oct 30 01:55:07.554720 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:55:07.554687    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:55:07.667390 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:07.667353    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s" event={"ID":"50a5c7e3-26ea-42d6-b760-2ac74b5bdda1","Type":"ContainerDied","Data":"c2cd6bfd36a8b426507f4f897a80185d2b6fb04a7b5269b6564043e029c8f97c"}
Oct 30 01:55:07.667892 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:07.667421    4425 scope.go:122] "RemoveContainer" containerID="c2c3e91f9224de206a584ce7e15d5b0de34b92d221f300c56f353dd98d17541c"
Oct 30 01:55:07.667892 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:07.667651    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"
Oct 30 01:55:07.669510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:07.669463    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw" event={"ID":"264d250a-bc4d-4fd6-a1fa-f0a446892681","Type":"ContainerStarted","Data":"eb3cc498ee55e2bc64475a772caf68d8f35453053c9cefe6e4605dc3cbedd8e7"}
Oct 30 01:55:07.669614 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:07.669536    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw" event={"ID":"264d250a-bc4d-4fd6-a1fa-f0a446892681","Type":"ContainerStarted","Data":"3cfb5cb48eda52571d572ecf4481fed6204072e4ba4cf8f3c62121473eebce7f"}
Oct 30 01:55:07.719809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:07.719721    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"]
Oct 30 01:55:07.724196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:07.724140    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9da49f-6c659f9c4d-frh9s"]
Oct 30 01:55:07.757391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:07.757343    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw" podStartSLOduration=1.7573298990000001 podStartE2EDuration="1.757329899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:07.753419543 +0000 UTC m=+4416.503074850" watchObservedRunningTime="2026-10-30 01:55:07.757329899 +0000 UTC m=+4416.506985206"
Oct 30 01:55:08.291081 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.291046    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"]
Oct 30 01:55:08.291710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.291687    4425 cpu_manager.go:414] "containerMap: removing container" podUID="50a5c7e3-26ea-42d6-b760-2ac74b5bdda1" containerName="payload-processing"
Oct 30 01:55:08.291710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.291706    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a5c7e3-26ea-42d6-b760-2ac74b5bdda1" containerName="payload-processing"
Oct 30 01:55:08.291859 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.291816    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a5c7e3-26ea-42d6-b760-2ac74b5bdda1" containerName="payload-processing"
Oct 30 01:55:08.295321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.295294    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"
Oct 30 01:55:08.314181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.314140    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"]
Oct 30 01:55:08.318984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.318958    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj"]
Oct 30 01:55:08.323044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.323019    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj"
Oct 30 01:55:08.346495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.344105    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj"]
Oct 30 01:55:08.473766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.473728    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td4x7\" (UniqueName: \"kubernetes.io/projected/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-kube-api-access-td4x7\") pod \"maas-api-revoke-sub-f7181d33-777cj\" (UID: \"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj"
Oct 30 01:55:08.473766 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.473766    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl44m\" (UniqueName: \"kubernetes.io/projected/f8d3d737-8a12-44f3-a668-58f45d389278-kube-api-access-jl44m\") pod \"maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb\" (UID: \"f8d3d737-8a12-44f3-a668-58f45d389278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"
Oct 30 01:55:08.473973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.473830    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8d3d737-8a12-44f3-a668-58f45d389278-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb\" (UID: \"f8d3d737-8a12-44f3-a668-58f45d389278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"
Oct 30 01:55:08.473973 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.473903    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7181d33-777cj\" (UID: \"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj"
Oct 30 01:55:08.474046 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.474006    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7181d33-777cj\" (UID: \"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj"
Oct 30 01:55:08.474084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.474058    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8d3d737-8a12-44f3-a668-58f45d389278-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb\" (UID: \"f8d3d737-8a12-44f3-a668-58f45d389278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"
Oct 30 01:55:08.574732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.574696    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7181d33-777cj\" (UID: \"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj"
Oct 30 01:55:08.574966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.574751    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8d3d737-8a12-44f3-a668-58f45d389278-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb\" (UID: \"f8d3d737-8a12-44f3-a668-58f45d389278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"
Oct 30 01:55:08.574966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.574886    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td4x7\" (UniqueName: \"kubernetes.io/projected/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-kube-api-access-td4x7\") pod \"maas-api-revoke-sub-f7181d33-777cj\" (UID: \"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj"
Oct 30 01:55:08.574966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.574921    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl44m\" (UniqueName: \"kubernetes.io/projected/f8d3d737-8a12-44f3-a668-58f45d389278-kube-api-access-jl44m\") pod \"maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb\" (UID: \"f8d3d737-8a12-44f3-a668-58f45d389278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"
Oct 30 01:55:08.575187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.574981    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8d3d737-8a12-44f3-a668-58f45d389278-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb\" (UID: \"f8d3d737-8a12-44f3-a668-58f45d389278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"
Oct 30 01:55:08.575187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.575039    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7181d33-777cj\" (UID: \"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj"
Oct 30 01:55:08.575570 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.575542    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8d3d737-8a12-44f3-a668-58f45d389278-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb\" (UID: \"f8d3d737-8a12-44f3-a668-58f45d389278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"
Oct 30 01:55:08.575701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.575683    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7181d33-777cj\" (UID: \"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj"
Oct 30 01:55:08.577223 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.577196    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7181d33-777cj\" (UID: \"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj"
Oct 30 01:55:08.577388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.577371    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8d3d737-8a12-44f3-a668-58f45d389278-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb\" (UID: \"f8d3d737-8a12-44f3-a668-58f45d389278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"
Oct 30 01:55:08.595720 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.595696    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td4x7\" (UniqueName: \"kubernetes.io/projected/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-kube-api-access-td4x7\") pod \"maas-api-revoke-sub-f7181d33-777cj\" (UID: \"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj"
Oct 30 01:55:08.595802 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.595753    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl44m\" (UniqueName: \"kubernetes.io/projected/f8d3d737-8a12-44f3-a668-58f45d389278-kube-api-access-jl44m\") pod \"maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb\" (UID: \"f8d3d737-8a12-44f3-a668-58f45d389278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"
Oct 30 01:55:08.610631 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.610583    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"
Oct 30 01:55:08.640015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.639980    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj"
Oct 30 01:55:08.692070 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.691361    4425 generic.go:356] "Generic (PLEG): container finished" podID="264d250a-bc4d-4fd6-a1fa-f0a446892681" containerID="eb3cc498ee55e2bc64475a772caf68d8f35453053c9cefe6e4605dc3cbedd8e7" exitCode=0
Oct 30 01:55:08.692070 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.691559    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw" event={"ID":"264d250a-bc4d-4fd6-a1fa-f0a446892681","Type":"ContainerDied","Data":"eb3cc498ee55e2bc64475a772caf68d8f35453053c9cefe6e4605dc3cbedd8e7"}
Oct 30 01:55:08.908564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.908487    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x" podUID="3ac3ee9c-d077-44ac-8be5-ed5555679412" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:55:08.933930 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.933904    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"]
Oct 30 01:55:08.935951 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:55:08.935926    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d3d737_8a12_44f3_a668_58f45d389278.slice/crio-19bff0a0ee28dc4578743da46387cea254e717a8ae761959d59fd005b6104d45 WatchSource:0}: Error finding container 19bff0a0ee28dc4578743da46387cea254e717a8ae761959d59fd005b6104d45: Status 404 returned error can't find the container with id 19bff0a0ee28dc4578743da46387cea254e717a8ae761959d59fd005b6104d45
Oct 30 01:55:08.997511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:08.997490    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj"]
Oct 30 01:55:09.000100 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:55:09.000075    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7f6bf7_22f5_4a95_adc7_9683a8c4c8c1.slice/crio-17ca82ba7ef64b7e030572363ca692bc8c5a9d86b7c6ef7d1bf67c4baba20dd4 WatchSource:0}: Error finding container 17ca82ba7ef64b7e030572363ca692bc8c5a9d86b7c6ef7d1bf67c4baba20dd4: Status 404 returned error can't find the container with id 17ca82ba7ef64b7e030572363ca692bc8c5a9d86b7c6ef7d1bf67c4baba20dd4
Oct 30 01:55:09.266523 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.266496    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"]
Oct 30 01:55:09.270511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.270494    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"
Oct 30 01:55:09.290551 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.290514    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"]
Oct 30 01:55:09.291339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.291315    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7facc24-9378-4007-959c-a346f03fcced-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr\" (UID: \"e7facc24-9378-4007-959c-a346f03fcced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"
Oct 30 01:55:09.291453 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.291417    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7facc24-9378-4007-959c-a346f03fcced-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr\" (UID: \"e7facc24-9378-4007-959c-a346f03fcced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"
Oct 30 01:55:09.291575 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.291553    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrc2z\" (UniqueName: \"kubernetes.io/projected/e7facc24-9378-4007-959c-a346f03fcced-kube-api-access-mrc2z\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr\" (UID: \"e7facc24-9378-4007-959c-a346f03fcced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"
Oct 30 01:55:09.392122 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.392089    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7facc24-9378-4007-959c-a346f03fcced-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr\" (UID: \"e7facc24-9378-4007-959c-a346f03fcced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"
Oct 30 01:55:09.392315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.392199    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrc2z\" (UniqueName: \"kubernetes.io/projected/e7facc24-9378-4007-959c-a346f03fcced-kube-api-access-mrc2z\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr\" (UID: \"e7facc24-9378-4007-959c-a346f03fcced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"
Oct 30 01:55:09.392315 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.392292    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7facc24-9378-4007-959c-a346f03fcced-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr\" (UID: \"e7facc24-9378-4007-959c-a346f03fcced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"
Oct 30 01:55:09.392765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.392737    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7facc24-9378-4007-959c-a346f03fcced-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr\" (UID: \"e7facc24-9378-4007-959c-a346f03fcced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"
Oct 30 01:55:09.394583 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.394557    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7facc24-9378-4007-959c-a346f03fcced-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr\" (UID: \"e7facc24-9378-4007-959c-a346f03fcced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"
Oct 30 01:55:09.413284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.413258    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrc2z\" (UniqueName: \"kubernetes.io/projected/e7facc24-9378-4007-959c-a346f03fcced-kube-api-access-mrc2z\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr\" (UID: \"e7facc24-9378-4007-959c-a346f03fcced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"
Oct 30 01:55:09.424280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.424233    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a5c7e3-26ea-42d6-b760-2ac74b5bdda1" path="/var/lib/kubelet/pods/50a5c7e3-26ea-42d6-b760-2ac74b5bdda1/volumes"
Oct 30 01:55:09.585367 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.585334    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"
Oct 30 01:55:09.719817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.718942    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb" event={"ID":"f8d3d737-8a12-44f3-a668-58f45d389278","Type":"ContainerStarted","Data":"c7df2ea1bd540aa5e316116b39779c649f54f3b0dca66d12cb8b828a03adcb8d"}
Oct 30 01:55:09.719817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.719000    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb" event={"ID":"f8d3d737-8a12-44f3-a668-58f45d389278","Type":"ContainerStarted","Data":"19bff0a0ee28dc4578743da46387cea254e717a8ae761959d59fd005b6104d45"}
Oct 30 01:55:09.722033 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.722008    4425 generic.go:356] "Generic (PLEG): container finished" podID="4c46fb47-d71a-4831-bf35-2acc1bb4c935" containerID="7441f9119204455033681f3e0b544d35642fdee66bcf70e330c206b9a16bed77" exitCode=0
Oct 30 01:55:09.722150 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.722076    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g" event={"ID":"4c46fb47-d71a-4831-bf35-2acc1bb4c935","Type":"ContainerDied","Data":"7441f9119204455033681f3e0b544d35642fdee66bcf70e330c206b9a16bed77"}
Oct 30 01:55:09.723444 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.723419    4425 generic.go:356] "Generic (PLEG): container finished" podID="3ac3ee9c-d077-44ac-8be5-ed5555679412" containerID="53ef72fdb7e25171fd039f365071887731d6e12a9ddf56266e5a8ab2dff03f9f" exitCode=0
Oct 30 01:55:09.723596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.723579    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x" event={"ID":"3ac3ee9c-d077-44ac-8be5-ed5555679412","Type":"ContainerDied","Data":"53ef72fdb7e25171fd039f365071887731d6e12a9ddf56266e5a8ab2dff03f9f"}
Oct 30 01:55:09.725361 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.725336    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj" event={"ID":"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1","Type":"ContainerStarted","Data":"334a592ebd0b3d038fa07f1148c6b7d9445ae818800bdf4f9322d83a23b572b5"}
Oct 30 01:55:09.725433 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.725382    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj" event={"ID":"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1","Type":"ContainerStarted","Data":"17ca82ba7ef64b7e030572363ca692bc8c5a9d86b7c6ef7d1bf67c4baba20dd4"}
Oct 30 01:55:09.759368 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.757317    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"
Oct 30 01:55:09.767845 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.767821    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"
Oct 30 01:55:09.794482 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.794440    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb" podStartSLOduration=1.794423104 podStartE2EDuration="1.794423104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:09.757441521 +0000 UTC m=+4418.507096825" watchObservedRunningTime="2026-10-30 01:55:09.794423104 +0000 UTC m=+4418.544078412"
Oct 30 01:55:09.796554 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.796528    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c46fb47-d71a-4831-bf35-2acc1bb4c935-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c46fb47-d71a-4831-bf35-2acc1bb4c935-plugins-config-volume\") pod \"4c46fb47-d71a-4831-bf35-2acc1bb4c935\" (UID: \"4c46fb47-d71a-4831-bf35-2acc1bb4c935\") "
Oct 30 01:55:09.796933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.796909    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c46fb47-d71a-4831-bf35-2acc1bb4c935-plugins-config-volume" pod "4c46fb47-d71a-4831-bf35-2acc1bb4c935" (UID: "4c46fb47-d71a-4831-bf35-2acc1bb4c935"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:09.798060 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.797039    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c46fb47-d71a-4831-bf35-2acc1bb4c935-kube-api-access-thzgx\" (UniqueName: \"kubernetes.io/projected/4c46fb47-d71a-4831-bf35-2acc1bb4c935-kube-api-access-thzgx\") pod \"4c46fb47-d71a-4831-bf35-2acc1bb4c935\" (UID: \"4c46fb47-d71a-4831-bf35-2acc1bb4c935\") "
Oct 30 01:55:09.798060 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.797557    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c46fb47-d71a-4831-bf35-2acc1bb4c935-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:09.807409 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.807379    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c46fb47-d71a-4831-bf35-2acc1bb4c935-kube-api-access-thzgx" pod "4c46fb47-d71a-4831-bf35-2acc1bb4c935" (UID: "4c46fb47-d71a-4831-bf35-2acc1bb4c935"). InnerVolumeSpecName "kube-api-access-thzgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:09.832250 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.832197    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj" podStartSLOduration=1.832180414 podStartE2EDuration="1.832180414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:09.829361766 +0000 UTC m=+4418.579017074" watchObservedRunningTime="2026-10-30 01:55:09.832180414 +0000 UTC m=+4418.581835717"
Oct 30 01:55:09.898505 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.898480    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac3ee9c-d077-44ac-8be5-ed5555679412-kube-api-access-2gdg8\" (UniqueName: \"kubernetes.io/projected/3ac3ee9c-d077-44ac-8be5-ed5555679412-kube-api-access-2gdg8\") pod \"3ac3ee9c-d077-44ac-8be5-ed5555679412\" (UID: \"3ac3ee9c-d077-44ac-8be5-ed5555679412\") "
Oct 30 01:55:09.898634 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.898515    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac3ee9c-d077-44ac-8be5-ed5555679412-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac3ee9c-d077-44ac-8be5-ed5555679412-plugins-config-volume\") pod \"3ac3ee9c-d077-44ac-8be5-ed5555679412\" (UID: \"3ac3ee9c-d077-44ac-8be5-ed5555679412\") "
Oct 30 01:55:09.898856 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.898843    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thzgx\" (UniqueName: \"kubernetes.io/projected/4c46fb47-d71a-4831-bf35-2acc1bb4c935-kube-api-access-thzgx\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:09.898949 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.898922    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac3ee9c-d077-44ac-8be5-ed5555679412-plugins-config-volume" pod "3ac3ee9c-d077-44ac-8be5-ed5555679412" (UID: "3ac3ee9c-d077-44ac-8be5-ed5555679412"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:09.900992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.900965    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac3ee9c-d077-44ac-8be5-ed5555679412-kube-api-access-2gdg8" pod "3ac3ee9c-d077-44ac-8be5-ed5555679412" (UID: "3ac3ee9c-d077-44ac-8be5-ed5555679412"). InnerVolumeSpecName "kube-api-access-2gdg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:09.925788 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.925765    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"]
Oct 30 01:55:09.951223 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:55:09.951149    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7facc24_9378_4007_959c_a346f03fcced.slice/crio-b1d3e6e68ebe90ec609341658d8b32e85f006352afd0fc92f4d1bb1e233828f8 WatchSource:0}: Error finding container b1d3e6e68ebe90ec609341658d8b32e85f006352afd0fc92f4d1bb1e233828f8: Status 404 returned error can't find the container with id b1d3e6e68ebe90ec609341658d8b32e85f006352afd0fc92f4d1bb1e233828f8
Oct 30 01:55:09.956637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:09.956615    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"
Oct 30 01:55:10.000510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.000489    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/264d250a-bc4d-4fd6-a1fa-f0a446892681-kube-api-access-dgzvb\" (UniqueName: \"kubernetes.io/projected/264d250a-bc4d-4fd6-a1fa-f0a446892681-kube-api-access-dgzvb\") pod \"264d250a-bc4d-4fd6-a1fa-f0a446892681\" (UID: \"264d250a-bc4d-4fd6-a1fa-f0a446892681\") "
Oct 30 01:55:10.000622 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.000559    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/264d250a-bc4d-4fd6-a1fa-f0a446892681-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/264d250a-bc4d-4fd6-a1fa-f0a446892681-maas-api-xks-ca\") pod \"264d250a-bc4d-4fd6-a1fa-f0a446892681\" (UID: \"264d250a-bc4d-4fd6-a1fa-f0a446892681\") "
Oct 30 01:55:10.000622 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.000610    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/264d250a-bc4d-4fd6-a1fa-f0a446892681-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/264d250a-bc4d-4fd6-a1fa-f0a446892681-maas-api-openshift-ca\") pod \"264d250a-bc4d-4fd6-a1fa-f0a446892681\" (UID: \"264d250a-bc4d-4fd6-a1fa-f0a446892681\") "
Oct 30 01:55:10.001044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.001020    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264d250a-bc4d-4fd6-a1fa-f0a446892681-maas-api-openshift-ca" pod "264d250a-bc4d-4fd6-a1fa-f0a446892681" (UID: "264d250a-bc4d-4fd6-a1fa-f0a446892681"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:10.001479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.001457    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/264d250a-bc4d-4fd6-a1fa-f0a446892681-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:10.001556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.001486    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gdg8\" (UniqueName: \"kubernetes.io/projected/3ac3ee9c-d077-44ac-8be5-ed5555679412-kube-api-access-2gdg8\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:10.001556 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.001501    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac3ee9c-d077-44ac-8be5-ed5555679412-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:10.002648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.002627    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/264d250a-bc4d-4fd6-a1fa-f0a446892681-maas-api-xks-ca" pod "264d250a-bc4d-4fd6-a1fa-f0a446892681" (UID: "264d250a-bc4d-4fd6-a1fa-f0a446892681"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:10.002872 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.002842    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264d250a-bc4d-4fd6-a1fa-f0a446892681-kube-api-access-dgzvb" pod "264d250a-bc4d-4fd6-a1fa-f0a446892681" (UID: "264d250a-bc4d-4fd6-a1fa-f0a446892681"). InnerVolumeSpecName "kube-api-access-dgzvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:10.102620 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.102594    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgzvb\" (UniqueName: \"kubernetes.io/projected/264d250a-bc4d-4fd6-a1fa-f0a446892681-kube-api-access-dgzvb\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:10.102620 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.102621    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/264d250a-bc4d-4fd6-a1fa-f0a446892681-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:10.732937 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.732898    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g" event={"ID":"4c46fb47-d71a-4831-bf35-2acc1bb4c935","Type":"ContainerDied","Data":"fc3d730dc1d55f6b427daefce85514a71a88288434aacf885eda40ec6ad8820e"}
Oct 30 01:55:10.733399 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.732962    4425 scope.go:122] "RemoveContainer" containerID="7441f9119204455033681f3e0b544d35642fdee66bcf70e330c206b9a16bed77"
Oct 30 01:55:10.733399 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.732900    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"
Oct 30 01:55:10.734478 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.734449    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x" event={"ID":"3ac3ee9c-d077-44ac-8be5-ed5555679412","Type":"ContainerDied","Data":"0e2737fcac31df6faab28c081e7650ac702a7a22a0620a4f02fcd461fe29953f"}
Oct 30 01:55:10.734478 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.734468    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"
Oct 30 01:55:10.736396 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.736366    4425 generic.go:356] "Generic (PLEG): container finished" podID="7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1" containerID="334a592ebd0b3d038fa07f1148c6b7d9445ae818800bdf4f9322d83a23b572b5" exitCode=0
Oct 30 01:55:10.736835 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.736449    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj" event={"ID":"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1","Type":"ContainerDied","Data":"334a592ebd0b3d038fa07f1148c6b7d9445ae818800bdf4f9322d83a23b572b5"}
Oct 30 01:55:10.738333 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.738307    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr" event={"ID":"e7facc24-9378-4007-959c-a346f03fcced","Type":"ContainerStarted","Data":"a63e4c01c37cc10b437a1d313791d7feb9839240a220116827e017b8442753cb"}
Oct 30 01:55:10.738519 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.738494    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr" event={"ID":"e7facc24-9378-4007-959c-a346f03fcced","Type":"ContainerStarted","Data":"b1d3e6e68ebe90ec609341658d8b32e85f006352afd0fc92f4d1bb1e233828f8"}
Oct 30 01:55:10.739876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.739859    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"
Oct 30 01:55:10.740062 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.739879    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw" event={"ID":"264d250a-bc4d-4fd6-a1fa-f0a446892681","Type":"ContainerDied","Data":"3cfb5cb48eda52571d572ecf4481fed6204072e4ba4cf8f3c62121473eebce7f"}
Oct 30 01:55:10.740278 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.740090    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cfb5cb48eda52571d572ecf4481fed6204072e4ba4cf8f3c62121473eebce7f"
Oct 30 01:55:10.741513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.741489    4425 generic.go:356] "Generic (PLEG): container finished" podID="f8d3d737-8a12-44f3-a668-58f45d389278" containerID="c7df2ea1bd540aa5e316116b39779c649f54f3b0dca66d12cb8b828a03adcb8d" exitCode=0
Oct 30 01:55:10.741614 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.741522    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb" event={"ID":"f8d3d737-8a12-44f3-a668-58f45d389278","Type":"ContainerDied","Data":"c7df2ea1bd540aa5e316116b39779c649f54f3b0dca66d12cb8b828a03adcb8d"}
Oct 30 01:55:10.745767 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.745694    4425 scope.go:122] "RemoveContainer" containerID="53ef72fdb7e25171fd039f365071887731d6e12a9ddf56266e5a8ab2dff03f9f"
Oct 30 01:55:10.869084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.869041    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr" podStartSLOduration=1.869025466 podStartE2EDuration="1.869025466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:10.867053553 +0000 UTC m=+4419.616708860" watchObservedRunningTime="2026-10-30 01:55:10.869025466 +0000 UTC m=+4419.618680774"
Oct 30 01:55:10.901370 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.901335    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"]
Oct 30 01:55:10.908374 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.908348    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-8ae41e-9565975b4-ln26x"]
Oct 30 01:55:10.936094 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.936059    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"]
Oct 30 01:55:10.944586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:10.944558    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-8ae41e-57d9645dc-ngw5g"]
Oct 30 01:55:11.340927 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:11.340881    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"]
Oct 30 01:55:11.348187 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:11.348135    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"]
Oct 30 01:55:11.373921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:11.373888    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"]
Oct 30 01:55:11.423021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:11.422989    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac3ee9c-d077-44ac-8be5-ed5555679412" path="/var/lib/kubelet/pods/3ac3ee9c-d077-44ac-8be5-ed5555679412/volumes"
Oct 30 01:55:11.423559 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:11.423535    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c46fb47-d71a-4831-bf35-2acc1bb4c935" path="/var/lib/kubelet/pods/4c46fb47-d71a-4831-bf35-2acc1bb4c935/volumes"
Oct 30 01:55:11.749723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:11.749640    4425 generic.go:356] "Generic (PLEG): container finished" podID="e7facc24-9378-4007-959c-a346f03fcced" containerID="a63e4c01c37cc10b437a1d313791d7feb9839240a220116827e017b8442753cb" exitCode=0
Oct 30 01:55:11.750185 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:11.749721    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr" event={"ID":"e7facc24-9378-4007-959c-a346f03fcced","Type":"ContainerDied","Data":"a63e4c01c37cc10b437a1d313791d7feb9839240a220116827e017b8442753cb"}
Oct 30 01:55:11.944425 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:11.944400    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"
Oct 30 01:55:11.946798 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:11.946781    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj"
Oct 30 01:55:12.023410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.023379    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8d3d737-8a12-44f3-a668-58f45d389278-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8d3d737-8a12-44f3-a668-58f45d389278-maas-api-openshift-ca\") pod \"f8d3d737-8a12-44f3-a668-58f45d389278\" (UID: \"f8d3d737-8a12-44f3-a668-58f45d389278\") "
Oct 30 01:55:12.023650 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.023471    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8d3d737-8a12-44f3-a668-58f45d389278-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8d3d737-8a12-44f3-a668-58f45d389278-maas-api-xks-ca\") pod \"f8d3d737-8a12-44f3-a668-58f45d389278\" (UID: \"f8d3d737-8a12-44f3-a668-58f45d389278\") "
Oct 30 01:55:12.023650 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.023509    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d3d737-8a12-44f3-a668-58f45d389278-kube-api-access-jl44m\" (UniqueName: \"kubernetes.io/projected/f8d3d737-8a12-44f3-a668-58f45d389278-kube-api-access-jl44m\") pod \"f8d3d737-8a12-44f3-a668-58f45d389278\" (UID: \"f8d3d737-8a12-44f3-a668-58f45d389278\") "
Oct 30 01:55:12.023650 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.023581    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-maas-api-xks-ca\") pod \"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1\" (UID: \"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1\") "
Oct 30 01:55:12.023781 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.023720    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-maas-api-openshift-ca\") pod \"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1\" (UID: \"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1\") "
Oct 30 01:55:12.023781 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.023747    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-kube-api-access-td4x7\" (UniqueName: \"kubernetes.io/projected/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-kube-api-access-td4x7\") pod \"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1\" (UID: \"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1\") "
Oct 30 01:55:12.023855 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.023798    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d3d737-8a12-44f3-a668-58f45d389278-maas-api-openshift-ca" pod "f8d3d737-8a12-44f3-a668-58f45d389278" (UID: "f8d3d737-8a12-44f3-a668-58f45d389278"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:12.024190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.024136    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8d3d737-8a12-44f3-a668-58f45d389278-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:12.024555 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.024526    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-maas-api-openshift-ca" pod "7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1" (UID: "7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:12.032265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.026685    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d3d737-8a12-44f3-a668-58f45d389278-kube-api-access-jl44m" pod "f8d3d737-8a12-44f3-a668-58f45d389278" (UID: "f8d3d737-8a12-44f3-a668-58f45d389278"). InnerVolumeSpecName "kube-api-access-jl44m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:12.032265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.026745    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-kube-api-access-td4x7" pod "7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1" (UID: "7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1"). InnerVolumeSpecName "kube-api-access-td4x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:12.032265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.026946    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8d3d737-8a12-44f3-a668-58f45d389278-maas-api-xks-ca" pod "f8d3d737-8a12-44f3-a668-58f45d389278" (UID: "f8d3d737-8a12-44f3-a668-58f45d389278"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:12.032265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.027174    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-maas-api-xks-ca" pod "7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1" (UID: "7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:12.125335 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.125307    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8d3d737-8a12-44f3-a668-58f45d389278-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:12.125335 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.125333    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl44m\" (UniqueName: \"kubernetes.io/projected/f8d3d737-8a12-44f3-a668-58f45d389278-kube-api-access-jl44m\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:12.125335 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.125344    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:12.125630 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.125357    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:12.125630 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.125369    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td4x7\" (UniqueName: \"kubernetes.io/projected/7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1-kube-api-access-td4x7\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:12.764005 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.763973    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj"
Oct 30 01:55:12.764422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.764001    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7181d33-777cj" event={"ID":"7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1","Type":"ContainerDied","Data":"17ca82ba7ef64b7e030572363ca692bc8c5a9d86b7c6ef7d1bf67c4baba20dd4"}
Oct 30 01:55:12.764422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.764050    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17ca82ba7ef64b7e030572363ca692bc8c5a9d86b7c6ef7d1bf67c4baba20dd4"
Oct 30 01:55:12.765452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.765435    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"
Oct 30 01:55:12.765559 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.765467    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb" event={"ID":"f8d3d737-8a12-44f3-a668-58f45d389278","Type":"ContainerDied","Data":"19bff0a0ee28dc4578743da46387cea254e717a8ae761959d59fd005b6104d45"}
Oct 30 01:55:12.765559 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.765502    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19bff0a0ee28dc4578743da46387cea254e717a8ae761959d59fd005b6104d45"
Oct 30 01:55:12.891506 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.891482    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"
Oct 30 01:55:12.933993 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.933959    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7facc24-9378-4007-959c-a346f03fcced-kube-api-access-mrc2z\" (UniqueName: \"kubernetes.io/projected/e7facc24-9378-4007-959c-a346f03fcced-kube-api-access-mrc2z\") pod \"e7facc24-9378-4007-959c-a346f03fcced\" (UID: \"e7facc24-9378-4007-959c-a346f03fcced\") "
Oct 30 01:55:12.934144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.934125    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7facc24-9378-4007-959c-a346f03fcced-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7facc24-9378-4007-959c-a346f03fcced-maas-api-openshift-ca\") pod \"e7facc24-9378-4007-959c-a346f03fcced\" (UID: \"e7facc24-9378-4007-959c-a346f03fcced\") "
Oct 30 01:55:12.934290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.934221    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7facc24-9378-4007-959c-a346f03fcced-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7facc24-9378-4007-959c-a346f03fcced-maas-api-xks-ca\") pod \"e7facc24-9378-4007-959c-a346f03fcced\" (UID: \"e7facc24-9378-4007-959c-a346f03fcced\") "
Oct 30 01:55:12.934503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.934473    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7facc24-9378-4007-959c-a346f03fcced-maas-api-openshift-ca" pod "e7facc24-9378-4007-959c-a346f03fcced" (UID: "e7facc24-9378-4007-959c-a346f03fcced"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:12.934804 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.934781    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7facc24-9378-4007-959c-a346f03fcced-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:12.936344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.936316    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7facc24-9378-4007-959c-a346f03fcced-maas-api-xks-ca" pod "e7facc24-9378-4007-959c-a346f03fcced" (UID: "e7facc24-9378-4007-959c-a346f03fcced"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:12.936468 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:12.936365    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7facc24-9378-4007-959c-a346f03fcced-kube-api-access-mrc2z" pod "e7facc24-9378-4007-959c-a346f03fcced" (UID: "e7facc24-9378-4007-959c-a346f03fcced"). InnerVolumeSpecName "kube-api-access-mrc2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:13.032793 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:13.032712    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"]
Oct 30 01:55:13.036047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:13.036016    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7facc24-9378-4007-959c-a346f03fcced-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:13.036047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:13.036046    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrc2z\" (UniqueName: \"kubernetes.io/projected/e7facc24-9378-4007-959c-a346f03fcced-kube-api-access-mrc2z\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:13.046355 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:13.046331    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"]
Oct 30 01:55:13.774531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:13.774504    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"
Oct 30 01:55:13.774980 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:13.774499    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr" event={"ID":"e7facc24-9378-4007-959c-a346f03fcced","Type":"ContainerDied","Data":"b1d3e6e68ebe90ec609341658d8b32e85f006352afd0fc92f4d1bb1e233828f8"}
Oct 30 01:55:13.774980 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:13.774645    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1d3e6e68ebe90ec609341658d8b32e85f006352afd0fc92f4d1bb1e233828f8"
Oct 30 01:55:18.691717 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.691677    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49"]
Oct 30 01:55:18.692199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692182    4425 cpu_manager.go:414] "containerMap: removing container" podUID="7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1" containerName="revoke-keys"
Oct 30 01:55:18.692199 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692198    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1" containerName="revoke-keys"
Oct 30 01:55:18.692303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692207    4425 cpu_manager.go:414] "containerMap: removing container" podUID="e7facc24-9378-4007-959c-a346f03fcced" containerName="revoke-keys"
Oct 30 01:55:18.692303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692212    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7facc24-9378-4007-959c-a346f03fcced" containerName="revoke-keys"
Oct 30 01:55:18.692303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692236    4425 cpu_manager.go:414] "containerMap: removing container" podUID="264d250a-bc4d-4fd6-a1fa-f0a446892681" containerName="revoke-keys"
Oct 30 01:55:18.692303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692240    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="264d250a-bc4d-4fd6-a1fa-f0a446892681" containerName="revoke-keys"
Oct 30 01:55:18.692303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692258    4425 cpu_manager.go:414] "containerMap: removing container" podUID="4c46fb47-d71a-4831-bf35-2acc1bb4c935" containerName="payload-pre-processing"
Oct 30 01:55:18.692303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692263    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c46fb47-d71a-4831-bf35-2acc1bb4c935" containerName="payload-pre-processing"
Oct 30 01:55:18.692303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692271    4425 cpu_manager.go:414] "containerMap: removing container" podUID="f8d3d737-8a12-44f3-a668-58f45d389278" containerName="revoke-keys"
Oct 30 01:55:18.692303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692275    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d3d737-8a12-44f3-a668-58f45d389278" containerName="revoke-keys"
Oct 30 01:55:18.692303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692282    4425 cpu_manager.go:414] "containerMap: removing container" podUID="3ac3ee9c-d077-44ac-8be5-ed5555679412" containerName="payload-processing"
Oct 30 01:55:18.692303 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692286    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac3ee9c-d077-44ac-8be5-ed5555679412" containerName="payload-processing"
Oct 30 01:55:18.692617 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692353    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c46fb47-d71a-4831-bf35-2acc1bb4c935" containerName="payload-pre-processing"
Oct 30 01:55:18.692617 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692361    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d3d737-8a12-44f3-a668-58f45d389278" containerName="revoke-keys"
Oct 30 01:55:18.692617 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692368    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac3ee9c-d077-44ac-8be5-ed5555679412" containerName="payload-processing"
Oct 30 01:55:18.692617 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692376    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1" containerName="revoke-keys"
Oct 30 01:55:18.692617 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692382    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7facc24-9378-4007-959c-a346f03fcced" containerName="revoke-keys"
Oct 30 01:55:18.692617 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.692391    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="264d250a-bc4d-4fd6-a1fa-f0a446892681" containerName="revoke-keys"
Oct 30 01:55:18.695606 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.695588    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49"
Oct 30 01:55:18.708488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.708470    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:55:18.718458 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.718436    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:55:18.718580 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.718479    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:55:18.718644 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.718479    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:55:18.722729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.722702    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49"]
Oct 30 01:55:18.797102 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.797071    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-116ece75-d5h49\" (UID: \"c5c9ddbd-6216-401d-ad22-dc4821de1bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49"
Oct 30 01:55:18.797265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.797208    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-116ece75-d5h49\" (UID: \"c5c9ddbd-6216-401d-ad22-dc4821de1bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49"
Oct 30 01:55:18.797347 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.797319    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbxf4\" (UniqueName: \"kubernetes.io/projected/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-kube-api-access-dbxf4\") pod \"maas-api-revoke-sub-116ece75-d5h49\" (UID: \"c5c9ddbd-6216-401d-ad22-dc4821de1bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49"
Oct 30 01:55:18.898313 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.898280    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-116ece75-d5h49\" (UID: \"c5c9ddbd-6216-401d-ad22-dc4821de1bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49"
Oct 30 01:55:18.898449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.898343    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbxf4\" (UniqueName: \"kubernetes.io/projected/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-kube-api-access-dbxf4\") pod \"maas-api-revoke-sub-116ece75-d5h49\" (UID: \"c5c9ddbd-6216-401d-ad22-dc4821de1bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49"
Oct 30 01:55:18.898449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.898433    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-116ece75-d5h49\" (UID: \"c5c9ddbd-6216-401d-ad22-dc4821de1bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49"
Oct 30 01:55:18.899029 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.899004    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-116ece75-d5h49\" (UID: \"c5c9ddbd-6216-401d-ad22-dc4821de1bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49"
Oct 30 01:55:18.900546 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.900520    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-116ece75-d5h49\" (UID: \"c5c9ddbd-6216-401d-ad22-dc4821de1bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49"
Oct 30 01:55:18.935822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:18.935790    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbxf4\" (UniqueName: \"kubernetes.io/projected/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-kube-api-access-dbxf4\") pod \"maas-api-revoke-sub-116ece75-d5h49\" (UID: \"c5c9ddbd-6216-401d-ad22-dc4821de1bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49"
Oct 30 01:55:19.009805 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:19.009781    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49"
Oct 30 01:55:19.313598 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:19.313571    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49"]
Oct 30 01:55:19.314761 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:55:19.314737    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c9ddbd_6216_401d_ad22_dc4821de1bc2.slice/crio-510cc37ecdfd3fb861ea7712d54bad3408cb4a153dbc634c04968b9a2ea1fc53 WatchSource:0}: Error finding container 510cc37ecdfd3fb861ea7712d54bad3408cb4a153dbc634c04968b9a2ea1fc53: Status 404 returned error can't find the container with id 510cc37ecdfd3fb861ea7712d54bad3408cb4a153dbc634c04968b9a2ea1fc53
Oct 30 01:55:19.484654 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:19.484624    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"]
Oct 30 01:55:19.513542 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:19.513510    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"]
Oct 30 01:55:19.818699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:19.818650    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49" event={"ID":"c5c9ddbd-6216-401d-ad22-dc4821de1bc2","Type":"ContainerStarted","Data":"448ec9a133ead107c368bec1ada4e3563e6bde55f4812d6ed44ad02c5c8c6667"}
Oct 30 01:55:19.819145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:19.818715    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49" event={"ID":"c5c9ddbd-6216-401d-ad22-dc4821de1bc2","Type":"ContainerStarted","Data":"510cc37ecdfd3fb861ea7712d54bad3408cb4a153dbc634c04968b9a2ea1fc53"}
Oct 30 01:55:19.856436 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:19.856354    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49" podStartSLOduration=1.856340656 podStartE2EDuration="1.856340656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:19.853173045 +0000 UTC m=+4428.602828351" watchObservedRunningTime="2026-10-30 01:55:19.856340656 +0000 UTC m=+4428.605995963"
Oct 30 01:55:20.827866 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:20.827833    4425 generic.go:356] "Generic (PLEG): container finished" podID="c5c9ddbd-6216-401d-ad22-dc4821de1bc2" containerID="448ec9a133ead107c368bec1ada4e3563e6bde55f4812d6ed44ad02c5c8c6667" exitCode=0
Oct 30 01:55:20.828344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:20.827915    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49" event={"ID":"c5c9ddbd-6216-401d-ad22-dc4821de1bc2","Type":"ContainerDied","Data":"448ec9a133ead107c368bec1ada4e3563e6bde55f4812d6ed44ad02c5c8c6667"}
Oct 30 01:55:21.974492 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:21.974466    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49"
Oct 30 01:55:22.033997 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:22.033961    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-maas-api-xks-ca\") pod \"c5c9ddbd-6216-401d-ad22-dc4821de1bc2\" (UID: \"c5c9ddbd-6216-401d-ad22-dc4821de1bc2\") "
Oct 30 01:55:22.034180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:22.034065    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-maas-api-openshift-ca\") pod \"c5c9ddbd-6216-401d-ad22-dc4821de1bc2\" (UID: \"c5c9ddbd-6216-401d-ad22-dc4821de1bc2\") "
Oct 30 01:55:22.034180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:22.034110    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-kube-api-access-dbxf4\" (UniqueName: \"kubernetes.io/projected/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-kube-api-access-dbxf4\") pod \"c5c9ddbd-6216-401d-ad22-dc4821de1bc2\" (UID: \"c5c9ddbd-6216-401d-ad22-dc4821de1bc2\") "
Oct 30 01:55:22.034957 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:22.034692    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-maas-api-openshift-ca" pod "c5c9ddbd-6216-401d-ad22-dc4821de1bc2" (UID: "c5c9ddbd-6216-401d-ad22-dc4821de1bc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:22.040173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:22.035435    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:22.040173 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:22.037675    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-kube-api-access-dbxf4" pod "c5c9ddbd-6216-401d-ad22-dc4821de1bc2" (UID: "c5c9ddbd-6216-401d-ad22-dc4821de1bc2"). InnerVolumeSpecName "kube-api-access-dbxf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:22.050182 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:22.050124    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-maas-api-xks-ca" pod "c5c9ddbd-6216-401d-ad22-dc4821de1bc2" (UID: "c5c9ddbd-6216-401d-ad22-dc4821de1bc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:22.136469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:22.136387    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:22.136469 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:22.136425    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbxf4\" (UniqueName: \"kubernetes.io/projected/c5c9ddbd-6216-401d-ad22-dc4821de1bc2-kube-api-access-dbxf4\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:22.140028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:22.139993    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:55:22.842563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:22.842517    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49"
Oct 30 01:55:22.842563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:22.842546    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-116ece75-d5h49" event={"ID":"c5c9ddbd-6216-401d-ad22-dc4821de1bc2","Type":"ContainerDied","Data":"510cc37ecdfd3fb861ea7712d54bad3408cb4a153dbc634c04968b9a2ea1fc53"}
Oct 30 01:55:22.842783 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:22.842601    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="510cc37ecdfd3fb861ea7712d54bad3408cb4a153dbc634c04968b9a2ea1fc53"
Oct 30 01:55:24.221600 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:24.221568    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"]
Oct 30 01:55:24.226888 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:24.226864    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c666c2-dvjcr"]
Oct 30 01:55:24.485718 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:24.485630    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj" podUID="c44e61e3-669d-4ac3-8686-68f72e4df64c" containerName="payload-processing" containerID="cri-o://5bfc0d0c7e3a421906c18d71ebf1342fceaa91c7253d76364e5a07c7732d27fe" gracePeriod=25
Oct 30 01:55:24.514638 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:24.514604    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx" podUID="60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e" containerName="payload-pre-processing" containerID="cri-o://03642b2e4f65bd9fe504e826e029fff1f74202dc3ba90f4942137a0726d18f9f" gracePeriod=25
Oct 30 01:55:25.421567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:25.421533    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7facc24-9378-4007-959c-a346f03fcced" path="/var/lib/kubelet/pods/e7facc24-9378-4007-959c-a346f03fcced/volumes"
Oct 30 01:55:26.362662 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.362599    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerName="main" containerID="cri-o://efb5688e0c2981dac4d9c1f551158004328395fc7cd8f2b349c21b7c8deb8ade" gracePeriod=45
Oct 30 01:55:26.369276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.369244    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" podUID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" containerName="main" containerID="cri-o://a2f8ad05532131ac2b563286ebb4bbf05c3ee95767e7372c61abe82741feddfd" gracePeriod=45
Oct 30 01:55:26.397513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.397481    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2" podUID="13f688db-25e0-4a88-9d1e-c3fd78b7f659" containerName="main" containerID="cri-o://e3a60e3181e4a5f5f52e0ceb5a8e76bf9fc1d828c354618608640ee44456fd78" gracePeriod=45
Oct 30 01:55:26.669093 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.669058    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj" podUID="c44e61e3-669d-4ac3-8686-68f72e4df64c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:55:26.669547 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.669101    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx" podUID="60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:55:26.673244 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.673219    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 01:55:26.687752 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.687731    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 01:55:26.691265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.691245    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 01:55:26.782059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782027    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-tmp-dir\") pod \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") "
Oct 30 01:55:26.782059 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782067    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-dshm\") pod \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") "
Oct 30 01:55:26.782339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782130    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-home\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-home\") pod \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") "
Oct 30 01:55:26.782339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782204    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-tls-certs\") pod \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") "
Oct 30 01:55:26.782339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782243    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-dshm\") pod \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") "
Oct 30 01:55:26.782339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782266    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-home\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-home\") pod \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") "
Oct 30 01:55:26.782339 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782302    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-home\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-home\") pod \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") "
Oct 30 01:55:26.782610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782345    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-model-cache\") pod \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") "
Oct 30 01:55:26.782610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782378    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-tmp-dir\") pod \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") "
Oct 30 01:55:26.782610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782391    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-tmp-dir" pod "13f688db-25e0-4a88-9d1e-c3fd78b7f659" (UID: "13f688db-25e0-4a88-9d1e-c3fd78b7f659"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:26.782610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782412    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-kube-api-access-24sdw\" (UniqueName: \"kubernetes.io/projected/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-kube-api-access-24sdw\") pod \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") "
Oct 30 01:55:26.782610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782462    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b18eab-634a-4cc9-8f05-10a2e3b937f2-kube-api-access-rhkzb\" (UniqueName: \"kubernetes.io/projected/16b18eab-634a-4cc9-8f05-10a2e3b937f2-kube-api-access-rhkzb\") pod \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") "
Oct 30 01:55:26.782610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782503    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-tmp-dir\") pod \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") "
Oct 30 01:55:26.782610 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782555    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16b18eab-634a-4cc9-8f05-10a2e3b937f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/16b18eab-634a-4cc9-8f05-10a2e3b937f2-tls-certs\") pod \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") "
Oct 30 01:55:26.782974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782613    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-home" pod "16b18eab-634a-4cc9-8f05-10a2e3b937f2" (UID: "16b18eab-634a-4cc9-8f05-10a2e3b937f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:26.782974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782813    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-home" pod "4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" (UID: "4bc38761-a8e2-4862-b37a-5bcc4e05f3ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:26.782974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782861    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13f688db-25e0-4a88-9d1e-c3fd78b7f659-tls-certs\" (UniqueName: \"kubernetes.io/secret/13f688db-25e0-4a88-9d1e-c3fd78b7f659-tls-certs\") pod \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") "
Oct 30 01:55:26.782974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782942    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-home" pod "13f688db-25e0-4a88-9d1e-c3fd78b7f659" (UID: "13f688db-25e0-4a88-9d1e-c3fd78b7f659"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:26.782974 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.782960    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-model-cache\") pod \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\" (UID: \"16b18eab-634a-4cc9-8f05-10a2e3b937f2\") "
Oct 30 01:55:26.783271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.783003    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f688db-25e0-4a88-9d1e-c3fd78b7f659-kube-api-access-f4zdd\" (UniqueName: \"kubernetes.io/projected/13f688db-25e0-4a88-9d1e-c3fd78b7f659-kube-api-access-f4zdd\") pod \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\" (UID: \"13f688db-25e0-4a88-9d1e-c3fd78b7f659\") "
Oct 30 01:55:26.783271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.783042    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-dshm\") pod \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") "
Oct 30 01:55:26.783271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.783073    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-model-cache\") pod \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\" (UID: \"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec\") "
Oct 30 01:55:26.783271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.783186    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-model-cache" pod "13f688db-25e0-4a88-9d1e-c3fd78b7f659" (UID: "13f688db-25e0-4a88-9d1e-c3fd78b7f659"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:26.783465 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.783421    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-tmp-dir" pod "16b18eab-634a-4cc9-8f05-10a2e3b937f2" (UID: "16b18eab-634a-4cc9-8f05-10a2e3b937f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:26.783637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.783615    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.783730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.783644    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.783730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.783655    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.783730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.783657    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-tmp-dir" pod "4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" (UID: "4bc38761-a8e2-4862-b37a-5bcc4e05f3ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:26.783730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.783667    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.783730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.783704    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.783730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.783724    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.784067 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.783866    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-model-cache" pod "4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" (UID: "4bc38761-a8e2-4862-b37a-5bcc4e05f3ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:26.784132 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.784076    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-model-cache" pod "16b18eab-634a-4cc9-8f05-10a2e3b937f2" (UID: "16b18eab-634a-4cc9-8f05-10a2e3b937f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:26.786392 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.786353    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-dshm" pod "4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" (UID: "4bc38761-a8e2-4862-b37a-5bcc4e05f3ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:26.786535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.786452    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-dshm" pod "16b18eab-634a-4cc9-8f05-10a2e3b937f2" (UID: "16b18eab-634a-4cc9-8f05-10a2e3b937f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:26.786740 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.786703    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-kube-api-access-24sdw" pod "4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" (UID: "4bc38761-a8e2-4862-b37a-5bcc4e05f3ec"). InnerVolumeSpecName "kube-api-access-24sdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:26.786851 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.786800    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f688db-25e0-4a88-9d1e-c3fd78b7f659-kube-api-access-f4zdd" pod "13f688db-25e0-4a88-9d1e-c3fd78b7f659" (UID: "13f688db-25e0-4a88-9d1e-c3fd78b7f659"). InnerVolumeSpecName "kube-api-access-f4zdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:26.787311 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.787287    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f688db-25e0-4a88-9d1e-c3fd78b7f659-tls-certs" pod "13f688db-25e0-4a88-9d1e-c3fd78b7f659" (UID: "13f688db-25e0-4a88-9d1e-c3fd78b7f659"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:26.787748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.787719    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-dshm" pod "13f688db-25e0-4a88-9d1e-c3fd78b7f659" (UID: "13f688db-25e0-4a88-9d1e-c3fd78b7f659"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:26.788701 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.788673    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b18eab-634a-4cc9-8f05-10a2e3b937f2-kube-api-access-rhkzb" pod "16b18eab-634a-4cc9-8f05-10a2e3b937f2" (UID: "16b18eab-634a-4cc9-8f05-10a2e3b937f2"). InnerVolumeSpecName "kube-api-access-rhkzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:26.789245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.789219    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16b18eab-634a-4cc9-8f05-10a2e3b937f2-tls-certs" pod "16b18eab-634a-4cc9-8f05-10a2e3b937f2" (UID: "16b18eab-634a-4cc9-8f05-10a2e3b937f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:26.789913 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.789871    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-tls-certs" pod "4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" (UID: "4bc38761-a8e2-4862-b37a-5bcc4e05f3ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:26.870820 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.870785    4425 generic.go:356] "Generic (PLEG): container finished" podID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerID="efb5688e0c2981dac4d9c1f551158004328395fc7cd8f2b349c21b7c8deb8ade" exitCode=0
Oct 30 01:55:26.871014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.870833    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"
Oct 30 01:55:26.871014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.870869    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" event={"ID":"16b18eab-634a-4cc9-8f05-10a2e3b937f2","Type":"ContainerDied","Data":"efb5688e0c2981dac4d9c1f551158004328395fc7cd8f2b349c21b7c8deb8ade"}
Oct 30 01:55:26.871014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.870913    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j" event={"ID":"16b18eab-634a-4cc9-8f05-10a2e3b937f2","Type":"ContainerDied","Data":"f35a5a639e2054cc98fd3f5f8d67a90a5aa831c4e76bb0f45e19e52aa417130d"}
Oct 30 01:55:26.871014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.870931    4425 scope.go:122] "RemoveContainer" containerID="efb5688e0c2981dac4d9c1f551158004328395fc7cd8f2b349c21b7c8deb8ade"
Oct 30 01:55:26.872603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.872584    4425 generic.go:356] "Generic (PLEG): container finished" podID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" containerID="a2f8ad05532131ac2b563286ebb4bbf05c3ee95767e7372c61abe82741feddfd" exitCode=0
Oct 30 01:55:26.872731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.872642    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"
Oct 30 01:55:26.872731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.872662    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" event={"ID":"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec","Type":"ContainerDied","Data":"a2f8ad05532131ac2b563286ebb4bbf05c3ee95767e7372c61abe82741feddfd"}
Oct 30 01:55:26.872731 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.872699    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk" event={"ID":"4bc38761-a8e2-4862-b37a-5bcc4e05f3ec","Type":"ContainerDied","Data":"19a0477f9899e2ab0e83b78d420fbafb01d16da5a4478efa214e982d26da482f"}
Oct 30 01:55:26.874047 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.874019    4425 generic.go:356] "Generic (PLEG): container finished" podID="13f688db-25e0-4a88-9d1e-c3fd78b7f659" containerID="e3a60e3181e4a5f5f52e0ceb5a8e76bf9fc1d828c354618608640ee44456fd78" exitCode=0
Oct 30 01:55:26.874186 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.874136    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2" event={"ID":"13f688db-25e0-4a88-9d1e-c3fd78b7f659","Type":"ContainerDied","Data":"e3a60e3181e4a5f5f52e0ceb5a8e76bf9fc1d828c354618608640ee44456fd78"}
Oct 30 01:55:26.874276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.874222    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2" event={"ID":"13f688db-25e0-4a88-9d1e-c3fd78b7f659","Type":"ContainerDied","Data":"23f5e141ca0b6f87d77044b423f564af06fb6cf6c3f61b7457ae8b4e3b02be96"}
Oct 30 01:55:26.874344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.874314    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"
Oct 30 01:55:26.883093 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.882889    4425 scope.go:122] "RemoveContainer" containerID="efb5688e0c2981dac4d9c1f551158004328395fc7cd8f2b349c21b7c8deb8ade"
Oct 30 01:55:26.883424 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:55:26.883379    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb5688e0c2981dac4d9c1f551158004328395fc7cd8f2b349c21b7c8deb8ade\": container with ID starting with efb5688e0c2981dac4d9c1f551158004328395fc7cd8f2b349c21b7c8deb8ade not found: ID does not exist" containerID="efb5688e0c2981dac4d9c1f551158004328395fc7cd8f2b349c21b7c8deb8ade"
Oct 30 01:55:26.883545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.883421    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb5688e0c2981dac4d9c1f551158004328395fc7cd8f2b349c21b7c8deb8ade"} err="failed to get container status \"efb5688e0c2981dac4d9c1f551158004328395fc7cd8f2b349c21b7c8deb8ade\": rpc error: code = NotFound desc = could not find container \"efb5688e0c2981dac4d9c1f551158004328395fc7cd8f2b349c21b7c8deb8ade\": container with ID starting with efb5688e0c2981dac4d9c1f551158004328395fc7cd8f2b349c21b7c8deb8ade not found: ID does not exist"
Oct 30 01:55:26.883545 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.883443    4425 scope.go:122] "RemoveContainer" containerID="a2f8ad05532131ac2b563286ebb4bbf05c3ee95767e7372c61abe82741feddfd"
Oct 30 01:55:26.884591 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.884559    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4zdd\" (UniqueName: \"kubernetes.io/projected/13f688db-25e0-4a88-9d1e-c3fd78b7f659-kube-api-access-f4zdd\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.884705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.884599    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.884705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.884615    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.884705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.884628    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.884705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.884638    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.884705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.884649    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13f688db-25e0-4a88-9d1e-c3fd78b7f659-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.884705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.884659    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.884705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.884672    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24sdw\" (UniqueName: \"kubernetes.io/projected/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec-kube-api-access-24sdw\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.884705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.884683    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhkzb\" (UniqueName: \"kubernetes.io/projected/16b18eab-634a-4cc9-8f05-10a2e3b937f2-kube-api-access-rhkzb\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.884705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.884694    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16b18eab-634a-4cc9-8f05-10a2e3b937f2-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.884705 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.884706    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13f688db-25e0-4a88-9d1e-c3fd78b7f659-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.885398 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.884716    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16b18eab-634a-4cc9-8f05-10a2e3b937f2-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:26.893222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.893202    4425 scope.go:122] "RemoveContainer" containerID="a2f8ad05532131ac2b563286ebb4bbf05c3ee95767e7372c61abe82741feddfd"
Oct 30 01:55:26.893483 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:55:26.893466    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f8ad05532131ac2b563286ebb4bbf05c3ee95767e7372c61abe82741feddfd\": container with ID starting with a2f8ad05532131ac2b563286ebb4bbf05c3ee95767e7372c61abe82741feddfd not found: ID does not exist" containerID="a2f8ad05532131ac2b563286ebb4bbf05c3ee95767e7372c61abe82741feddfd"
Oct 30 01:55:26.893563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.893494    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f8ad05532131ac2b563286ebb4bbf05c3ee95767e7372c61abe82741feddfd"} err="failed to get container status \"a2f8ad05532131ac2b563286ebb4bbf05c3ee95767e7372c61abe82741feddfd\": rpc error: code = NotFound desc = could not find container \"a2f8ad05532131ac2b563286ebb4bbf05c3ee95767e7372c61abe82741feddfd\": container with ID starting with a2f8ad05532131ac2b563286ebb4bbf05c3ee95767e7372c61abe82741feddfd not found: ID does not exist"
Oct 30 01:55:26.893563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.893518    4425 scope.go:122] "RemoveContainer" containerID="e3a60e3181e4a5f5f52e0ceb5a8e76bf9fc1d828c354618608640ee44456fd78"
Oct 30 01:55:26.902651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.902632    4425 scope.go:122] "RemoveContainer" containerID="e3a60e3181e4a5f5f52e0ceb5a8e76bf9fc1d828c354618608640ee44456fd78"
Oct 30 01:55:26.902936 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:55:26.902908    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a60e3181e4a5f5f52e0ceb5a8e76bf9fc1d828c354618608640ee44456fd78\": container with ID starting with e3a60e3181e4a5f5f52e0ceb5a8e76bf9fc1d828c354618608640ee44456fd78 not found: ID does not exist" containerID="e3a60e3181e4a5f5f52e0ceb5a8e76bf9fc1d828c354618608640ee44456fd78"
Oct 30 01:55:26.903022 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.902944    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a60e3181e4a5f5f52e0ceb5a8e76bf9fc1d828c354618608640ee44456fd78"} err="failed to get container status \"e3a60e3181e4a5f5f52e0ceb5a8e76bf9fc1d828c354618608640ee44456fd78\": rpc error: code = NotFound desc = could not find container \"e3a60e3181e4a5f5f52e0ceb5a8e76bf9fc1d828c354618608640ee44456fd78\": container with ID starting with e3a60e3181e4a5f5f52e0ceb5a8e76bf9fc1d828c354618608640ee44456fd78 not found: ID does not exist"
Oct 30 01:55:26.940828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.940800    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"]
Oct 30 01:55:26.945093 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.945065    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/distinct-w1-664c2b-kserve-7555dc55d9-cxvsk"]
Oct 30 01:55:26.973908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.973887    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"]
Oct 30 01:55:26.980865 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:26.980844    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/distinct-2-w1-664c2b-kserve-5c78796994-5qr9j"]
Oct 30 01:55:27.009741 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:27.009719    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"]
Oct 30 01:55:27.016637 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:27.016618    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-664c2b/simulator-w1-664c2b-kserve-5d9fc9597f-jqdt2"]
Oct 30 01:55:27.422200 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:27.422146    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13f688db-25e0-4a88-9d1e-c3fd78b7f659" path="/var/lib/kubelet/pods/13f688db-25e0-4a88-9d1e-c3fd78b7f659/volumes"
Oct 30 01:55:27.422686 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:27.422666    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" path="/var/lib/kubelet/pods/16b18eab-634a-4cc9-8f05-10a2e3b937f2/volumes"
Oct 30 01:55:27.423138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:27.423118    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" path="/var/lib/kubelet/pods/4bc38761-a8e2-4862-b37a-5bcc4e05f3ec/volumes"
Oct 30 01:55:28.054174 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.054114    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc" podUID="c8d1f900-b8c4-43dc-814e-9491e560ea3f" containerName="main" containerID="cri-o://dd892509fabdc34976778242280e1b11177b8a51e602f6a706618a784e287036" gracePeriod=45
Oct 30 01:55:28.070726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.070652    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z" podUID="b4ed47ff-0328-4645-be5f-6a6d51a94fc8" containerName="main" containerID="cri-o://71280ea5a3fbd2368220d0cdb817112130d7413f6c3f37390d7d5a7c59cb5796" gracePeriod=45
Oct 30 01:55:28.216032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.215992    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc" podUID="c8d1f900-b8c4-43dc-814e-9491e560ea3f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": dial tcp 10.133.0.47:8000: connect: connection refused"
Oct 30 01:55:28.318688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.318622    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 01:55:28.329175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.329139    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 01:55:28.402536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.402501    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-dshm\") pod \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") "
Oct 30 01:55:28.402694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.402574    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-tmp-dir\") pod \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") "
Oct 30 01:55:28.402694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.402613    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-model-cache\") pod \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") "
Oct 30 01:55:28.402694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.402645    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8d1f900-b8c4-43dc-814e-9491e560ea3f-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8d1f900-b8c4-43dc-814e-9491e560ea3f-tls-certs\") pod \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") "
Oct 30 01:55:28.402694 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.402669    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-dshm\") pod \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") "
Oct 30 01:55:28.402900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.402719    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-tmp-dir\") pod \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") "
Oct 30 01:55:28.402900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.402757    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-home\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-home\") pod \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") "
Oct 30 01:55:28.402900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.402829    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d1f900-b8c4-43dc-814e-9491e560ea3f-kube-api-access-kmznk\" (UniqueName: \"kubernetes.io/projected/c8d1f900-b8c4-43dc-814e-9491e560ea3f-kube-api-access-kmznk\") pod \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\" (UID: \"c8d1f900-b8c4-43dc-814e-9491e560ea3f\") "
Oct 30 01:55:28.402900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.402886    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-kube-api-access-pp75j\" (UniqueName: \"kubernetes.io/projected/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-kube-api-access-pp75j\") pod \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") "
Oct 30 01:55:28.403045 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.402921    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-tls-certs\") pod \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") "
Oct 30 01:55:28.403045 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.403029    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-model-cache\") pod \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") "
Oct 30 01:55:28.403107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.403058    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-home\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-home\") pod \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\" (UID: \"b4ed47ff-0328-4645-be5f-6a6d51a94fc8\") "
Oct 30 01:55:28.403765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.403737    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-home" pod "b4ed47ff-0328-4645-be5f-6a6d51a94fc8" (UID: "b4ed47ff-0328-4645-be5f-6a6d51a94fc8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:28.404340 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.404305    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-tmp-dir" pod "c8d1f900-b8c4-43dc-814e-9491e560ea3f" (UID: "c8d1f900-b8c4-43dc-814e-9491e560ea3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:28.404464 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.404361    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-tmp-dir" pod "b4ed47ff-0328-4645-be5f-6a6d51a94fc8" (UID: "b4ed47ff-0328-4645-be5f-6a6d51a94fc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:28.404464 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.404395    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-home" pod "c8d1f900-b8c4-43dc-814e-9491e560ea3f" (UID: "c8d1f900-b8c4-43dc-814e-9491e560ea3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:28.404764 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.404729    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-model-cache" pod "c8d1f900-b8c4-43dc-814e-9491e560ea3f" (UID: "c8d1f900-b8c4-43dc-814e-9491e560ea3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:28.404984 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.404960    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-model-cache" pod "b4ed47ff-0328-4645-be5f-6a6d51a94fc8" (UID: "b4ed47ff-0328-4645-be5f-6a6d51a94fc8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:28.407072 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.407046    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-dshm" pod "c8d1f900-b8c4-43dc-814e-9491e560ea3f" (UID: "c8d1f900-b8c4-43dc-814e-9491e560ea3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:28.407541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.407502    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-tls-certs" pod "b4ed47ff-0328-4645-be5f-6a6d51a94fc8" (UID: "b4ed47ff-0328-4645-be5f-6a6d51a94fc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:28.407774 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.407738    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d1f900-b8c4-43dc-814e-9491e560ea3f-tls-certs" pod "c8d1f900-b8c4-43dc-814e-9491e560ea3f" (UID: "c8d1f900-b8c4-43dc-814e-9491e560ea3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:28.409343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.409089    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-dshm" pod "b4ed47ff-0328-4645-be5f-6a6d51a94fc8" (UID: "b4ed47ff-0328-4645-be5f-6a6d51a94fc8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:28.409343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.409210    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-kube-api-access-pp75j" pod "b4ed47ff-0328-4645-be5f-6a6d51a94fc8" (UID: "b4ed47ff-0328-4645-be5f-6a6d51a94fc8"). InnerVolumeSpecName "kube-api-access-pp75j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:28.409343 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.409256    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d1f900-b8c4-43dc-814e-9491e560ea3f-kube-api-access-kmznk" pod "c8d1f900-b8c4-43dc-814e-9491e560ea3f" (UID: "c8d1f900-b8c4-43dc-814e-9491e560ea3f"). InnerVolumeSpecName "kube-api-access-kmznk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:28.504112 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.504081    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.504112 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.504108    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.504363 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.504121    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8d1f900-b8c4-43dc-814e-9491e560ea3f-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.504363 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.504132    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.504363 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.504144    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.504363 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.504169    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8d1f900-b8c4-43dc-814e-9491e560ea3f-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.504363 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.504203    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmznk\" (UniqueName: \"kubernetes.io/projected/c8d1f900-b8c4-43dc-814e-9491e560ea3f-kube-api-access-kmznk\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.504363 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.504217    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp75j\" (UniqueName: \"kubernetes.io/projected/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-kube-api-access-pp75j\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.504363 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.504230    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.504363 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.504246    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.504363 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.504255    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.504363 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.504266    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4ed47ff-0328-4645-be5f-6a6d51a94fc8-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:28.892666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.892634    4425 generic.go:356] "Generic (PLEG): container finished" podID="c8d1f900-b8c4-43dc-814e-9491e560ea3f" containerID="dd892509fabdc34976778242280e1b11177b8a51e602f6a706618a784e287036" exitCode=0
Oct 30 01:55:28.892828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.892676    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"
Oct 30 01:55:28.892828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.892720    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc" event={"ID":"c8d1f900-b8c4-43dc-814e-9491e560ea3f","Type":"ContainerDied","Data":"dd892509fabdc34976778242280e1b11177b8a51e602f6a706618a784e287036"}
Oct 30 01:55:28.892828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.892758    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc" event={"ID":"c8d1f900-b8c4-43dc-814e-9491e560ea3f","Type":"ContainerDied","Data":"0ab026a093cf666df6998d3effd5932a357c871fb73c5b38e5bc83c213fba5a2"}
Oct 30 01:55:28.892828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.892774    4425 scope.go:122] "RemoveContainer" containerID="dd892509fabdc34976778242280e1b11177b8a51e602f6a706618a784e287036"
Oct 30 01:55:28.894291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.894266    4425 generic.go:356] "Generic (PLEG): container finished" podID="b4ed47ff-0328-4645-be5f-6a6d51a94fc8" containerID="71280ea5a3fbd2368220d0cdb817112130d7413f6c3f37390d7d5a7c59cb5796" exitCode=0
Oct 30 01:55:28.894413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.894314    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"
Oct 30 01:55:28.894413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.894345    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z" event={"ID":"b4ed47ff-0328-4645-be5f-6a6d51a94fc8","Type":"ContainerDied","Data":"71280ea5a3fbd2368220d0cdb817112130d7413f6c3f37390d7d5a7c59cb5796"}
Oct 30 01:55:28.894413 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.894387    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z" event={"ID":"b4ed47ff-0328-4645-be5f-6a6d51a94fc8","Type":"ContainerDied","Data":"a4997ed4ea51745a47e80e65fe58361c0680f61e07600b22056efde89c3edb3c"}
Oct 30 01:55:28.904280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.904234    4425 scope.go:122] "RemoveContainer" containerID="dd892509fabdc34976778242280e1b11177b8a51e602f6a706618a784e287036"
Oct 30 01:55:28.904578 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:55:28.904555    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd892509fabdc34976778242280e1b11177b8a51e602f6a706618a784e287036\": container with ID starting with dd892509fabdc34976778242280e1b11177b8a51e602f6a706618a784e287036 not found: ID does not exist" containerID="dd892509fabdc34976778242280e1b11177b8a51e602f6a706618a784e287036"
Oct 30 01:55:28.904647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.904587    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd892509fabdc34976778242280e1b11177b8a51e602f6a706618a784e287036"} err="failed to get container status \"dd892509fabdc34976778242280e1b11177b8a51e602f6a706618a784e287036\": rpc error: code = NotFound desc = could not find container \"dd892509fabdc34976778242280e1b11177b8a51e602f6a706618a784e287036\": container with ID starting with dd892509fabdc34976778242280e1b11177b8a51e602f6a706618a784e287036 not found: ID does not exist"
Oct 30 01:55:28.904647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.904610    4425 scope.go:122] "RemoveContainer" containerID="71280ea5a3fbd2368220d0cdb817112130d7413f6c3f37390d7d5a7c59cb5796"
Oct 30 01:55:28.913190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.913144    4425 scope.go:122] "RemoveContainer" containerID="71280ea5a3fbd2368220d0cdb817112130d7413f6c3f37390d7d5a7c59cb5796"
Oct 30 01:55:28.913792 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:55:28.913770    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71280ea5a3fbd2368220d0cdb817112130d7413f6c3f37390d7d5a7c59cb5796\": container with ID starting with 71280ea5a3fbd2368220d0cdb817112130d7413f6c3f37390d7d5a7c59cb5796 not found: ID does not exist" containerID="71280ea5a3fbd2368220d0cdb817112130d7413f6c3f37390d7d5a7c59cb5796"
Oct 30 01:55:28.913841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.913799    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71280ea5a3fbd2368220d0cdb817112130d7413f6c3f37390d7d5a7c59cb5796"} err="failed to get container status \"71280ea5a3fbd2368220d0cdb817112130d7413f6c3f37390d7d5a7c59cb5796\": rpc error: code = NotFound desc = could not find container \"71280ea5a3fbd2368220d0cdb817112130d7413f6c3f37390d7d5a7c59cb5796\": container with ID starting with 71280ea5a3fbd2368220d0cdb817112130d7413f6c3f37390d7d5a7c59cb5796 not found: ID does not exist"
Oct 30 01:55:28.940671 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.940646    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"]
Oct 30 01:55:28.951680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.951656    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/premium-w2-72ed13-kserve-5b66f7f4cb-9vdjc"]
Oct 30 01:55:28.971537 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.971517    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"]
Oct 30 01:55:28.976092 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:28.976073    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-72ed13/simulator-w2-72ed13-kserve-554f57dbc7-mxv7z"]
Oct 30 01:55:29.365530 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.365497    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s"]
Oct 30 01:55:29.365954 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.365941    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c8d1f900-b8c4-43dc-814e-9491e560ea3f" containerName="main"
Oct 30 01:55:29.366014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.365954    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d1f900-b8c4-43dc-814e-9491e560ea3f" containerName="main"
Oct 30 01:55:29.366014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.365974    4425 cpu_manager.go:414] "containerMap: removing container" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerName="main"
Oct 30 01:55:29.366014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.365979    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerName="main"
Oct 30 01:55:29.366014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.365990    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c5c9ddbd-6216-401d-ad22-dc4821de1bc2" containerName="revoke-keys"
Oct 30 01:55:29.366014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.365995    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c9ddbd-6216-401d-ad22-dc4821de1bc2" containerName="revoke-keys"
Oct 30 01:55:29.366014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.366004    4425 cpu_manager.go:414] "containerMap: removing container" podUID="13f688db-25e0-4a88-9d1e-c3fd78b7f659" containerName="main"
Oct 30 01:55:29.366014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.366009    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f688db-25e0-4a88-9d1e-c3fd78b7f659" containerName="main"
Oct 30 01:55:29.366254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.366029    4425 cpu_manager.go:414] "containerMap: removing container" podUID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" containerName="main"
Oct 30 01:55:29.366254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.366034    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" containerName="main"
Oct 30 01:55:29.366254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.366044    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b4ed47ff-0328-4645-be5f-6a6d51a94fc8" containerName="main"
Oct 30 01:55:29.366254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.366049    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ed47ff-0328-4645-be5f-6a6d51a94fc8" containerName="main"
Oct 30 01:55:29.366254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.366117    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc38761-a8e2-4862-b37a-5bcc4e05f3ec" containerName="main"
Oct 30 01:55:29.366254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.366127    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ed47ff-0328-4645-be5f-6a6d51a94fc8" containerName="main"
Oct 30 01:55:29.366254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.366134    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d1f900-b8c4-43dc-814e-9491e560ea3f" containerName="main"
Oct 30 01:55:29.366254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.366141    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b18eab-634a-4cc9-8f05-10a2e3b937f2" containerName="main"
Oct 30 01:55:29.366254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.366148    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c9ddbd-6216-401d-ad22-dc4821de1bc2" containerName="revoke-keys"
Oct 30 01:55:29.366254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.366168    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f688db-25e0-4a88-9d1e-c3fd78b7f659" containerName="main"
Oct 30 01:55:29.370732 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.370710    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s"
Oct 30 01:55:29.386777 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.386755    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:55:29.387333 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.387313    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:55:29.387333 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.387347    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:55:29.387520 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.387386    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:55:29.400233 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.400209    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s"]
Oct 30 01:55:29.404520 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.404475    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm"]
Oct 30 01:55:29.409197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.409118    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm"
Oct 30 01:55:29.425999 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.425973    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ed47ff-0328-4645-be5f-6a6d51a94fc8" path="/var/lib/kubelet/pods/b4ed47ff-0328-4645-be5f-6a6d51a94fc8/volumes"
Oct 30 01:55:29.426616 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.426592    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8d1f900-b8c4-43dc-814e-9491e560ea3f" path="/var/lib/kubelet/pods/c8d1f900-b8c4-43dc-814e-9491e560ea3f/volumes"
Oct 30 01:55:29.440765 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.440737    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm"]
Oct 30 01:55:29.470192 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.470145    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf"]
Oct 30 01:55:29.474191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.474147    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf"
Oct 30 01:55:29.501661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.501613    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf"]
Oct 30 01:55:29.515211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.515139    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1279943c-c7glm\" (UID: \"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm"
Oct 30 01:55:29.515352 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.515310    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f615383-3fb8-4e8e-8681-465e88538678-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0684d5d8-hfw4s\" (UID: \"3f615383-3fb8-4e8e-8681-465e88538678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s"
Oct 30 01:55:29.515514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.515494    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw2h5\" (UniqueName: \"kubernetes.io/projected/3f615383-3fb8-4e8e-8681-465e88538678-kube-api-access-gw2h5\") pod \"maas-api-revoke-sub-0684d5d8-hfw4s\" (UID: \"3f615383-3fb8-4e8e-8681-465e88538678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s"
Oct 30 01:55:29.515574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.515560    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1279943c-c7glm\" (UID: \"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm"
Oct 30 01:55:29.515699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.515673    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f615383-3fb8-4e8e-8681-465e88538678-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0684d5d8-hfw4s\" (UID: \"3f615383-3fb8-4e8e-8681-465e88538678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s"
Oct 30 01:55:29.515829 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.515811    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sswq\" (UniqueName: \"kubernetes.io/projected/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-kube-api-access-8sswq\") pod \"maas-api-revoke-sub-1279943c-c7glm\" (UID: \"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm"
Oct 30 01:55:29.616953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.616869    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1279943c-c7glm\" (UID: \"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm"
Oct 30 01:55:29.616953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.616915    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c65fec65-f9ed-410d-abd0-f9ac048defe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6502d6e-tgwhf\" (UID: \"c65fec65-f9ed-410d-abd0-f9ac048defe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf"
Oct 30 01:55:29.616953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.616949    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f615383-3fb8-4e8e-8681-465e88538678-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0684d5d8-hfw4s\" (UID: \"3f615383-3fb8-4e8e-8681-465e88538678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s"
Oct 30 01:55:29.617267 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.616979    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c65fec65-f9ed-410d-abd0-f9ac048defe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6502d6e-tgwhf\" (UID: \"c65fec65-f9ed-410d-abd0-f9ac048defe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf"
Oct 30 01:55:29.617267 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.617194    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw2h5\" (UniqueName: \"kubernetes.io/projected/3f615383-3fb8-4e8e-8681-465e88538678-kube-api-access-gw2h5\") pod \"maas-api-revoke-sub-0684d5d8-hfw4s\" (UID: \"3f615383-3fb8-4e8e-8681-465e88538678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s"
Oct 30 01:55:29.617267 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.617253    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1279943c-c7glm\" (UID: \"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm"
Oct 30 01:55:29.617430 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.617324    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f615383-3fb8-4e8e-8681-465e88538678-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0684d5d8-hfw4s\" (UID: \"3f615383-3fb8-4e8e-8681-465e88538678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s"
Oct 30 01:55:29.617430 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.617363    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl2fr\" (UniqueName: \"kubernetes.io/projected/c65fec65-f9ed-410d-abd0-f9ac048defe6-kube-api-access-dl2fr\") pod \"maas-api-revoke-sub-b6502d6e-tgwhf\" (UID: \"c65fec65-f9ed-410d-abd0-f9ac048defe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf"
Oct 30 01:55:29.617542 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.617443    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sswq\" (UniqueName: \"kubernetes.io/projected/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-kube-api-access-8sswq\") pod \"maas-api-revoke-sub-1279943c-c7glm\" (UID: \"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm"
Oct 30 01:55:29.617771 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.617740    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1279943c-c7glm\" (UID: \"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm"
Oct 30 01:55:29.618105 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.618079    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f615383-3fb8-4e8e-8681-465e88538678-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0684d5d8-hfw4s\" (UID: \"3f615383-3fb8-4e8e-8681-465e88538678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s"
Oct 30 01:55:29.619462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.619437    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1279943c-c7glm\" (UID: \"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm"
Oct 30 01:55:29.619586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.619563    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f615383-3fb8-4e8e-8681-465e88538678-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0684d5d8-hfw4s\" (UID: \"3f615383-3fb8-4e8e-8681-465e88538678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s"
Oct 30 01:55:29.637971 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.637947    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sswq\" (UniqueName: \"kubernetes.io/projected/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-kube-api-access-8sswq\") pod \"maas-api-revoke-sub-1279943c-c7glm\" (UID: \"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm"
Oct 30 01:55:29.638139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.638119    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw2h5\" (UniqueName: \"kubernetes.io/projected/3f615383-3fb8-4e8e-8681-465e88538678-kube-api-access-gw2h5\") pod \"maas-api-revoke-sub-0684d5d8-hfw4s\" (UID: \"3f615383-3fb8-4e8e-8681-465e88538678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s"
Oct 30 01:55:29.682726 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.682693    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"]
Oct 30 01:55:29.685529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.685500    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s"
Oct 30 01:55:29.718272 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.718239    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2fr\" (UniqueName: \"kubernetes.io/projected/c65fec65-f9ed-410d-abd0-f9ac048defe6-kube-api-access-dl2fr\") pod \"maas-api-revoke-sub-b6502d6e-tgwhf\" (UID: \"c65fec65-f9ed-410d-abd0-f9ac048defe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf"
Oct 30 01:55:29.718412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.718353    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c65fec65-f9ed-410d-abd0-f9ac048defe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6502d6e-tgwhf\" (UID: \"c65fec65-f9ed-410d-abd0-f9ac048defe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf"
Oct 30 01:55:29.718412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.718393    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c65fec65-f9ed-410d-abd0-f9ac048defe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6502d6e-tgwhf\" (UID: \"c65fec65-f9ed-410d-abd0-f9ac048defe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf"
Oct 30 01:55:29.719119 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.719097    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c65fec65-f9ed-410d-abd0-f9ac048defe6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6502d6e-tgwhf\" (UID: \"c65fec65-f9ed-410d-abd0-f9ac048defe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf"
Oct 30 01:55:29.721284 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.721233    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c65fec65-f9ed-410d-abd0-f9ac048defe6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6502d6e-tgwhf\" (UID: \"c65fec65-f9ed-410d-abd0-f9ac048defe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf"
Oct 30 01:55:29.726190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.726142    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm"
Oct 30 01:55:29.740265 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.740213    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl2fr\" (UniqueName: \"kubernetes.io/projected/c65fec65-f9ed-410d-abd0-f9ac048defe6-kube-api-access-dl2fr\") pod \"maas-api-revoke-sub-b6502d6e-tgwhf\" (UID: \"c65fec65-f9ed-410d-abd0-f9ac048defe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf"
Oct 30 01:55:29.787403 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:29.787366    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf"
Oct 30 01:55:30.012427 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:30.012381    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s"]
Oct 30 01:55:30.013289 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:55:30.013238    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f615383_3fb8_4e8e_8681_465e88538678.slice/crio-44fa32721ea0ace5b4408ac892c23e74bfa06c4fd2c81a4454b49ba2a81a39c1 WatchSource:0}: Error finding container 44fa32721ea0ace5b4408ac892c23e74bfa06c4fd2c81a4454b49ba2a81a39c1: Status 404 returned error can't find the container with id 44fa32721ea0ace5b4408ac892c23e74bfa06c4fd2c81a4454b49ba2a81a39c1
Oct 30 01:55:30.101144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:30.100178    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm"]
Oct 30 01:55:30.103487 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:55:30.103451    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefab5e9d_fc6d_45b2_bdee_4d1c73ffa7e7.slice/crio-1c9c16390d07bf6a7edadc0c1edf5ba284b2560b1cae6dfcbf16bbad045e430b WatchSource:0}: Error finding container 1c9c16390d07bf6a7edadc0c1edf5ba284b2560b1cae6dfcbf16bbad045e430b: Status 404 returned error can't find the container with id 1c9c16390d07bf6a7edadc0c1edf5ba284b2560b1cae6dfcbf16bbad045e430b
Oct 30 01:55:30.143975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:30.143949    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf"]
Oct 30 01:55:30.144776 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:55:30.144752    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65fec65_f9ed_410d_abd0_f9ac048defe6.slice/crio-89dfb28dd8112b150edf2a6c97347c311b0b9e851bb796a4207f319975200ce3 WatchSource:0}: Error finding container 89dfb28dd8112b150edf2a6c97347c311b0b9e851bb796a4207f319975200ce3: Status 404 returned error can't find the container with id 89dfb28dd8112b150edf2a6c97347c311b0b9e851bb796a4207f319975200ce3
Oct 30 01:55:30.937721 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:30.937682    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm" event={"ID":"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7","Type":"ContainerStarted","Data":"4ea649ec049944707b675b5259402fb05efb1bafb9c52b6ceff98c1d7b850a60"}
Oct 30 01:55:30.938111 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:30.937747    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm" event={"ID":"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7","Type":"ContainerStarted","Data":"1c9c16390d07bf6a7edadc0c1edf5ba284b2560b1cae6dfcbf16bbad045e430b"}
Oct 30 01:55:30.939451 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:30.939424    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf" event={"ID":"c65fec65-f9ed-410d-abd0-f9ac048defe6","Type":"ContainerStarted","Data":"acabce4015b06e8eb4bb9292a6adcf2cb8341d71c7dfb57d508d64b7fae564db"}
Oct 30 01:55:30.939553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:30.939467    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf" event={"ID":"c65fec65-f9ed-410d-abd0-f9ac048defe6","Type":"ContainerStarted","Data":"89dfb28dd8112b150edf2a6c97347c311b0b9e851bb796a4207f319975200ce3"}
Oct 30 01:55:30.941084 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:30.941059    4425 generic.go:356] "Generic (PLEG): container finished" podID="3f615383-3fb8-4e8e-8681-465e88538678" containerID="8a631de2fbbe834fb5f18537cfdb528db337dfc9f6190dbf0521c07b885c0d7d" exitCode=0
Oct 30 01:55:30.941206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:30.941135    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s" event={"ID":"3f615383-3fb8-4e8e-8681-465e88538678","Type":"ContainerDied","Data":"8a631de2fbbe834fb5f18537cfdb528db337dfc9f6190dbf0521c07b885c0d7d"}
Oct 30 01:55:30.941206 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:30.941192    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s" event={"ID":"3f615383-3fb8-4e8e-8681-465e88538678","Type":"ContainerStarted","Data":"44fa32721ea0ace5b4408ac892c23e74bfa06c4fd2c81a4454b49ba2a81a39c1"}
Oct 30 01:55:30.974846 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:30.974809    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm" podStartSLOduration=1.97479705 podStartE2EDuration="1.97479705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:30.972616843 +0000 UTC m=+4439.722272151" watchObservedRunningTime="2026-10-30 01:55:30.97479705 +0000 UTC m=+4439.724452358"
Oct 30 01:55:31.041299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:31.041248    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf" podStartSLOduration=2.041232113 podStartE2EDuration="2.041232113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:31.041063807 +0000 UTC m=+4439.790719111" watchObservedRunningTime="2026-10-30 01:55:31.041232113 +0000 UTC m=+4439.790887422"
Oct 30 01:55:31.954980 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:31.954939    4425 generic.go:356] "Generic (PLEG): container finished" podID="c65fec65-f9ed-410d-abd0-f9ac048defe6" containerID="acabce4015b06e8eb4bb9292a6adcf2cb8341d71c7dfb57d508d64b7fae564db" exitCode=0
Oct 30 01:55:31.955452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:31.955024    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf" event={"ID":"c65fec65-f9ed-410d-abd0-f9ac048defe6","Type":"ContainerDied","Data":"acabce4015b06e8eb4bb9292a6adcf2cb8341d71c7dfb57d508d64b7fae564db"}
Oct 30 01:55:31.957045 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:31.957015    4425 generic.go:356] "Generic (PLEG): container finished" podID="efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7" containerID="4ea649ec049944707b675b5259402fb05efb1bafb9c52b6ceff98c1d7b850a60" exitCode=0
Oct 30 01:55:31.957203 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:31.957093    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm" event={"ID":"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7","Type":"ContainerDied","Data":"4ea649ec049944707b675b5259402fb05efb1bafb9c52b6ceff98c1d7b850a60"}
Oct 30 01:55:32.094659 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.094631    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"]
Oct 30 01:55:32.099115 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.099093    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"
Oct 30 01:55:32.122692 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.122662    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s"
Oct 30 01:55:32.175648 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.175614    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"]
Oct 30 01:55:32.248699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.248621    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f615383-3fb8-4e8e-8681-465e88538678-kube-api-access-gw2h5\" (UniqueName: \"kubernetes.io/projected/3f615383-3fb8-4e8e-8681-465e88538678-kube-api-access-gw2h5\") pod \"3f615383-3fb8-4e8e-8681-465e88538678\" (UID: \"3f615383-3fb8-4e8e-8681-465e88538678\") "
Oct 30 01:55:32.248872 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.248797    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f615383-3fb8-4e8e-8681-465e88538678-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f615383-3fb8-4e8e-8681-465e88538678-maas-api-openshift-ca\") pod \"3f615383-3fb8-4e8e-8681-465e88538678\" (UID: \"3f615383-3fb8-4e8e-8681-465e88538678\") "
Oct 30 01:55:32.248872 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.248851    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f615383-3fb8-4e8e-8681-465e88538678-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f615383-3fb8-4e8e-8681-465e88538678-maas-api-xks-ca\") pod \"3f615383-3fb8-4e8e-8681-465e88538678\" (UID: \"3f615383-3fb8-4e8e-8681-465e88538678\") "
Oct 30 01:55:32.249057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.249038    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckphk\" (UniqueName: \"kubernetes.io/projected/620d3db0-1161-4b1f-b32f-48daf5ed5077-kube-api-access-ckphk\") pod \"maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc\" (UID: \"620d3db0-1161-4b1f-b32f-48daf5ed5077\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"
Oct 30 01:55:32.249269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.249240    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f615383-3fb8-4e8e-8681-465e88538678-maas-api-openshift-ca" pod "3f615383-3fb8-4e8e-8681-465e88538678" (UID: "3f615383-3fb8-4e8e-8681-465e88538678"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:32.249269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.249257    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/620d3db0-1161-4b1f-b32f-48daf5ed5077-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc\" (UID: \"620d3db0-1161-4b1f-b32f-48daf5ed5077\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"
Oct 30 01:55:32.249459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.249363    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/620d3db0-1161-4b1f-b32f-48daf5ed5077-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc\" (UID: \"620d3db0-1161-4b1f-b32f-48daf5ed5077\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"
Oct 30 01:55:32.249459 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.249442    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f615383-3fb8-4e8e-8681-465e88538678-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:32.250858 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.250831    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f615383-3fb8-4e8e-8681-465e88538678-maas-api-xks-ca" pod "3f615383-3fb8-4e8e-8681-465e88538678" (UID: "3f615383-3fb8-4e8e-8681-465e88538678"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:32.250966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.250920    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f615383-3fb8-4e8e-8681-465e88538678-kube-api-access-gw2h5" pod "3f615383-3fb8-4e8e-8681-465e88538678" (UID: "3f615383-3fb8-4e8e-8681-465e88538678"). InnerVolumeSpecName "kube-api-access-gw2h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:32.350739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.350700    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckphk\" (UniqueName: \"kubernetes.io/projected/620d3db0-1161-4b1f-b32f-48daf5ed5077-kube-api-access-ckphk\") pod \"maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc\" (UID: \"620d3db0-1161-4b1f-b32f-48daf5ed5077\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"
Oct 30 01:55:32.350906 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.350812    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/620d3db0-1161-4b1f-b32f-48daf5ed5077-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc\" (UID: \"620d3db0-1161-4b1f-b32f-48daf5ed5077\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"
Oct 30 01:55:32.350906 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.350864    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/620d3db0-1161-4b1f-b32f-48daf5ed5077-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc\" (UID: \"620d3db0-1161-4b1f-b32f-48daf5ed5077\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"
Oct 30 01:55:32.350987 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.350914    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f615383-3fb8-4e8e-8681-465e88538678-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:32.350987 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.350925    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw2h5\" (UniqueName: \"kubernetes.io/projected/3f615383-3fb8-4e8e-8681-465e88538678-kube-api-access-gw2h5\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:32.351555 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.351535    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/620d3db0-1161-4b1f-b32f-48daf5ed5077-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc\" (UID: \"620d3db0-1161-4b1f-b32f-48daf5ed5077\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"
Oct 30 01:55:32.354027 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.354005    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/620d3db0-1161-4b1f-b32f-48daf5ed5077-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc\" (UID: \"620d3db0-1161-4b1f-b32f-48daf5ed5077\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"
Oct 30 01:55:32.372749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.372718    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckphk\" (UniqueName: \"kubernetes.io/projected/620d3db0-1161-4b1f-b32f-48daf5ed5077-kube-api-access-ckphk\") pod \"maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc\" (UID: \"620d3db0-1161-4b1f-b32f-48daf5ed5077\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"
Oct 30 01:55:32.432445 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.432415    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"
Oct 30 01:55:32.748523 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.748442    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"]
Oct 30 01:55:32.751321 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:55:32.751272    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620d3db0_1161_4b1f_b32f_48daf5ed5077.slice/crio-4fd979a4a46ee95c5f5369b01756fe064927244f192a06af5f7656f3397e1f6f WatchSource:0}: Error finding container 4fd979a4a46ee95c5f5369b01756fe064927244f192a06af5f7656f3397e1f6f: Status 404 returned error can't find the container with id 4fd979a4a46ee95c5f5369b01756fe064927244f192a06af5f7656f3397e1f6f
Oct 30 01:55:32.964703 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.964667    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc" event={"ID":"620d3db0-1161-4b1f-b32f-48daf5ed5077","Type":"ContainerStarted","Data":"4fd979a4a46ee95c5f5369b01756fe064927244f192a06af5f7656f3397e1f6f"}
Oct 30 01:55:32.966065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.966032    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s"
Oct 30 01:55:32.966214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.966041    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0684d5d8-hfw4s" event={"ID":"3f615383-3fb8-4e8e-8681-465e88538678","Type":"ContainerDied","Data":"44fa32721ea0ace5b4408ac892c23e74bfa06c4fd2c81a4454b49ba2a81a39c1"}
Oct 30 01:55:32.966214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:32.966167    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44fa32721ea0ace5b4408ac892c23e74bfa06c4fd2c81a4454b49ba2a81a39c1"
Oct 30 01:55:33.145447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.145418    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm"
Oct 30 01:55:33.166300 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.166279    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf"
Oct 30 01:55:33.259450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.259415    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c65fec65-f9ed-410d-abd0-f9ac048defe6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c65fec65-f9ed-410d-abd0-f9ac048defe6-maas-api-xks-ca\") pod \"c65fec65-f9ed-410d-abd0-f9ac048defe6\" (UID: \"c65fec65-f9ed-410d-abd0-f9ac048defe6\") "
Oct 30 01:55:33.259450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.259451    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-maas-api-xks-ca\") pod \"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7\" (UID: \"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7\") "
Oct 30 01:55:33.259707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.259533    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-maas-api-openshift-ca\") pod \"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7\" (UID: \"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7\") "
Oct 30 01:55:33.259707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.259586    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c65fec65-f9ed-410d-abd0-f9ac048defe6-kube-api-access-dl2fr\" (UniqueName: \"kubernetes.io/projected/c65fec65-f9ed-410d-abd0-f9ac048defe6-kube-api-access-dl2fr\") pod \"c65fec65-f9ed-410d-abd0-f9ac048defe6\" (UID: \"c65fec65-f9ed-410d-abd0-f9ac048defe6\") "
Oct 30 01:55:33.259707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.259655    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c65fec65-f9ed-410d-abd0-f9ac048defe6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c65fec65-f9ed-410d-abd0-f9ac048defe6-maas-api-openshift-ca\") pod \"c65fec65-f9ed-410d-abd0-f9ac048defe6\" (UID: \"c65fec65-f9ed-410d-abd0-f9ac048defe6\") "
Oct 30 01:55:33.259707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.259693    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-kube-api-access-8sswq\" (UniqueName: \"kubernetes.io/projected/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-kube-api-access-8sswq\") pod \"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7\" (UID: \"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7\") "
Oct 30 01:55:33.259995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.259912    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-maas-api-openshift-ca" pod "efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7" (UID: "efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:33.260188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.260124    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:33.260276 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.260240    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c65fec65-f9ed-410d-abd0-f9ac048defe6-maas-api-openshift-ca" pod "c65fec65-f9ed-410d-abd0-f9ac048defe6" (UID: "c65fec65-f9ed-410d-abd0-f9ac048defe6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:33.261964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.261930    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-maas-api-xks-ca" pod "efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7" (UID: "efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:33.262055 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.261975    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c65fec65-f9ed-410d-abd0-f9ac048defe6-maas-api-xks-ca" pod "c65fec65-f9ed-410d-abd0-f9ac048defe6" (UID: "c65fec65-f9ed-410d-abd0-f9ac048defe6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:33.262280 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.262259    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65fec65-f9ed-410d-abd0-f9ac048defe6-kube-api-access-dl2fr" pod "c65fec65-f9ed-410d-abd0-f9ac048defe6" (UID: "c65fec65-f9ed-410d-abd0-f9ac048defe6"). InnerVolumeSpecName "kube-api-access-dl2fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:33.262340 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.262282    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-kube-api-access-8sswq" pod "efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7" (UID: "efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7"). InnerVolumeSpecName "kube-api-access-8sswq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:33.361651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.361564    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c65fec65-f9ed-410d-abd0-f9ac048defe6-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:33.361651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.361602    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:33.361651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.361619    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl2fr\" (UniqueName: \"kubernetes.io/projected/c65fec65-f9ed-410d-abd0-f9ac048defe6-kube-api-access-dl2fr\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:33.361651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.361632    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c65fec65-f9ed-410d-abd0-f9ac048defe6-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:33.361651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.361646    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sswq\" (UniqueName: \"kubernetes.io/projected/efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7-kube-api-access-8sswq\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:33.506285 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.506253    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"]
Oct 30 01:55:33.506867 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.506844    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c65fec65-f9ed-410d-abd0-f9ac048defe6" containerName="revoke-keys"
Oct 30 01:55:33.506992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.506869    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c65fec65-f9ed-410d-abd0-f9ac048defe6" containerName="revoke-keys"
Oct 30 01:55:33.506992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.506895    4425 cpu_manager.go:414] "containerMap: removing container" podUID="efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7" containerName="revoke-keys"
Oct 30 01:55:33.506992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.506903    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7" containerName="revoke-keys"
Oct 30 01:55:33.506992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.506950    4425 cpu_manager.go:414] "containerMap: removing container" podUID="3f615383-3fb8-4e8e-8681-465e88538678" containerName="revoke-keys"
Oct 30 01:55:33.506992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.506955    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f615383-3fb8-4e8e-8681-465e88538678" containerName="revoke-keys"
Oct 30 01:55:33.507321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.507039    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f615383-3fb8-4e8e-8681-465e88538678" containerName="revoke-keys"
Oct 30 01:55:33.507321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.507050    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c65fec65-f9ed-410d-abd0-f9ac048defe6" containerName="revoke-keys"
Oct 30 01:55:33.507321 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.507058    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7" containerName="revoke-keys"
Oct 30 01:55:33.518751 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.518728    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"
Oct 30 01:55:33.531431 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.531406    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"]
Oct 30 01:55:33.665367 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.665268    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v\" (UID: \"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"
Oct 30 01:55:33.665543 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.665482    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v\" (UID: \"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"
Oct 30 01:55:33.665615 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.665525    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq9rw\" (UniqueName: \"kubernetes.io/projected/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-kube-api-access-vq9rw\") pod \"maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v\" (UID: \"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"
Oct 30 01:55:33.767101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.767070    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v\" (UID: \"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"
Oct 30 01:55:33.767290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.767225    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v\" (UID: \"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"
Oct 30 01:55:33.767290 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.767255    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq9rw\" (UniqueName: \"kubernetes.io/projected/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-kube-api-access-vq9rw\") pod \"maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v\" (UID: \"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"
Oct 30 01:55:33.767737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.767712    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v\" (UID: \"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"
Oct 30 01:55:33.769336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.769314    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v\" (UID: \"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"
Oct 30 01:55:33.788237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.788216    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq9rw\" (UniqueName: \"kubernetes.io/projected/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-kube-api-access-vq9rw\") pod \"maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v\" (UID: \"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"
Oct 30 01:55:33.834783 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.834747    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"
Oct 30 01:55:33.974513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.974440    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm"
Oct 30 01:55:33.974513 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.974491    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1279943c-c7glm" event={"ID":"efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7","Type":"ContainerDied","Data":"1c9c16390d07bf6a7edadc0c1edf5ba284b2560b1cae6dfcbf16bbad045e430b"}
Oct 30 01:55:33.974923 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.974545    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c9c16390d07bf6a7edadc0c1edf5ba284b2560b1cae6dfcbf16bbad045e430b"
Oct 30 01:55:33.976331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.976274    4425 generic.go:356] "Generic (PLEG): container finished" podID="620d3db0-1161-4b1f-b32f-48daf5ed5077" containerID="c12bdc4bbf0bce519316272e562734798270b9a4324ca4794f9988ff600a82a9" exitCode=0
Oct 30 01:55:33.976471 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.976351    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc" event={"ID":"620d3db0-1161-4b1f-b32f-48daf5ed5077","Type":"ContainerDied","Data":"c12bdc4bbf0bce519316272e562734798270b9a4324ca4794f9988ff600a82a9"}
Oct 30 01:55:33.978065 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.978037    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf"
Oct 30 01:55:33.978197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.978038    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6502d6e-tgwhf" event={"ID":"c65fec65-f9ed-410d-abd0-f9ac048defe6","Type":"ContainerDied","Data":"89dfb28dd8112b150edf2a6c97347c311b0b9e851bb796a4207f319975200ce3"}
Oct 30 01:55:33.978197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:33.978141    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89dfb28dd8112b150edf2a6c97347c311b0b9e851bb796a4207f319975200ce3"
Oct 30 01:55:34.139876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:34.139852    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"]
Oct 30 01:55:34.141574 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:55:34.141551    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f35be2f_ef4a_4630_8a7f_c53fcf8ed977.slice/crio-25274d3ce6215433bbfd6e1c30dd9ef7b2f5a11824b6d679d4030e64d6a2a981 WatchSource:0}: Error finding container 25274d3ce6215433bbfd6e1c30dd9ef7b2f5a11824b6d679d4030e64d6a2a981: Status 404 returned error can't find the container with id 25274d3ce6215433bbfd6e1c30dd9ef7b2f5a11824b6d679d4030e64d6a2a981
Oct 30 01:55:34.986557 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:34.986516    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v" event={"ID":"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977","Type":"ContainerStarted","Data":"c32e5ef2f4968f4a068be9e23abe4efd5999e8902e4386ad67d8767f6a25a829"}
Oct 30 01:55:34.987009 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:34.986582    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v" event={"ID":"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977","Type":"ContainerStarted","Data":"25274d3ce6215433bbfd6e1c30dd9ef7b2f5a11824b6d679d4030e64d6a2a981"}
Oct 30 01:55:35.024669 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:35.024605    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v" podStartSLOduration=2.024587979 podStartE2EDuration="2.024587979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:35.021533423 +0000 UTC m=+4443.771188729" watchObservedRunningTime="2026-10-30 01:55:35.024587979 +0000 UTC m=+4443.774243288"
Oct 30 01:55:35.147197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:35.147172    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"
Oct 30 01:55:35.283360 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:35.283311    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/620d3db0-1161-4b1f-b32f-48daf5ed5077-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/620d3db0-1161-4b1f-b32f-48daf5ed5077-maas-api-openshift-ca\") pod \"620d3db0-1161-4b1f-b32f-48daf5ed5077\" (UID: \"620d3db0-1161-4b1f-b32f-48daf5ed5077\") "
Oct 30 01:55:35.283535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:35.283388    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620d3db0-1161-4b1f-b32f-48daf5ed5077-kube-api-access-ckphk\" (UniqueName: \"kubernetes.io/projected/620d3db0-1161-4b1f-b32f-48daf5ed5077-kube-api-access-ckphk\") pod \"620d3db0-1161-4b1f-b32f-48daf5ed5077\" (UID: \"620d3db0-1161-4b1f-b32f-48daf5ed5077\") "
Oct 30 01:55:35.283620 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:35.283536    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/620d3db0-1161-4b1f-b32f-48daf5ed5077-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/620d3db0-1161-4b1f-b32f-48daf5ed5077-maas-api-xks-ca\") pod \"620d3db0-1161-4b1f-b32f-48daf5ed5077\" (UID: \"620d3db0-1161-4b1f-b32f-48daf5ed5077\") "
Oct 30 01:55:35.283700 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:35.283674    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620d3db0-1161-4b1f-b32f-48daf5ed5077-maas-api-openshift-ca" pod "620d3db0-1161-4b1f-b32f-48daf5ed5077" (UID: "620d3db0-1161-4b1f-b32f-48daf5ed5077"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:35.283900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:35.283884    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/620d3db0-1161-4b1f-b32f-48daf5ed5077-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:35.285510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:35.285481    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620d3db0-1161-4b1f-b32f-48daf5ed5077-kube-api-access-ckphk" pod "620d3db0-1161-4b1f-b32f-48daf5ed5077" (UID: "620d3db0-1161-4b1f-b32f-48daf5ed5077"). InnerVolumeSpecName "kube-api-access-ckphk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:35.285679 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:35.285659    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/620d3db0-1161-4b1f-b32f-48daf5ed5077-maas-api-xks-ca" pod "620d3db0-1161-4b1f-b32f-48daf5ed5077" (UID: "620d3db0-1161-4b1f-b32f-48daf5ed5077"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:35.385019 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:35.384991    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/620d3db0-1161-4b1f-b32f-48daf5ed5077-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:35.385019 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:35.385016    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckphk\" (UniqueName: \"kubernetes.io/projected/620d3db0-1161-4b1f-b32f-48daf5ed5077-kube-api-access-ckphk\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:35.995240 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:35.995212    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"
Oct 30 01:55:35.995680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:35.995254    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc" event={"ID":"620d3db0-1161-4b1f-b32f-48daf5ed5077","Type":"ContainerDied","Data":"4fd979a4a46ee95c5f5369b01756fe064927244f192a06af5f7656f3397e1f6f"}
Oct 30 01:55:35.995680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:35.995308    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fd979a4a46ee95c5f5369b01756fe064927244f192a06af5f7656f3397e1f6f"
Oct 30 01:55:35.997197 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:35.997139    4425 generic.go:356] "Generic (PLEG): container finished" podID="4f35be2f-ef4a-4630-8a7f-c53fcf8ed977" containerID="c32e5ef2f4968f4a068be9e23abe4efd5999e8902e4386ad67d8767f6a25a829" exitCode=0
Oct 30 01:55:35.997336 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:35.997255    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v" event={"ID":"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977","Type":"ContainerDied","Data":"c32e5ef2f4968f4a068be9e23abe4efd5999e8902e4386ad67d8767f6a25a829"}
Oct 30 01:55:36.667977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:36.667935    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx" podUID="60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:55:36.668283 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:36.668256    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj" podUID="c44e61e3-669d-4ac3-8686-68f72e4df64c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:55:37.159391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:37.159364    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"
Oct 30 01:55:37.203838 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:37.203760    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-maas-api-xks-ca\") pod \"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977\" (UID: \"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977\") "
Oct 30 01:55:37.203838 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:37.203815    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-kube-api-access-vq9rw\" (UniqueName: \"kubernetes.io/projected/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-kube-api-access-vq9rw\") pod \"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977\" (UID: \"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977\") "
Oct 30 01:55:37.204068 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:37.203853    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-maas-api-openshift-ca\") pod \"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977\" (UID: \"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977\") "
Oct 30 01:55:37.204191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:37.204168    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-maas-api-openshift-ca" pod "4f35be2f-ef4a-4630-8a7f-c53fcf8ed977" (UID: "4f35be2f-ef4a-4630-8a7f-c53fcf8ed977"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:37.204319 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:37.204301    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:37.206215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:37.206187    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-kube-api-access-vq9rw" pod "4f35be2f-ef4a-4630-8a7f-c53fcf8ed977" (UID: "4f35be2f-ef4a-4630-8a7f-c53fcf8ed977"). InnerVolumeSpecName "kube-api-access-vq9rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:37.206414 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:37.206388    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-maas-api-xks-ca" pod "4f35be2f-ef4a-4630-8a7f-c53fcf8ed977" (UID: "4f35be2f-ef4a-4630-8a7f-c53fcf8ed977"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:37.305248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:37.305208    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:37.305248 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:37.305242    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq9rw\" (UniqueName: \"kubernetes.io/projected/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977-kube-api-access-vq9rw\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:37.554083 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:55:37.554038    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:55:38.010707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:38.010682    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"
Oct 30 01:55:38.010854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:38.010724    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v" event={"ID":"4f35be2f-ef4a-4630-8a7f-c53fcf8ed977","Type":"ContainerDied","Data":"25274d3ce6215433bbfd6e1c30dd9ef7b2f5a11824b6d679d4030e64d6a2a981"}
Oct 30 01:55:38.010854 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:38.010773    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25274d3ce6215433bbfd6e1c30dd9ef7b2f5a11824b6d679d4030e64d6a2a981"
Oct 30 01:55:38.331337 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:38.331246    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"]
Oct 30 01:55:38.340749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:38.340718    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"]
Oct 30 01:55:38.360938 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:38.359873    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"]
Oct 30 01:55:42.465364 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:42.465333    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"]
Oct 30 01:55:42.492638 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:42.492607    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"]
Oct 30 01:55:43.310271 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:43.310239    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"]
Oct 30 01:55:43.319033 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:43.319003    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-72ed13-kndmb"]
Oct 30 01:55:43.424139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:43.424100    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8d3d737-8a12-44f3-a668-58f45d389278" path="/var/lib/kubelet/pods/f8d3d737-8a12-44f3-a668-58f45d389278/volumes"
Oct 30 01:55:43.859762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:43.859730    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"]
Oct 30 01:55:43.869460 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:43.869412    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-664c2b-vngzw"]
Oct 30 01:55:44.710191 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.709041    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf" podUID="81efdf05-b9a8-4ee2-9451-7cdc213207fb" containerName="main" containerID="cri-o://b156d65e05f4fcc07484913793a003f366b9e0ab5731e4b2dc1507700bbe6b2d" gracePeriod=45
Oct 30 01:55:44.843374 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.843345    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"
Oct 30 01:55:44.845607 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.845563    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"
Oct 30 01:55:44.890422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.887026    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c44e61e3-669d-4ac3-8686-68f72e4df64c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44e61e3-669d-4ac3-8686-68f72e4df64c-plugins-config-volume\") pod \"c44e61e3-669d-4ac3-8686-68f72e4df64c\" (UID: \"c44e61e3-669d-4ac3-8686-68f72e4df64c\") "
Oct 30 01:55:44.890422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.887109    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e-kube-api-access-8qg8f\" (UniqueName: \"kubernetes.io/projected/60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e-kube-api-access-8qg8f\") pod \"60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e\" (UID: \"60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e\") "
Oct 30 01:55:44.890422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.887192    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e-plugins-config-volume\") pod \"60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e\" (UID: \"60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e\") "
Oct 30 01:55:44.890422 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.887364    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c44e61e3-669d-4ac3-8686-68f72e4df64c-kube-api-access-pvkst\" (UniqueName: \"kubernetes.io/projected/c44e61e3-669d-4ac3-8686-68f72e4df64c-kube-api-access-pvkst\") pod \"c44e61e3-669d-4ac3-8686-68f72e4df64c\" (UID: \"c44e61e3-669d-4ac3-8686-68f72e4df64c\") "
Oct 30 01:55:44.895878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.891322    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e-kube-api-access-8qg8f" pod "60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e" (UID: "60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e"). InnerVolumeSpecName "kube-api-access-8qg8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:44.895878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.891692    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44e61e3-669d-4ac3-8686-68f72e4df64c-plugins-config-volume" pod "c44e61e3-669d-4ac3-8686-68f72e4df64c" (UID: "c44e61e3-669d-4ac3-8686-68f72e4df64c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:44.895878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.892058    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e-plugins-config-volume" pod "60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e" (UID: "60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:44.895878 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.892865    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44e61e3-669d-4ac3-8686-68f72e4df64c-kube-api-access-pvkst" pod "c44e61e3-669d-4ac3-8686-68f72e4df64c" (UID: "c44e61e3-669d-4ac3-8686-68f72e4df64c"). InnerVolumeSpecName "kube-api-access-pvkst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:44.981564 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.981538    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:55:44.988639 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.988613    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-tmp-dir\") pod \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") "
Oct 30 01:55:44.988754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.988700    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-model-cache\") pod \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") "
Oct 30 01:55:44.988754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.988738    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81efdf05-b9a8-4ee2-9451-7cdc213207fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/81efdf05-b9a8-4ee2-9451-7cdc213207fb-tls-certs\") pod \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") "
Oct 30 01:55:44.988840 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.988772    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-dshm\") pod \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") "
Oct 30 01:55:44.988872 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.988835    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81efdf05-b9a8-4ee2-9451-7cdc213207fb-kube-api-access-hkj2v\" (UniqueName: \"kubernetes.io/projected/81efdf05-b9a8-4ee2-9451-7cdc213207fb-kube-api-access-hkj2v\") pod \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") "
Oct 30 01:55:44.988924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.988866    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-tmp-dir" pod "81efdf05-b9a8-4ee2-9451-7cdc213207fb" (UID: "81efdf05-b9a8-4ee2-9451-7cdc213207fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:44.988924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.988886    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-home\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-home\") pod \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\" (UID: \"81efdf05-b9a8-4ee2-9451-7cdc213207fb\") "
Oct 30 01:55:44.989462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.989195    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-model-cache" pod "81efdf05-b9a8-4ee2-9451-7cdc213207fb" (UID: "81efdf05-b9a8-4ee2-9451-7cdc213207fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:44.989462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.989312    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-home" pod "81efdf05-b9a8-4ee2-9451-7cdc213207fb" (UID: "81efdf05-b9a8-4ee2-9451-7cdc213207fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:44.989462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.989324    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:44.989462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.989364    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvkst\" (UniqueName: \"kubernetes.io/projected/c44e61e3-669d-4ac3-8686-68f72e4df64c-kube-api-access-pvkst\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:44.989462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.989381    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44e61e3-669d-4ac3-8686-68f72e4df64c-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:44.989462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.989398    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:44.989462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.989414    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qg8f\" (UniqueName: \"kubernetes.io/projected/60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e-kube-api-access-8qg8f\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:44.989462 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.989429    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:44.991097 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.991075    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-dshm" pod "81efdf05-b9a8-4ee2-9451-7cdc213207fb" (UID: "81efdf05-b9a8-4ee2-9451-7cdc213207fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:44.991214 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.991105    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81efdf05-b9a8-4ee2-9451-7cdc213207fb-kube-api-access-hkj2v" pod "81efdf05-b9a8-4ee2-9451-7cdc213207fb" (UID: "81efdf05-b9a8-4ee2-9451-7cdc213207fb"). InnerVolumeSpecName "kube-api-access-hkj2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:44.991270 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:44.991255    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81efdf05-b9a8-4ee2-9451-7cdc213207fb-tls-certs" pod "81efdf05-b9a8-4ee2-9451-7cdc213207fb" (UID: "81efdf05-b9a8-4ee2-9451-7cdc213207fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:45.058220 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.058176    4425 generic.go:356] "Generic (PLEG): container finished" podID="81efdf05-b9a8-4ee2-9451-7cdc213207fb" containerID="b156d65e05f4fcc07484913793a003f366b9e0ab5731e4b2dc1507700bbe6b2d" exitCode=0
Oct 30 01:55:45.058399 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.058222    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf" event={"ID":"81efdf05-b9a8-4ee2-9451-7cdc213207fb","Type":"ContainerDied","Data":"b156d65e05f4fcc07484913793a003f366b9e0ab5731e4b2dc1507700bbe6b2d"}
Oct 30 01:55:45.058399 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.058247    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"
Oct 30 01:55:45.058399 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.058272    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf" event={"ID":"81efdf05-b9a8-4ee2-9451-7cdc213207fb","Type":"ContainerDied","Data":"2d422fead3ffa1b781ad3763c6573fe676f19589740c1cb35a9239238c2001fa"}
Oct 30 01:55:45.058399 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.058291    4425 scope.go:122] "RemoveContainer" containerID="b156d65e05f4fcc07484913793a003f366b9e0ab5731e4b2dc1507700bbe6b2d"
Oct 30 01:55:45.060331 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.060306    4425 generic.go:356] "Generic (PLEG): container finished" podID="c44e61e3-669d-4ac3-8686-68f72e4df64c" containerID="5bfc0d0c7e3a421906c18d71ebf1342fceaa91c7253d76364e5a07c7732d27fe" exitCode=0
Oct 30 01:55:45.060443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.060348    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"
Oct 30 01:55:45.060443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.060368    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj" event={"ID":"c44e61e3-669d-4ac3-8686-68f72e4df64c","Type":"ContainerDied","Data":"5bfc0d0c7e3a421906c18d71ebf1342fceaa91c7253d76364e5a07c7732d27fe"}
Oct 30 01:55:45.060544 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.060454    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj" event={"ID":"c44e61e3-669d-4ac3-8686-68f72e4df64c","Type":"ContainerDied","Data":"34c2a34a0998775a407a1e293e9c6702480f89db8ca8d916de05e407fb73f666"}
Oct 30 01:55:45.062050 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.062026    4425 generic.go:356] "Generic (PLEG): container finished" podID="60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e" containerID="03642b2e4f65bd9fe504e826e029fff1f74202dc3ba90f4942137a0726d18f9f" exitCode=0
Oct 30 01:55:45.062211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.062069    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"
Oct 30 01:55:45.062211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.062101    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx" event={"ID":"60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e","Type":"ContainerDied","Data":"03642b2e4f65bd9fe504e826e029fff1f74202dc3ba90f4942137a0726d18f9f"}
Oct 30 01:55:45.062211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.062138    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx" event={"ID":"60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e","Type":"ContainerDied","Data":"fbf28af0e6ba601a39c5a1f6a9c1116645aa44898fb88b1087ae463a743c439c"}
Oct 30 01:55:45.071448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.071393    4425 scope.go:122] "RemoveContainer" containerID="b156d65e05f4fcc07484913793a003f366b9e0ab5731e4b2dc1507700bbe6b2d"
Oct 30 01:55:45.071800 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:55:45.071780    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b156d65e05f4fcc07484913793a003f366b9e0ab5731e4b2dc1507700bbe6b2d\": container with ID starting with b156d65e05f4fcc07484913793a003f366b9e0ab5731e4b2dc1507700bbe6b2d not found: ID does not exist" containerID="b156d65e05f4fcc07484913793a003f366b9e0ab5731e4b2dc1507700bbe6b2d"
Oct 30 01:55:45.071896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.071811    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b156d65e05f4fcc07484913793a003f366b9e0ab5731e4b2dc1507700bbe6b2d"} err="failed to get container status \"b156d65e05f4fcc07484913793a003f366b9e0ab5731e4b2dc1507700bbe6b2d\": rpc error: code = NotFound desc = could not find container \"b156d65e05f4fcc07484913793a003f366b9e0ab5731e4b2dc1507700bbe6b2d\": container with ID starting with b156d65e05f4fcc07484913793a003f366b9e0ab5731e4b2dc1507700bbe6b2d not found: ID does not exist"
Oct 30 01:55:45.071896 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.071834    4425 scope.go:122] "RemoveContainer" containerID="5bfc0d0c7e3a421906c18d71ebf1342fceaa91c7253d76364e5a07c7732d27fe"
Oct 30 01:55:45.082073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.082053    4425 scope.go:122] "RemoveContainer" containerID="5bfc0d0c7e3a421906c18d71ebf1342fceaa91c7253d76364e5a07c7732d27fe"
Oct 30 01:55:45.082389 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:55:45.082364    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bfc0d0c7e3a421906c18d71ebf1342fceaa91c7253d76364e5a07c7732d27fe\": container with ID starting with 5bfc0d0c7e3a421906c18d71ebf1342fceaa91c7253d76364e5a07c7732d27fe not found: ID does not exist" containerID="5bfc0d0c7e3a421906c18d71ebf1342fceaa91c7253d76364e5a07c7732d27fe"
Oct 30 01:55:45.082450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.082399    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bfc0d0c7e3a421906c18d71ebf1342fceaa91c7253d76364e5a07c7732d27fe"} err="failed to get container status \"5bfc0d0c7e3a421906c18d71ebf1342fceaa91c7253d76364e5a07c7732d27fe\": rpc error: code = NotFound desc = could not find container \"5bfc0d0c7e3a421906c18d71ebf1342fceaa91c7253d76364e5a07c7732d27fe\": container with ID starting with 5bfc0d0c7e3a421906c18d71ebf1342fceaa91c7253d76364e5a07c7732d27fe not found: ID does not exist"
Oct 30 01:55:45.082450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.082420    4425 scope.go:122] "RemoveContainer" containerID="03642b2e4f65bd9fe504e826e029fff1f74202dc3ba90f4942137a0726d18f9f"
Oct 30 01:55:45.091188 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.091141    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81efdf05-b9a8-4ee2-9451-7cdc213207fb-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:45.091274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.091191    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:45.091274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.091201    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkj2v\" (UniqueName: \"kubernetes.io/projected/81efdf05-b9a8-4ee2-9451-7cdc213207fb-kube-api-access-hkj2v\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:45.091274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.091208    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81efdf05-b9a8-4ee2-9451-7cdc213207fb-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:45.093039 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.093012    4425 scope.go:122] "RemoveContainer" containerID="03642b2e4f65bd9fe504e826e029fff1f74202dc3ba90f4942137a0726d18f9f"
Oct 30 01:55:45.093364 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:55:45.093345    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03642b2e4f65bd9fe504e826e029fff1f74202dc3ba90f4942137a0726d18f9f\": container with ID starting with 03642b2e4f65bd9fe504e826e029fff1f74202dc3ba90f4942137a0726d18f9f not found: ID does not exist" containerID="03642b2e4f65bd9fe504e826e029fff1f74202dc3ba90f4942137a0726d18f9f"
Oct 30 01:55:45.093461 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.093372    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03642b2e4f65bd9fe504e826e029fff1f74202dc3ba90f4942137a0726d18f9f"} err="failed to get container status \"03642b2e4f65bd9fe504e826e029fff1f74202dc3ba90f4942137a0726d18f9f\": rpc error: code = NotFound desc = could not find container \"03642b2e4f65bd9fe504e826e029fff1f74202dc3ba90f4942137a0726d18f9f\": container with ID starting with 03642b2e4f65bd9fe504e826e029fff1f74202dc3ba90f4942137a0726d18f9f not found: ID does not exist"
Oct 30 01:55:45.128714 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.128690    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"]
Oct 30 01:55:45.134354 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.134334    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4107fc/rate-test-model-4107fc-kserve-7bc6564cd5-lqmwf"]
Oct 30 01:55:45.171220 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.171200    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"]
Oct 30 01:55:45.173800 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.173779    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c666c2-74d7686794-6wpzj"]
Oct 30 01:55:45.204204 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.204184    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"]
Oct 30 01:55:45.207114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.207091    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c666c2-6f8bcd595-shsqx"]
Oct 30 01:55:45.422190 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.422109    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="264d250a-bc4d-4fd6-a1fa-f0a446892681" path="/var/lib/kubelet/pods/264d250a-bc4d-4fd6-a1fa-f0a446892681/volumes"
Oct 30 01:55:45.422514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.422499    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e" path="/var/lib/kubelet/pods/60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e/volumes"
Oct 30 01:55:45.422895 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.422882    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81efdf05-b9a8-4ee2-9451-7cdc213207fb" path="/var/lib/kubelet/pods/81efdf05-b9a8-4ee2-9451-7cdc213207fb/volumes"
Oct 30 01:55:45.423539 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:45.423523    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c44e61e3-669d-4ac3-8686-68f72e4df64c" path="/var/lib/kubelet/pods/c44e61e3-669d-4ac3-8686-68f72e4df64c/volumes"
Oct 30 01:55:47.493555 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.493508    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r" podUID="8e763d6f-32ac-4739-9e2d-feca894a8987" containerName="payload-pre-processing" containerID="cri-o://964227f2dd072fd7eb6c8b5fa6ab20ffddfbcb9c23cc8e8a5ec71f5b01a3754b" gracePeriod=25
Oct 30 01:55:47.539661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.539611    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r" podUID="8e763d6f-32ac-4739-9e2d-feca894a8987" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:55:47.843377 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.843344    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq"]
Oct 30 01:55:47.843807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.843795    4425 cpu_manager.go:414] "containerMap: removing container" podUID="81efdf05-b9a8-4ee2-9451-7cdc213207fb" containerName="main"
Oct 30 01:55:47.843849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.843809    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81efdf05-b9a8-4ee2-9451-7cdc213207fb" containerName="main"
Oct 30 01:55:47.843849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.843829    4425 cpu_manager.go:414] "containerMap: removing container" podUID="4f35be2f-ef4a-4630-8a7f-c53fcf8ed977" containerName="revoke-keys"
Oct 30 01:55:47.843849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.843834    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f35be2f-ef4a-4630-8a7f-c53fcf8ed977" containerName="revoke-keys"
Oct 30 01:55:47.843849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.843844    4425 cpu_manager.go:414] "containerMap: removing container" podUID="620d3db0-1161-4b1f-b32f-48daf5ed5077" containerName="revoke-keys"
Oct 30 01:55:47.843849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.843849    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620d3db0-1161-4b1f-b32f-48daf5ed5077" containerName="revoke-keys"
Oct 30 01:55:47.844008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.843860    4425 cpu_manager.go:414] "containerMap: removing container" podUID="c44e61e3-669d-4ac3-8686-68f72e4df64c" containerName="payload-processing"
Oct 30 01:55:47.844008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.843865    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44e61e3-669d-4ac3-8686-68f72e4df64c" containerName="payload-processing"
Oct 30 01:55:47.844008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.843874    4425 cpu_manager.go:414] "containerMap: removing container" podUID="60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e" containerName="payload-pre-processing"
Oct 30 01:55:47.844008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.843879    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e" containerName="payload-pre-processing"
Oct 30 01:55:47.844008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.843952    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="81efdf05-b9a8-4ee2-9451-7cdc213207fb" containerName="main"
Oct 30 01:55:47.844008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.843963    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f35be2f-ef4a-4630-8a7f-c53fcf8ed977" containerName="revoke-keys"
Oct 30 01:55:47.844008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.843969    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="620d3db0-1161-4b1f-b32f-48daf5ed5077" containerName="revoke-keys"
Oct 30 01:55:47.844008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.843976    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44e61e3-669d-4ac3-8686-68f72e4df64c" containerName="payload-processing"
Oct 30 01:55:47.844008 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.843982    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c5e5c8-a1f3-41b1-b8a3-8fecf93e1b9e" containerName="payload-pre-processing"
Oct 30 01:55:47.848996 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.848955    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq"
Oct 30 01:55:47.861479 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.861454    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk"]
Oct 30 01:55:47.864979 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.864961    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk"
Oct 30 01:55:47.867640 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.867616    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq"]
Oct 30 01:55:47.868149 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.868124    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:55:47.873977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.873955    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:55:47.873977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.873939    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:55:47.874128 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.873955    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:55:47.917999 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.913074    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk"]
Oct 30 01:55:47.919448 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.919416    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/019931b1-92c8-48a2-88be-2a0fb9b18450-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f6d1463-v6wwq\" (UID: \"019931b1-92c8-48a2-88be-2a0fb9b18450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq"
Oct 30 01:55:47.919583 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.919548    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk7gl\" (UniqueName: \"kubernetes.io/projected/d672322e-d956-4654-9e06-490aa9fb1518-kube-api-access-zk7gl\") pod \"maas-api-revoke-sub-03fd1035-9wzjk\" (UID: \"d672322e-d956-4654-9e06-490aa9fb1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk"
Oct 30 01:55:47.919650 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.919620    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d672322e-d956-4654-9e06-490aa9fb1518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03fd1035-9wzjk\" (UID: \"d672322e-d956-4654-9e06-490aa9fb1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk"
Oct 30 01:55:47.919703 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.919660    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/019931b1-92c8-48a2-88be-2a0fb9b18450-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f6d1463-v6wwq\" (UID: \"019931b1-92c8-48a2-88be-2a0fb9b18450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq"
Oct 30 01:55:47.919782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.919718    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48jtj\" (UniqueName: \"kubernetes.io/projected/019931b1-92c8-48a2-88be-2a0fb9b18450-kube-api-access-48jtj\") pod \"maas-api-revoke-sub-9f6d1463-v6wwq\" (UID: \"019931b1-92c8-48a2-88be-2a0fb9b18450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq"
Oct 30 01:55:47.919782 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:47.919745    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d672322e-d956-4654-9e06-490aa9fb1518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03fd1035-9wzjk\" (UID: \"d672322e-d956-4654-9e06-490aa9fb1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk"
Oct 30 01:55:48.020784 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.020744    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk7gl\" (UniqueName: \"kubernetes.io/projected/d672322e-d956-4654-9e06-490aa9fb1518-kube-api-access-zk7gl\") pod \"maas-api-revoke-sub-03fd1035-9wzjk\" (UID: \"d672322e-d956-4654-9e06-490aa9fb1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk"
Oct 30 01:55:48.020784 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.020790    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d672322e-d956-4654-9e06-490aa9fb1518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03fd1035-9wzjk\" (UID: \"d672322e-d956-4654-9e06-490aa9fb1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk"
Oct 30 01:55:48.021017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.020820    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/019931b1-92c8-48a2-88be-2a0fb9b18450-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f6d1463-v6wwq\" (UID: \"019931b1-92c8-48a2-88be-2a0fb9b18450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq"
Oct 30 01:55:48.021017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.020855    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48jtj\" (UniqueName: \"kubernetes.io/projected/019931b1-92c8-48a2-88be-2a0fb9b18450-kube-api-access-48jtj\") pod \"maas-api-revoke-sub-9f6d1463-v6wwq\" (UID: \"019931b1-92c8-48a2-88be-2a0fb9b18450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq"
Oct 30 01:55:48.021017 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.020876    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d672322e-d956-4654-9e06-490aa9fb1518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03fd1035-9wzjk\" (UID: \"d672322e-d956-4654-9e06-490aa9fb1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk"
Oct 30 01:55:48.021225 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.021089    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/019931b1-92c8-48a2-88be-2a0fb9b18450-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f6d1463-v6wwq\" (UID: \"019931b1-92c8-48a2-88be-2a0fb9b18450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq"
Oct 30 01:55:48.021629 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.021600    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d672322e-d956-4654-9e06-490aa9fb1518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03fd1035-9wzjk\" (UID: \"d672322e-d956-4654-9e06-490aa9fb1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk"
Oct 30 01:55:48.021964 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.021937    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/019931b1-92c8-48a2-88be-2a0fb9b18450-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f6d1463-v6wwq\" (UID: \"019931b1-92c8-48a2-88be-2a0fb9b18450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq"
Oct 30 01:55:48.023848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.023822    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d672322e-d956-4654-9e06-490aa9fb1518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03fd1035-9wzjk\" (UID: \"d672322e-d956-4654-9e06-490aa9fb1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk"
Oct 30 01:55:48.024511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.024488    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/019931b1-92c8-48a2-88be-2a0fb9b18450-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f6d1463-v6wwq\" (UID: \"019931b1-92c8-48a2-88be-2a0fb9b18450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq"
Oct 30 01:55:48.042473 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.042429    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8" podUID="be776653-8806-4154-96d5-87ce0af15d97" containerName="payload-processing" containerID="cri-o://399e0e35809f1fff95994362539fa2c16aa092d3c61bcf67c1bc060f80facb1b" gracePeriod=25
Oct 30 01:55:48.057179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.051704    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk7gl\" (UniqueName: \"kubernetes.io/projected/d672322e-d956-4654-9e06-490aa9fb1518-kube-api-access-zk7gl\") pod \"maas-api-revoke-sub-03fd1035-9wzjk\" (UID: \"d672322e-d956-4654-9e06-490aa9fb1518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk"
Oct 30 01:55:48.057179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.056887    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48jtj\" (UniqueName: \"kubernetes.io/projected/019931b1-92c8-48a2-88be-2a0fb9b18450-kube-api-access-48jtj\") pod \"maas-api-revoke-sub-9f6d1463-v6wwq\" (UID: \"019931b1-92c8-48a2-88be-2a0fb9b18450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq"
Oct 30 01:55:48.164626 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.164550    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq"
Oct 30 01:55:48.180828 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.180796    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk"
Oct 30 01:55:48.197016 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.196973    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"]
Oct 30 01:55:48.208977 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.208938    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2f820e-dfmbc"]
Oct 30 01:55:48.490103 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:55:48.490071    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019931b1_92c8_48a2_88be_2a0fb9b18450.slice/crio-b006453acae6b24e3c4dfbe369e8caf6fba149651813163564524d10d2f56fb6 WatchSource:0}: Error finding container b006453acae6b24e3c4dfbe369e8caf6fba149651813163564524d10d2f56fb6: Status 404 returned error can't find the container with id b006453acae6b24e3c4dfbe369e8caf6fba149651813163564524d10d2f56fb6
Oct 30 01:55:48.490405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.490380    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq"]
Oct 30 01:55:48.517488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.517462    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk"]
Oct 30 01:55:48.522034 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:55:48.522006    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd672322e_d956_4654_9e06_490aa9fb1518.slice/crio-6d691d176cecc2e952806f9d6124cc49bb6ba616091c3d87ea8f6761e3c27a7b WatchSource:0}: Error finding container 6d691d176cecc2e952806f9d6124cc49bb6ba616091c3d87ea8f6761e3c27a7b: Status 404 returned error can't find the container with id 6d691d176cecc2e952806f9d6124cc49bb6ba616091c3d87ea8f6761e3c27a7b
Oct 30 01:55:48.551421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:48.551182    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8" podUID="be776653-8806-4154-96d5-87ce0af15d97" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:55:49.120972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:49.120931    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq" event={"ID":"019931b1-92c8-48a2-88be-2a0fb9b18450","Type":"ContainerStarted","Data":"48946c635883038c7eb18b09b8239dc6c9b36ad275eb37ad483bfac2e07a8c63"}
Oct 30 01:55:49.121181 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:49.120997    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq" event={"ID":"019931b1-92c8-48a2-88be-2a0fb9b18450","Type":"ContainerStarted","Data":"b006453acae6b24e3c4dfbe369e8caf6fba149651813163564524d10d2f56fb6"}
Oct 30 01:55:49.123955 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:49.123922    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk" event={"ID":"d672322e-d956-4654-9e06-490aa9fb1518","Type":"ContainerStarted","Data":"877ed164676014ee5eff910deca09c84a403ab762b16f26e16d9456e0c8fc4ad"}
Oct 30 01:55:49.124075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:49.123974    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk" event={"ID":"d672322e-d956-4654-9e06-490aa9fb1518","Type":"ContainerStarted","Data":"6d691d176cecc2e952806f9d6124cc49bb6ba616091c3d87ea8f6761e3c27a7b"}
Oct 30 01:55:49.160990 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:49.160945    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq" podStartSLOduration=2.160932446 podStartE2EDuration="2.160932446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:49.157797856 +0000 UTC m=+4457.907453165" watchObservedRunningTime="2026-10-30 01:55:49.160932446 +0000 UTC m=+4457.910587797"
Oct 30 01:55:49.202520 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:49.202465    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk" podStartSLOduration=2.202448612 podStartE2EDuration="2.202448612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:49.200570927 +0000 UTC m=+4457.950226234" watchObservedRunningTime="2026-10-30 01:55:49.202448612 +0000 UTC m=+4457.952103920"
Oct 30 01:55:49.421820 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:49.421791    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="620d3db0-1161-4b1f-b32f-48daf5ed5077" path="/var/lib/kubelet/pods/620d3db0-1161-4b1f-b32f-48daf5ed5077/volumes"
Oct 30 01:55:50.132508 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:50.132473    4425 generic.go:356] "Generic (PLEG): container finished" podID="019931b1-92c8-48a2-88be-2a0fb9b18450" containerID="48946c635883038c7eb18b09b8239dc6c9b36ad275eb37ad483bfac2e07a8c63" exitCode=0
Oct 30 01:55:50.132914 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:50.132523    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq" event={"ID":"019931b1-92c8-48a2-88be-2a0fb9b18450","Type":"ContainerDied","Data":"48946c635883038c7eb18b09b8239dc6c9b36ad275eb37ad483bfac2e07a8c63"}
Oct 30 01:55:50.134131 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:50.134108    4425 generic.go:356] "Generic (PLEG): container finished" podID="d672322e-d956-4654-9e06-490aa9fb1518" containerID="877ed164676014ee5eff910deca09c84a403ab762b16f26e16d9456e0c8fc4ad" exitCode=0
Oct 30 01:55:50.134263 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:50.134183    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk" event={"ID":"d672322e-d956-4654-9e06-490aa9fb1518","Type":"ContainerDied","Data":"877ed164676014ee5eff910deca09c84a403ab762b16f26e16d9456e0c8fc4ad"}
Oct 30 01:55:51.301015 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.300991    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq"
Oct 30 01:55:51.303314 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.303296    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk"
Oct 30 01:55:51.361077 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.361052    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/019931b1-92c8-48a2-88be-2a0fb9b18450-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/019931b1-92c8-48a2-88be-2a0fb9b18450-maas-api-xks-ca\") pod \"019931b1-92c8-48a2-88be-2a0fb9b18450\" (UID: \"019931b1-92c8-48a2-88be-2a0fb9b18450\") "
Oct 30 01:55:51.361268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.361113    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d672322e-d956-4654-9e06-490aa9fb1518-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d672322e-d956-4654-9e06-490aa9fb1518-maas-api-openshift-ca\") pod \"d672322e-d956-4654-9e06-490aa9fb1518\" (UID: \"d672322e-d956-4654-9e06-490aa9fb1518\") "
Oct 30 01:55:51.361268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.361199    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/019931b1-92c8-48a2-88be-2a0fb9b18450-kube-api-access-48jtj\" (UniqueName: \"kubernetes.io/projected/019931b1-92c8-48a2-88be-2a0fb9b18450-kube-api-access-48jtj\") pod \"019931b1-92c8-48a2-88be-2a0fb9b18450\" (UID: \"019931b1-92c8-48a2-88be-2a0fb9b18450\") "
Oct 30 01:55:51.361268 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.361221    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/019931b1-92c8-48a2-88be-2a0fb9b18450-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/019931b1-92c8-48a2-88be-2a0fb9b18450-maas-api-openshift-ca\") pod \"019931b1-92c8-48a2-88be-2a0fb9b18450\" (UID: \"019931b1-92c8-48a2-88be-2a0fb9b18450\") "
Oct 30 01:55:51.361439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.361291    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d672322e-d956-4654-9e06-490aa9fb1518-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d672322e-d956-4654-9e06-490aa9fb1518-maas-api-xks-ca\") pod \"d672322e-d956-4654-9e06-490aa9fb1518\" (UID: \"d672322e-d956-4654-9e06-490aa9fb1518\") "
Oct 30 01:55:51.361439 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.361339    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d672322e-d956-4654-9e06-490aa9fb1518-kube-api-access-zk7gl\" (UniqueName: \"kubernetes.io/projected/d672322e-d956-4654-9e06-490aa9fb1518-kube-api-access-zk7gl\") pod \"d672322e-d956-4654-9e06-490aa9fb1518\" (UID: \"d672322e-d956-4654-9e06-490aa9fb1518\") "
Oct 30 01:55:51.361563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.361537    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d672322e-d956-4654-9e06-490aa9fb1518-maas-api-openshift-ca" pod "d672322e-d956-4654-9e06-490aa9fb1518" (UID: "d672322e-d956-4654-9e06-490aa9fb1518"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:51.361641 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.361616    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/019931b1-92c8-48a2-88be-2a0fb9b18450-maas-api-openshift-ca" pod "019931b1-92c8-48a2-88be-2a0fb9b18450" (UID: "019931b1-92c8-48a2-88be-2a0fb9b18450"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:51.361801 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.361782    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/019931b1-92c8-48a2-88be-2a0fb9b18450-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:51.361870 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.361805    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d672322e-d956-4654-9e06-490aa9fb1518-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:51.363692 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.363659    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d672322e-d956-4654-9e06-490aa9fb1518-kube-api-access-zk7gl" pod "d672322e-d956-4654-9e06-490aa9fb1518" (UID: "d672322e-d956-4654-9e06-490aa9fb1518"). InnerVolumeSpecName "kube-api-access-zk7gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:51.363986 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.363962    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/019931b1-92c8-48a2-88be-2a0fb9b18450-maas-api-xks-ca" pod "019931b1-92c8-48a2-88be-2a0fb9b18450" (UID: "019931b1-92c8-48a2-88be-2a0fb9b18450"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:51.364118 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.364101    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d672322e-d956-4654-9e06-490aa9fb1518-maas-api-xks-ca" pod "d672322e-d956-4654-9e06-490aa9fb1518" (UID: "d672322e-d956-4654-9e06-490aa9fb1518"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:51.370463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.370441    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/019931b1-92c8-48a2-88be-2a0fb9b18450-kube-api-access-48jtj" pod "019931b1-92c8-48a2-88be-2a0fb9b18450" (UID: "019931b1-92c8-48a2-88be-2a0fb9b18450"). InnerVolumeSpecName "kube-api-access-48jtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:51.462924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.462897    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48jtj\" (UniqueName: \"kubernetes.io/projected/019931b1-92c8-48a2-88be-2a0fb9b18450-kube-api-access-48jtj\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:51.462924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.462921    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d672322e-d956-4654-9e06-490aa9fb1518-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:51.462924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.462931    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk7gl\" (UniqueName: \"kubernetes.io/projected/d672322e-d956-4654-9e06-490aa9fb1518-kube-api-access-zk7gl\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:51.462924 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:51.462943    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/019931b1-92c8-48a2-88be-2a0fb9b18450-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:52.149497 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:52.149460    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq"
Oct 30 01:55:52.149784 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:52.149488    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f6d1463-v6wwq" event={"ID":"019931b1-92c8-48a2-88be-2a0fb9b18450","Type":"ContainerDied","Data":"b006453acae6b24e3c4dfbe369e8caf6fba149651813163564524d10d2f56fb6"}
Oct 30 01:55:52.149784 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:52.149545    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b006453acae6b24e3c4dfbe369e8caf6fba149651813163564524d10d2f56fb6"
Oct 30 01:55:52.150744 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:52.150724    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk"
Oct 30 01:55:52.150871 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:52.150755    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03fd1035-9wzjk" event={"ID":"d672322e-d956-4654-9e06-490aa9fb1518","Type":"ContainerDied","Data":"6d691d176cecc2e952806f9d6124cc49bb6ba616091c3d87ea8f6761e3c27a7b"}
Oct 30 01:55:52.150871 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:52.150796    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d691d176cecc2e952806f9d6124cc49bb6ba616091c3d87ea8f6761e3c27a7b"
Oct 30 01:55:52.982936 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:52.982903    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"]
Oct 30 01:55:52.983521 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:52.983115    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr" podUID="998232e0-4d7b-4a4b-9fd1-5eb005157b11" containerName="istio-proxy" containerID="cri-o://cde5f6edf92232a90174fc5976205803909583e7b1e3994bf76707307795412f" gracePeriod=30
Oct 30 01:55:53.352596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.352545    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f" podUID="0824ed99-cf93-4047-b19f-0ad807263acf" containerName="main" containerID="cri-o://b76b959e601a3ff544ff0f9b10cf6f78e86389b1f64eb367563f787bb3be6769" gracePeriod=45
Oct 30 01:55:53.362196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.362148    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7" podUID="834166ed-5c63-4854-9e89-c45041553630" containerName="main" containerID="cri-o://1d9fcf60190de75e8075d350084589723f3ee8ea276930870f980a0161106997" gracePeriod=45
Oct 30 01:55:53.381198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.379352    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s" podUID="b8b27e40-2516-4d18-8c84-5e0925b3f4d3" containerName="main" containerID="cri-o://e3f65bdc61c5ec08e2a303255fb947d0cb3afdf8704bde493afbddd9f7edb1f4" gracePeriod=45
Oct 30 01:55:53.712933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.712901    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 01:55:53.718007 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.717986    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 01:55:53.746322 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.746299    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 01:55:53.793510 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.793481    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-model-cache\") pod \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") "
Oct 30 01:55:53.793696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.793554    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-tmp-dir\") pod \"0824ed99-cf93-4047-b19f-0ad807263acf\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") "
Oct 30 01:55:53.793696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.793599    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/834166ed-5c63-4854-9e89-c45041553630-kube-api-access-8f8k2\" (UniqueName: \"kubernetes.io/projected/834166ed-5c63-4854-9e89-c45041553630-kube-api-access-8f8k2\") pod \"834166ed-5c63-4854-9e89-c45041553630\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") "
Oct 30 01:55:53.793696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.793628    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0824ed99-cf93-4047-b19f-0ad807263acf-kube-api-access-gthgp\" (UniqueName: \"kubernetes.io/projected/0824ed99-cf93-4047-b19f-0ad807263acf-kube-api-access-gthgp\") pod \"0824ed99-cf93-4047-b19f-0ad807263acf\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") "
Oct 30 01:55:53.793696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.793663    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0824ed99-cf93-4047-b19f-0ad807263acf-tls-certs\" (UniqueName: \"kubernetes.io/secret/0824ed99-cf93-4047-b19f-0ad807263acf-tls-certs\") pod \"0824ed99-cf93-4047-b19f-0ad807263acf\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") "
Oct 30 01:55:53.793696 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.793690    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-home\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-home\") pod \"834166ed-5c63-4854-9e89-c45041553630\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") "
Oct 30 01:55:53.793995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.793764    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-model-cache" pod "b8b27e40-2516-4d18-8c84-5e0925b3f4d3" (UID: "b8b27e40-2516-4d18-8c84-5e0925b3f4d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:53.793995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.793828    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-home\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-home\") pod \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") "
Oct 30 01:55:53.793995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.793857    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/834166ed-5c63-4854-9e89-c45041553630-tls-certs\" (UniqueName: \"kubernetes.io/secret/834166ed-5c63-4854-9e89-c45041553630-tls-certs\") pod \"834166ed-5c63-4854-9e89-c45041553630\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") "
Oct 30 01:55:53.793995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.793877    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-tmp-dir\") pod \"834166ed-5c63-4854-9e89-c45041553630\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") "
Oct 30 01:55:53.793995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.793864    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-tmp-dir" pod "0824ed99-cf93-4047-b19f-0ad807263acf" (UID: "0824ed99-cf93-4047-b19f-0ad807263acf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:53.793995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.793915    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-model-cache\") pod \"834166ed-5c63-4854-9e89-c45041553630\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") "
Oct 30 01:55:53.793995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.793948    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-tmp-dir\") pod \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") "
Oct 30 01:55:53.793995 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.793979    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-model-cache\") pod \"0824ed99-cf93-4047-b19f-0ad807263acf\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") "
Oct 30 01:55:53.795006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.794005    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-dshm\") pod \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") "
Oct 30 01:55:53.795006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.794037    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-dshm\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-dshm\") pod \"834166ed-5c63-4854-9e89-c45041553630\" (UID: \"834166ed-5c63-4854-9e89-c45041553630\") "
Oct 30 01:55:53.795006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.794042    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-home" pod "b8b27e40-2516-4d18-8c84-5e0925b3f4d3" (UID: "b8b27e40-2516-4d18-8c84-5e0925b3f4d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:53.795006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.794110    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-kube-api-access-d97kg\" (UniqueName: \"kubernetes.io/projected/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-kube-api-access-d97kg\") pod \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") "
Oct 30 01:55:53.795006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.794112    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-tmp-dir" pod "834166ed-5c63-4854-9e89-c45041553630" (UID: "834166ed-5c63-4854-9e89-c45041553630"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:53.795006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.794230    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-tls-certs\") pod \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\" (UID: \"b8b27e40-2516-4d18-8c84-5e0925b3f4d3\") "
Oct 30 01:55:53.795006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.794290    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-home\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-home\") pod \"0824ed99-cf93-4047-b19f-0ad807263acf\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") "
Oct 30 01:55:53.795006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.794350    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-dshm\") pod \"0824ed99-cf93-4047-b19f-0ad807263acf\" (UID: \"0824ed99-cf93-4047-b19f-0ad807263acf\") "
Oct 30 01:55:53.795006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.794475    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-model-cache" pod "834166ed-5c63-4854-9e89-c45041553630" (UID: "834166ed-5c63-4854-9e89-c45041553630"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:53.795006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.794634    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-tmp-dir" pod "b8b27e40-2516-4d18-8c84-5e0925b3f4d3" (UID: "b8b27e40-2516-4d18-8c84-5e0925b3f4d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:53.795006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.794782    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-home" pod "834166ed-5c63-4854-9e89-c45041553630" (UID: "834166ed-5c63-4854-9e89-c45041553630"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:53.795006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.794827    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.795723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.795055    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.795723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.795081    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.795723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.795095    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.795723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.795107    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.795723 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.795141    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.796560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.796532    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-model-cache" pod "0824ed99-cf93-4047-b19f-0ad807263acf" (UID: "0824ed99-cf93-4047-b19f-0ad807263acf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:53.796831 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.796810    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-home" pod "0824ed99-cf93-4047-b19f-0ad807263acf" (UID: "0824ed99-cf93-4047-b19f-0ad807263acf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:53.797309 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.797281    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/834166ed-5c63-4854-9e89-c45041553630-tls-certs" pod "834166ed-5c63-4854-9e89-c45041553630" (UID: "834166ed-5c63-4854-9e89-c45041553630"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:53.797491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.797466    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834166ed-5c63-4854-9e89-c45041553630-kube-api-access-8f8k2" pod "834166ed-5c63-4854-9e89-c45041553630" (UID: "834166ed-5c63-4854-9e89-c45041553630"). InnerVolumeSpecName "kube-api-access-8f8k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:53.798259 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.798220    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-tls-certs" pod "b8b27e40-2516-4d18-8c84-5e0925b3f4d3" (UID: "b8b27e40-2516-4d18-8c84-5e0925b3f4d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:53.799222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.798807    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-dshm" pod "0824ed99-cf93-4047-b19f-0ad807263acf" (UID: "0824ed99-cf93-4047-b19f-0ad807263acf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:53.799222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.798883    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0824ed99-cf93-4047-b19f-0ad807263acf-tls-certs" pod "0824ed99-cf93-4047-b19f-0ad807263acf" (UID: "0824ed99-cf93-4047-b19f-0ad807263acf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:55:53.799222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.798980    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0824ed99-cf93-4047-b19f-0ad807263acf-kube-api-access-gthgp" pod "0824ed99-cf93-4047-b19f-0ad807263acf" (UID: "0824ed99-cf93-4047-b19f-0ad807263acf"). InnerVolumeSpecName "kube-api-access-gthgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:53.801666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.799572    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-dshm" pod "834166ed-5c63-4854-9e89-c45041553630" (UID: "834166ed-5c63-4854-9e89-c45041553630"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:53.801666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.799602    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-kube-api-access-d97kg" pod "b8b27e40-2516-4d18-8c84-5e0925b3f4d3" (UID: "b8b27e40-2516-4d18-8c84-5e0925b3f4d3"). InnerVolumeSpecName "kube-api-access-d97kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:53.801666 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.800293    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-dshm" pod "b8b27e40-2516-4d18-8c84-5e0925b3f4d3" (UID: "b8b27e40-2516-4d18-8c84-5e0925b3f4d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:53.896921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.896832    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.896921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.896871    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f8k2\" (UniqueName: \"kubernetes.io/projected/834166ed-5c63-4854-9e89-c45041553630-kube-api-access-8f8k2\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.896921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.896887    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gthgp\" (UniqueName: \"kubernetes.io/projected/0824ed99-cf93-4047-b19f-0ad807263acf-kube-api-access-gthgp\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.896921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.896901    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0824ed99-cf93-4047-b19f-0ad807263acf-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.896921 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.896917    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.897442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.896930    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834166ed-5c63-4854-9e89-c45041553630-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.897442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.896942    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.897442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.896954    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.897442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.896968    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834166ed-5c63-4854-9e89-c45041553630-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.897442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.896980    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d97kg\" (UniqueName: \"kubernetes.io/projected/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-kube-api-access-d97kg\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.897442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.896994    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b27e40-2516-4d18-8c84-5e0925b3f4d3-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.897442 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.897009    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0824ed99-cf93-4047-b19f-0ad807263acf-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:53.958346 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:53.958305    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:55:54.167829 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.167742    4425 generic.go:356] "Generic (PLEG): container finished" podID="0824ed99-cf93-4047-b19f-0ad807263acf" containerID="b76b959e601a3ff544ff0f9b10cf6f78e86389b1f64eb367563f787bb3be6769" exitCode=0
Oct 30 01:55:54.167829 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.167789    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"
Oct 30 01:55:54.168358 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.167823    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f" event={"ID":"0824ed99-cf93-4047-b19f-0ad807263acf","Type":"ContainerDied","Data":"b76b959e601a3ff544ff0f9b10cf6f78e86389b1f64eb367563f787bb3be6769"}
Oct 30 01:55:54.168358 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.167869    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f" event={"ID":"0824ed99-cf93-4047-b19f-0ad807263acf","Type":"ContainerDied","Data":"619cff782a4c0aee4b77c29336f40bf5f8478ec544c360962e163158ce8cf28c"}
Oct 30 01:55:54.168358 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.167886    4425 scope.go:122] "RemoveContainer" containerID="b76b959e601a3ff544ff0f9b10cf6f78e86389b1f64eb367563f787bb3be6769"
Oct 30 01:55:54.169451 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.169432    4425 generic.go:356] "Generic (PLEG): container finished" podID="834166ed-5c63-4854-9e89-c45041553630" containerID="1d9fcf60190de75e8075d350084589723f3ee8ea276930870f980a0161106997" exitCode=0
Oct 30 01:55:54.169586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.169498    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"
Oct 30 01:55:54.169660 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.169534    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7" event={"ID":"834166ed-5c63-4854-9e89-c45041553630","Type":"ContainerDied","Data":"1d9fcf60190de75e8075d350084589723f3ee8ea276930870f980a0161106997"}
Oct 30 01:55:54.169711 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.169653    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7" event={"ID":"834166ed-5c63-4854-9e89-c45041553630","Type":"ContainerDied","Data":"ed73f89f1ad4427985f670a7c69f855663db32459eba6b59441610329f97404f"}
Oct 30 01:55:54.170966 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.170946    4425 generic.go:356] "Generic (PLEG): container finished" podID="b8b27e40-2516-4d18-8c84-5e0925b3f4d3" containerID="e3f65bdc61c5ec08e2a303255fb947d0cb3afdf8704bde493afbddd9f7edb1f4" exitCode=0
Oct 30 01:55:54.171100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.170995    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"
Oct 30 01:55:54.171100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.171010    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s" event={"ID":"b8b27e40-2516-4d18-8c84-5e0925b3f4d3","Type":"ContainerDied","Data":"e3f65bdc61c5ec08e2a303255fb947d0cb3afdf8704bde493afbddd9f7edb1f4"}
Oct 30 01:55:54.171100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.171033    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s" event={"ID":"b8b27e40-2516-4d18-8c84-5e0925b3f4d3","Type":"ContainerDied","Data":"49b49b291239bffbb813a19aed6deb69860f697e977e43e5679d9a8508fd49f8"}
Oct 30 01:55:54.181274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.180339    4425 scope.go:122] "RemoveContainer" containerID="b76b959e601a3ff544ff0f9b10cf6f78e86389b1f64eb367563f787bb3be6769"
Oct 30 01:55:54.181274 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:55:54.180669    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b76b959e601a3ff544ff0f9b10cf6f78e86389b1f64eb367563f787bb3be6769\": container with ID starting with b76b959e601a3ff544ff0f9b10cf6f78e86389b1f64eb367563f787bb3be6769 not found: ID does not exist" containerID="b76b959e601a3ff544ff0f9b10cf6f78e86389b1f64eb367563f787bb3be6769"
Oct 30 01:55:54.181274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.180705    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b76b959e601a3ff544ff0f9b10cf6f78e86389b1f64eb367563f787bb3be6769"} err="failed to get container status \"b76b959e601a3ff544ff0f9b10cf6f78e86389b1f64eb367563f787bb3be6769\": rpc error: code = NotFound desc = could not find container \"b76b959e601a3ff544ff0f9b10cf6f78e86389b1f64eb367563f787bb3be6769\": container with ID starting with b76b959e601a3ff544ff0f9b10cf6f78e86389b1f64eb367563f787bb3be6769 not found: ID does not exist"
Oct 30 01:55:54.181274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.180728    4425 scope.go:122] "RemoveContainer" containerID="1d9fcf60190de75e8075d350084589723f3ee8ea276930870f980a0161106997"
Oct 30 01:55:54.191480 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.191463    4425 scope.go:122] "RemoveContainer" containerID="1d9fcf60190de75e8075d350084589723f3ee8ea276930870f980a0161106997"
Oct 30 01:55:54.191714 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:55:54.191696    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d9fcf60190de75e8075d350084589723f3ee8ea276930870f980a0161106997\": container with ID starting with 1d9fcf60190de75e8075d350084589723f3ee8ea276930870f980a0161106997 not found: ID does not exist" containerID="1d9fcf60190de75e8075d350084589723f3ee8ea276930870f980a0161106997"
Oct 30 01:55:54.191767 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.191721    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d9fcf60190de75e8075d350084589723f3ee8ea276930870f980a0161106997"} err="failed to get container status \"1d9fcf60190de75e8075d350084589723f3ee8ea276930870f980a0161106997\": rpc error: code = NotFound desc = could not find container \"1d9fcf60190de75e8075d350084589723f3ee8ea276930870f980a0161106997\": container with ID starting with 1d9fcf60190de75e8075d350084589723f3ee8ea276930870f980a0161106997 not found: ID does not exist"
Oct 30 01:55:54.191767 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.191738    4425 scope.go:122] "RemoveContainer" containerID="e3f65bdc61c5ec08e2a303255fb947d0cb3afdf8704bde493afbddd9f7edb1f4"
Oct 30 01:55:54.200096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.200077    4425 scope.go:122] "RemoveContainer" containerID="e3f65bdc61c5ec08e2a303255fb947d0cb3afdf8704bde493afbddd9f7edb1f4"
Oct 30 01:55:54.200373 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:55:54.200352    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f65bdc61c5ec08e2a303255fb947d0cb3afdf8704bde493afbddd9f7edb1f4\": container with ID starting with e3f65bdc61c5ec08e2a303255fb947d0cb3afdf8704bde493afbddd9f7edb1f4 not found: ID does not exist" containerID="e3f65bdc61c5ec08e2a303255fb947d0cb3afdf8704bde493afbddd9f7edb1f4"
Oct 30 01:55:54.200443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.200381    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f65bdc61c5ec08e2a303255fb947d0cb3afdf8704bde493afbddd9f7edb1f4"} err="failed to get container status \"e3f65bdc61c5ec08e2a303255fb947d0cb3afdf8704bde493afbddd9f7edb1f4\": rpc error: code = NotFound desc = could not find container \"e3f65bdc61c5ec08e2a303255fb947d0cb3afdf8704bde493afbddd9f7edb1f4\": container with ID starting with e3f65bdc61c5ec08e2a303255fb947d0cb3afdf8704bde493afbddd9f7edb1f4 not found: ID does not exist"
Oct 30 01:55:54.252574 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.252551    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"]
Oct 30 01:55:54.278230 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.278204    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/distinct-w0-9bfe36-kserve-bdb675f8-k2dj7"]
Oct 30 01:55:54.354994 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.354968    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"]
Oct 30 01:55:54.373975 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.373950    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/distinct-2-w0-9bfe36-kserve-7c78b75dfc-sxn5f"]
Oct 30 01:55:54.427527 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.427451    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"]
Oct 30 01:55:54.432186 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:54.432146    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9bfe36/simulator-w0-9bfe36-kserve-6b9d5ddb5d-w8r9s"]
Oct 30 01:55:55.420903 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:55.420860    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0824ed99-cf93-4047-b19f-0ad807263acf" path="/var/lib/kubelet/pods/0824ed99-cf93-4047-b19f-0ad807263acf/volumes"
Oct 30 01:55:55.421495 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:55.421473    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="834166ed-5c63-4854-9e89-c45041553630" path="/var/lib/kubelet/pods/834166ed-5c63-4854-9e89-c45041553630/volumes"
Oct 30 01:55:55.421914 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:55.421894    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b27e40-2516-4d18-8c84-5e0925b3f4d3" path="/var/lib/kubelet/pods/b8b27e40-2516-4d18-8c84-5e0925b3f4d3/volumes"
Oct 30 01:55:56.805490 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:56.805452    4425 patch_prober.go:29] interesting pod/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:55:56.805874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:56.805514    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr" podUID="998232e0-4d7b-4a4b-9fd1-5eb005157b11" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:55:57.540014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:57.539971    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r" podUID="8e763d6f-32ac-4739-9e2d-feca894a8987" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:55:58.034184 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.034139    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"]
Oct 30 01:55:58.034809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.034786    4425 cpu_manager.go:414] "containerMap: removing container" podUID="d672322e-d956-4654-9e06-490aa9fb1518" containerName="revoke-keys"
Oct 30 01:55:58.034901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.034811    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d672322e-d956-4654-9e06-490aa9fb1518" containerName="revoke-keys"
Oct 30 01:55:58.034901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.034825    4425 cpu_manager.go:414] "containerMap: removing container" podUID="0824ed99-cf93-4047-b19f-0ad807263acf" containerName="main"
Oct 30 01:55:58.034901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.034832    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0824ed99-cf93-4047-b19f-0ad807263acf" containerName="main"
Oct 30 01:55:58.034901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.034855    4425 cpu_manager.go:414] "containerMap: removing container" podUID="019931b1-92c8-48a2-88be-2a0fb9b18450" containerName="revoke-keys"
Oct 30 01:55:58.034901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.034862    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="019931b1-92c8-48a2-88be-2a0fb9b18450" containerName="revoke-keys"
Oct 30 01:55:58.034901 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.034896    4425 cpu_manager.go:414] "containerMap: removing container" podUID="834166ed-5c63-4854-9e89-c45041553630" containerName="main"
Oct 30 01:55:58.035282 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.034903    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="834166ed-5c63-4854-9e89-c45041553630" containerName="main"
Oct 30 01:55:58.035282 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.034922    4425 cpu_manager.go:414] "containerMap: removing container" podUID="b8b27e40-2516-4d18-8c84-5e0925b3f4d3" containerName="main"
Oct 30 01:55:58.035282 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.034930    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b27e40-2516-4d18-8c84-5e0925b3f4d3" containerName="main"
Oct 30 01:55:58.035282 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.035038    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="d672322e-d956-4654-9e06-490aa9fb1518" containerName="revoke-keys"
Oct 30 01:55:58.035282 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.035051    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="0824ed99-cf93-4047-b19f-0ad807263acf" containerName="main"
Oct 30 01:55:58.035282 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.035065    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="019931b1-92c8-48a2-88be-2a0fb9b18450" containerName="revoke-keys"
Oct 30 01:55:58.035282 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.035078    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="834166ed-5c63-4854-9e89-c45041553630" containerName="main"
Oct 30 01:55:58.035282 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.035087    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b27e40-2516-4d18-8c84-5e0925b3f4d3" containerName="main"
Oct 30 01:55:58.038633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.038612    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"
Oct 30 01:55:58.051095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.051073    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:55:58.051212 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.051139    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:55:58.065215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.065147    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"]
Oct 30 01:55:58.088914 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.088892    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:55:58.089021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.088921    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:55:58.137848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.137816    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvczk\" (UniqueName: \"kubernetes.io/projected/82735c67-cda8-490f-a0a9-42db3b56992a-kube-api-access-nvczk\") pod \"maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz\" (UID: \"82735c67-cda8-490f-a0a9-42db3b56992a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"
Oct 30 01:55:58.138013 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.137860    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82735c67-cda8-490f-a0a9-42db3b56992a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz\" (UID: \"82735c67-cda8-490f-a0a9-42db3b56992a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"
Oct 30 01:55:58.138144 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.138016    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82735c67-cda8-490f-a0a9-42db3b56992a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz\" (UID: \"82735c67-cda8-490f-a0a9-42db3b56992a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"
Oct 30 01:55:58.208587 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.208551    4425 generic.go:356] "Generic (PLEG): container finished" podID="998232e0-4d7b-4a4b-9fd1-5eb005157b11" containerID="cde5f6edf92232a90174fc5976205803909583e7b1e3994bf76707307795412f" exitCode=0
Oct 30 01:55:58.208756 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.208616    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr" event={"ID":"998232e0-4d7b-4a4b-9fd1-5eb005157b11","Type":"ContainerDied","Data":"cde5f6edf92232a90174fc5976205803909583e7b1e3994bf76707307795412f"}
Oct 30 01:55:58.238569 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.238543    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82735c67-cda8-490f-a0a9-42db3b56992a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz\" (UID: \"82735c67-cda8-490f-a0a9-42db3b56992a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"
Oct 30 01:55:58.238676 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.238618    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvczk\" (UniqueName: \"kubernetes.io/projected/82735c67-cda8-490f-a0a9-42db3b56992a-kube-api-access-nvczk\") pod \"maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz\" (UID: \"82735c67-cda8-490f-a0a9-42db3b56992a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"
Oct 30 01:55:58.238676 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.238647    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82735c67-cda8-490f-a0a9-42db3b56992a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz\" (UID: \"82735c67-cda8-490f-a0a9-42db3b56992a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"
Oct 30 01:55:58.239327 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.239303    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82735c67-cda8-490f-a0a9-42db3b56992a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz\" (UID: \"82735c67-cda8-490f-a0a9-42db3b56992a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"
Oct 30 01:55:58.240793 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.240764    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82735c67-cda8-490f-a0a9-42db3b56992a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz\" (UID: \"82735c67-cda8-490f-a0a9-42db3b56992a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"
Oct 30 01:55:58.241452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.241436    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 01:55:58.259218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.259151    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvczk\" (UniqueName: \"kubernetes.io/projected/82735c67-cda8-490f-a0a9-42db3b56992a-kube-api-access-nvczk\") pod \"maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz\" (UID: \"82735c67-cda8-490f-a0a9-42db3b56992a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"
Oct 30 01:55:58.339711 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.339673    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-envoy\") pod \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") "
Oct 30 01:55:58.339889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.339725    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998232e0-4d7b-4a4b-9fd1-5eb005157b11-kube-api-access-lcfc6\" (UniqueName: \"kubernetes.io/projected/998232e0-4d7b-4a4b-9fd1-5eb005157b11-kube-api-access-lcfc6\") pod \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") "
Oct 30 01:55:58.339889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.339756    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-podinfo\") pod \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") "
Oct 30 01:55:58.339889 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.339786    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-data\") pod \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") "
Oct 30 01:55:58.340064 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.339926    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-token\" (UniqueName: \"kubernetes.io/projected/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-token\") pod \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") "
Oct 30 01:55:58.340064 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.339963    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-credential-socket\") pod \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") "
Oct 30 01:55:58.340064 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.340000    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-workload-socket\") pod \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") "
Oct 30 01:55:58.340274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.340065    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-workload-certs\") pod \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") "
Oct 30 01:55:58.340274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.340115    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-data" pod "998232e0-4d7b-4a4b-9fd1-5eb005157b11" (UID: "998232e0-4d7b-4a4b-9fd1-5eb005157b11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:58.340274 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.340123    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istiod-ca-cert\") pod \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\" (UID: \"998232e0-4d7b-4a4b-9fd1-5eb005157b11\") "
Oct 30 01:55:58.340456 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.340351    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-workload-socket" pod "998232e0-4d7b-4a4b-9fd1-5eb005157b11" (UID: "998232e0-4d7b-4a4b-9fd1-5eb005157b11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:58.340516 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.340455    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istiod-ca-cert" pod "998232e0-4d7b-4a4b-9fd1-5eb005157b11" (UID: "998232e0-4d7b-4a4b-9fd1-5eb005157b11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:55:58.340584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.340554    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-workload-certs" pod "998232e0-4d7b-4a4b-9fd1-5eb005157b11" (UID: "998232e0-4d7b-4a4b-9fd1-5eb005157b11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:58.340584 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.340574    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-credential-socket" pod "998232e0-4d7b-4a4b-9fd1-5eb005157b11" (UID: "998232e0-4d7b-4a4b-9fd1-5eb005157b11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:58.340739 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.340721    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:58.340809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.340745    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:58.340809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.340758    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:58.340809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.340771    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:58.340809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.340785    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:58.342430 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.342410    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-podinfo" pod "998232e0-4d7b-4a4b-9fd1-5eb005157b11" (UID: "998232e0-4d7b-4a4b-9fd1-5eb005157b11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:55:58.342511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.342458    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-token" pod "998232e0-4d7b-4a4b-9fd1-5eb005157b11" (UID: "998232e0-4d7b-4a4b-9fd1-5eb005157b11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:58.342552 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.342513    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-envoy" pod "998232e0-4d7b-4a4b-9fd1-5eb005157b11" (UID: "998232e0-4d7b-4a4b-9fd1-5eb005157b11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:55:58.342552 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.342535    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998232e0-4d7b-4a4b-9fd1-5eb005157b11-kube-api-access-lcfc6" pod "998232e0-4d7b-4a4b-9fd1-5eb005157b11" (UID: "998232e0-4d7b-4a4b-9fd1-5eb005157b11"). InnerVolumeSpecName "kube-api-access-lcfc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:55:58.375665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.375635    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"
Oct 30 01:55:58.446822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.446794    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:58.446822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.446822    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcfc6\" (UniqueName: \"kubernetes.io/projected/998232e0-4d7b-4a4b-9fd1-5eb005157b11-kube-api-access-lcfc6\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:58.446822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.446832    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:58.447096 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.446847    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998232e0-4d7b-4a4b-9fd1-5eb005157b11-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:55:58.549870 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.549831    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8" podUID="be776653-8806-4154-96d5-87ce0af15d97" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:55:58.678071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:58.678046    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"]
Oct 30 01:55:58.678962 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:55:58.678939    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82735c67_cda8_490f_a0a9_42db3b56992a.slice/crio-e5c0be014db8654c96921c53319ed89ebee79ebe11d476f428a9a658ecae393d WatchSource:0}: Error finding container e5c0be014db8654c96921c53319ed89ebee79ebe11d476f428a9a658ecae393d: Status 404 returned error can't find the container with id e5c0be014db8654c96921c53319ed89ebee79ebe11d476f428a9a658ecae393d
Oct 30 01:55:59.216402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:59.216328    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"
Oct 30 01:55:59.216402 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:59.216360    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr" event={"ID":"998232e0-4d7b-4a4b-9fd1-5eb005157b11","Type":"ContainerDied","Data":"8ef843bf9c9f1cd2f45cb1889d256d110c8bb7f0ab84f58ccb36d66ae765e8bd"}
Oct 30 01:55:59.216848 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:59.216424    4425 scope.go:122] "RemoveContainer" containerID="cde5f6edf92232a90174fc5976205803909583e7b1e3994bf76707307795412f"
Oct 30 01:55:59.224568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:59.224532    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz" event={"ID":"82735c67-cda8-490f-a0a9-42db3b56992a","Type":"ContainerStarted","Data":"38e058b267db315595ca753492188c8a93dde29b6baa16b4a725239540a8d622"}
Oct 30 01:55:59.224713 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:59.224595    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz" event={"ID":"82735c67-cda8-490f-a0a9-42db3b56992a","Type":"ContainerStarted","Data":"e5c0be014db8654c96921c53319ed89ebee79ebe11d476f428a9a658ecae393d"}
Oct 30 01:55:59.280399 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:59.280355    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz" podStartSLOduration=1.280341913 podStartE2EDuration="1.280341913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:59.276788721 +0000 UTC m=+4468.026444031" watchObservedRunningTime="2026-10-30 01:55:59.280341913 +0000 UTC m=+4468.029997221"
Oct 30 01:55:59.309147 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:59.309119    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"]
Oct 30 01:55:59.319026 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:59.318997    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-72ed13-openshift-default-74c5595989-x4whr"]
Oct 30 01:55:59.429380 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:55:59.429328    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998232e0-4d7b-4a4b-9fd1-5eb005157b11" path="/var/lib/kubelet/pods/998232e0-4d7b-4a4b-9fd1-5eb005157b11/volumes"
Oct 30 01:56:00.233962 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:00.233925    4425 generic.go:356] "Generic (PLEG): container finished" podID="82735c67-cda8-490f-a0a9-42db3b56992a" containerID="38e058b267db315595ca753492188c8a93dde29b6baa16b4a725239540a8d622" exitCode=0
Oct 30 01:56:00.234359 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:00.234002    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz" event={"ID":"82735c67-cda8-490f-a0a9-42db3b56992a","Type":"ContainerDied","Data":"38e058b267db315595ca753492188c8a93dde29b6baa16b4a725239540a8d622"}
Oct 30 01:56:01.377524 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:01.377504    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"
Oct 30 01:56:01.477491 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:01.477463    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82735c67-cda8-490f-a0a9-42db3b56992a-kube-api-access-nvczk\" (UniqueName: \"kubernetes.io/projected/82735c67-cda8-490f-a0a9-42db3b56992a-kube-api-access-nvczk\") pod \"82735c67-cda8-490f-a0a9-42db3b56992a\" (UID: \"82735c67-cda8-490f-a0a9-42db3b56992a\") "
Oct 30 01:56:01.477633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:01.477505    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82735c67-cda8-490f-a0a9-42db3b56992a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82735c67-cda8-490f-a0a9-42db3b56992a-maas-api-openshift-ca\") pod \"82735c67-cda8-490f-a0a9-42db3b56992a\" (UID: \"82735c67-cda8-490f-a0a9-42db3b56992a\") "
Oct 30 01:56:01.477633 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:01.477570    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82735c67-cda8-490f-a0a9-42db3b56992a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82735c67-cda8-490f-a0a9-42db3b56992a-maas-api-xks-ca\") pod \"82735c67-cda8-490f-a0a9-42db3b56992a\" (UID: \"82735c67-cda8-490f-a0a9-42db3b56992a\") "
Oct 30 01:56:01.477913 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:01.477884    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82735c67-cda8-490f-a0a9-42db3b56992a-maas-api-openshift-ca" pod "82735c67-cda8-490f-a0a9-42db3b56992a" (UID: "82735c67-cda8-490f-a0a9-42db3b56992a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:56:01.479595 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:01.479561    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82735c67-cda8-490f-a0a9-42db3b56992a-maas-api-xks-ca" pod "82735c67-cda8-490f-a0a9-42db3b56992a" (UID: "82735c67-cda8-490f-a0a9-42db3b56992a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:56:01.479729 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:01.479711    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82735c67-cda8-490f-a0a9-42db3b56992a-kube-api-access-nvczk" pod "82735c67-cda8-490f-a0a9-42db3b56992a" (UID: "82735c67-cda8-490f-a0a9-42db3b56992a"). InnerVolumeSpecName "kube-api-access-nvczk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:56:01.579215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:01.579191    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvczk\" (UniqueName: \"kubernetes.io/projected/82735c67-cda8-490f-a0a9-42db3b56992a-kube-api-access-nvczk\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:01.579215 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:01.579213    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82735c67-cda8-490f-a0a9-42db3b56992a-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:01.579344 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:01.579224    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82735c67-cda8-490f-a0a9-42db3b56992a-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:02.248049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:02.248017    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"
Oct 30 01:56:02.248269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:02.248056    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz" event={"ID":"82735c67-cda8-490f-a0a9-42db3b56992a","Type":"ContainerDied","Data":"e5c0be014db8654c96921c53319ed89ebee79ebe11d476f428a9a658ecae393d"}
Oct 30 01:56:02.248269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:02.248103    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5c0be014db8654c96921c53319ed89ebee79ebe11d476f428a9a658ecae393d"
Oct 30 01:56:07.036503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:07.036462    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"]
Oct 30 01:56:07.044863 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:07.044832    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a5bd9df-hxvcl"]
Oct 30 01:56:07.421688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:07.421601    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c936006-5a77-484b-85d9-d74842f1e607" path="/var/lib/kubelet/pods/8c936006-5a77-484b-85d9-d74842f1e607/volumes"
Oct 30 01:56:07.538350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:07.538321    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"]
Oct 30 01:56:07.538609 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:56:07.538594    4425 logging.go:55] [core] [Channel #24188 SubChannel #24189]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.226:50052", ServerName: "10.133.0.226:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.226:50052: connect: connection refused"
Oct 30 01:56:07.548535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:07.548498    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bfe36-kvr2v"]
Oct 30 01:56:07.558017 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:56:07.557968    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:56:07.752009 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:07.751987    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"
Oct 30 01:56:07.840827 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:07.840794    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e763d6f-32ac-4739-9e2d-feca894a8987-kube-api-access-fpn6f\" (UniqueName: \"kubernetes.io/projected/8e763d6f-32ac-4739-9e2d-feca894a8987-kube-api-access-fpn6f\") pod \"8e763d6f-32ac-4739-9e2d-feca894a8987\" (UID: \"8e763d6f-32ac-4739-9e2d-feca894a8987\") "
Oct 30 01:56:07.840992 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:07.840944    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e763d6f-32ac-4739-9e2d-feca894a8987-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e763d6f-32ac-4739-9e2d-feca894a8987-plugins-config-volume\") pod \"8e763d6f-32ac-4739-9e2d-feca894a8987\" (UID: \"8e763d6f-32ac-4739-9e2d-feca894a8987\") "
Oct 30 01:56:07.841291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:07.841271    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e763d6f-32ac-4739-9e2d-feca894a8987-plugins-config-volume" pod "8e763d6f-32ac-4739-9e2d-feca894a8987" (UID: "8e763d6f-32ac-4739-9e2d-feca894a8987"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:56:07.843019 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:07.842998    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e763d6f-32ac-4739-9e2d-feca894a8987-kube-api-access-fpn6f" pod "8e763d6f-32ac-4739-9e2d-feca894a8987" (UID: "8e763d6f-32ac-4739-9e2d-feca894a8987"). InnerVolumeSpecName "kube-api-access-fpn6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:56:07.941788 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:07.941747    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e763d6f-32ac-4739-9e2d-feca894a8987-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:07.941788 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:07.941784    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpn6f\" (UniqueName: \"kubernetes.io/projected/8e763d6f-32ac-4739-9e2d-feca894a8987-kube-api-access-fpn6f\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:08.278032 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.278008    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"
Oct 30 01:56:08.290179 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.290123    4425 generic.go:356] "Generic (PLEG): container finished" podID="8e763d6f-32ac-4739-9e2d-feca894a8987" containerID="964227f2dd072fd7eb6c8b5fa6ab20ffddfbcb9c23cc8e8a5ec71f5b01a3754b" exitCode=0
Oct 30 01:56:08.290305 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.290182    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"
Oct 30 01:56:08.290305 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.290189    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r" event={"ID":"8e763d6f-32ac-4739-9e2d-feca894a8987","Type":"ContainerDied","Data":"964227f2dd072fd7eb6c8b5fa6ab20ffddfbcb9c23cc8e8a5ec71f5b01a3754b"}
Oct 30 01:56:08.290305 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.290260    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r" event={"ID":"8e763d6f-32ac-4739-9e2d-feca894a8987","Type":"ContainerDied","Data":"2b99b1d7f64f7868fd0eee84929bfc7d9c352646dca06e16bd5e2e8f477b206f"}
Oct 30 01:56:08.290305 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.290291    4425 scope.go:122] "RemoveContainer" containerID="964227f2dd072fd7eb6c8b5fa6ab20ffddfbcb9c23cc8e8a5ec71f5b01a3754b"
Oct 30 01:56:08.291806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.291782    4425 generic.go:356] "Generic (PLEG): container finished" podID="be776653-8806-4154-96d5-87ce0af15d97" containerID="399e0e35809f1fff95994362539fa2c16aa092d3c61bcf67c1bc060f80facb1b" exitCode=0
Oct 30 01:56:08.291915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.291830    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"
Oct 30 01:56:08.291915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.291859    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8" event={"ID":"be776653-8806-4154-96d5-87ce0af15d97","Type":"ContainerDied","Data":"399e0e35809f1fff95994362539fa2c16aa092d3c61bcf67c1bc060f80facb1b"}
Oct 30 01:56:08.291915 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.291895    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8" event={"ID":"be776653-8806-4154-96d5-87ce0af15d97","Type":"ContainerDied","Data":"de92ed495f235e82172a4c380be4ddd4e81fa16a0198e5a523b1616a2598599e"}
Oct 30 01:56:08.300447 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.300427    4425 scope.go:122] "RemoveContainer" containerID="964227f2dd072fd7eb6c8b5fa6ab20ffddfbcb9c23cc8e8a5ec71f5b01a3754b"
Oct 30 01:56:08.300695 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:56:08.300675    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"964227f2dd072fd7eb6c8b5fa6ab20ffddfbcb9c23cc8e8a5ec71f5b01a3754b\": container with ID starting with 964227f2dd072fd7eb6c8b5fa6ab20ffddfbcb9c23cc8e8a5ec71f5b01a3754b not found: ID does not exist" containerID="964227f2dd072fd7eb6c8b5fa6ab20ffddfbcb9c23cc8e8a5ec71f5b01a3754b"
Oct 30 01:56:08.300747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.300702    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964227f2dd072fd7eb6c8b5fa6ab20ffddfbcb9c23cc8e8a5ec71f5b01a3754b"} err="failed to get container status \"964227f2dd072fd7eb6c8b5fa6ab20ffddfbcb9c23cc8e8a5ec71f5b01a3754b\": rpc error: code = NotFound desc = could not find container \"964227f2dd072fd7eb6c8b5fa6ab20ffddfbcb9c23cc8e8a5ec71f5b01a3754b\": container with ID starting with 964227f2dd072fd7eb6c8b5fa6ab20ffddfbcb9c23cc8e8a5ec71f5b01a3754b not found: ID does not exist"
Oct 30 01:56:08.300747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.300719    4425 scope.go:122] "RemoveContainer" containerID="399e0e35809f1fff95994362539fa2c16aa092d3c61bcf67c1bc060f80facb1b"
Oct 30 01:56:08.310677 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.310659    4425 scope.go:122] "RemoveContainer" containerID="399e0e35809f1fff95994362539fa2c16aa092d3c61bcf67c1bc060f80facb1b"
Oct 30 01:56:08.310969 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:56:08.310953    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399e0e35809f1fff95994362539fa2c16aa092d3c61bcf67c1bc060f80facb1b\": container with ID starting with 399e0e35809f1fff95994362539fa2c16aa092d3c61bcf67c1bc060f80facb1b not found: ID does not exist" containerID="399e0e35809f1fff95994362539fa2c16aa092d3c61bcf67c1bc060f80facb1b"
Oct 30 01:56:08.311010 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.310977    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399e0e35809f1fff95994362539fa2c16aa092d3c61bcf67c1bc060f80facb1b"} err="failed to get container status \"399e0e35809f1fff95994362539fa2c16aa092d3c61bcf67c1bc060f80facb1b\": rpc error: code = NotFound desc = could not find container \"399e0e35809f1fff95994362539fa2c16aa092d3c61bcf67c1bc060f80facb1b\": container with ID starting with 399e0e35809f1fff95994362539fa2c16aa092d3c61bcf67c1bc060f80facb1b not found: ID does not exist"
Oct 30 01:56:08.345923 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.345880    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be776653-8806-4154-96d5-87ce0af15d97-kube-api-access-7ww6j\" (UniqueName: \"kubernetes.io/projected/be776653-8806-4154-96d5-87ce0af15d97-kube-api-access-7ww6j\") pod \"be776653-8806-4154-96d5-87ce0af15d97\" (UID: \"be776653-8806-4154-96d5-87ce0af15d97\") "
Oct 30 01:56:08.346137 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.346113    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be776653-8806-4154-96d5-87ce0af15d97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be776653-8806-4154-96d5-87ce0af15d97-plugins-config-volume\") pod \"be776653-8806-4154-96d5-87ce0af15d97\" (UID: \"be776653-8806-4154-96d5-87ce0af15d97\") "
Oct 30 01:56:08.346553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.346530    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be776653-8806-4154-96d5-87ce0af15d97-plugins-config-volume" pod "be776653-8806-4154-96d5-87ce0af15d97" (UID: "be776653-8806-4154-96d5-87ce0af15d97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:56:08.346989 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.346970    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"]
Oct 30 01:56:08.347911 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.347888    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be776653-8806-4154-96d5-87ce0af15d97-kube-api-access-7ww6j" pod "be776653-8806-4154-96d5-87ce0af15d97" (UID: "be776653-8806-4154-96d5-87ce0af15d97"). InnerVolumeSpecName "kube-api-access-7ww6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:56:08.353812 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.353786    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:56:08.357033 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.357010    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r"]
Oct 30 01:56:08.447567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.447531    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ww6j\" (UniqueName: \"kubernetes.io/projected/be776653-8806-4154-96d5-87ce0af15d97-kube-api-access-7ww6j\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:08.447567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.447562    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be776653-8806-4154-96d5-87ce0af15d97-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:08.472998 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.472973    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"]
Oct 30 01:56:08.487183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.487147    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"]
Oct 30 01:56:08.538452 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.538414    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2f820e-69dd5b495f-67h9r" podUID="8e763d6f-32ac-4739-9e2d-feca894a8987" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.226:50052\" within 1s: context deadline exceeded"
Oct 30 01:56:08.643763 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.643736    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"]
Oct 30 01:56:08.647796 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:08.647771    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2f820e-759c8c6d6b-s26t8"]
Oct 30 01:56:09.423585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:09.423548    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f35be2f-ef4a-4630-8a7f-c53fcf8ed977" path="/var/lib/kubelet/pods/4f35be2f-ef4a-4630-8a7f-c53fcf8ed977/volumes"
Oct 30 01:56:09.424119 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:09.424098    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e763d6f-32ac-4739-9e2d-feca894a8987" path="/var/lib/kubelet/pods/8e763d6f-32ac-4739-9e2d-feca894a8987/volumes"
Oct 30 01:56:09.424529 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:09.424509    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be776653-8806-4154-96d5-87ce0af15d97" path="/var/lib/kubelet/pods/be776653-8806-4154-96d5-87ce0af15d97/volumes"
Oct 30 01:56:13.473534 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:13.473493    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr" podUID="534f39ab-7cfa-43b1-bd72-ddfa1e8f702d" containerName="payload-processing" containerID="cri-o://46d0797f7b9535068c592eacfeae58c1f50af2ea61b90ed59205ca49965b459a" gracePeriod=25
Oct 30 01:56:13.487943 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:13.487911    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv" podUID="a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1" containerName="payload-pre-processing" containerID="cri-o://6404b1bb1d27408c32b30d3f43a9bdfe9bb9457f2e711f1a8a4372fd18ead5af" gracePeriod=25
Oct 30 01:56:14.989352 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:14.989316    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"]
Oct 30 01:56:14.990028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:14.989995    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg" podUID="fd21ddd7-1c9b-4543-9c58-b1656798e9ef" containerName="istio-proxy" containerID="cri-o://9b23b87592b851433ec0016cbe7d9fb8e24e5f9c3a337482328ae8b30691a438" gracePeriod=30
Oct 30 01:56:17.045362 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:17.045312    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"]
Oct 30 01:56:17.054822 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:17.054788    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811266ed-rq5kq"]
Oct 30 01:56:17.152089 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:17.152054    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"]
Oct 30 01:56:17.154455 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:17.154427    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1afb3e-hvddz"]
Oct 30 01:56:17.420707 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:17.420630    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03f31bab-7494-41c2-9f5e-16a326fa7082" path="/var/lib/kubelet/pods/03f31bab-7494-41c2-9f5e-16a326fa7082/volumes"
Oct 30 01:56:17.421071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:17.421057    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82735c67-cda8-490f-a0a9-42db3b56992a" path="/var/lib/kubelet/pods/82735c67-cda8-490f-a0a9-42db3b56992a/volumes"
Oct 30 01:56:20.179488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.179455    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr" podUID="534f39ab-7cfa-43b1-bd72-ddfa1e8f702d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:56:20.179488 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.179483    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv" podUID="a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:56:20.247045 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.247019    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 01:56:20.359391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.359313    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-workload-certs\") pod \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") "
Oct 30 01:56:20.359391 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.359368    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-data\") pod \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") "
Oct 30 01:56:20.359588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.359416    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-token\" (UniqueName: \"kubernetes.io/projected/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-token\") pod \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") "
Oct 30 01:56:20.359588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.359494    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-workload-socket\") pod \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") "
Oct 30 01:56:20.359588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.359532    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-kube-api-access-sdx2f\" (UniqueName: \"kubernetes.io/projected/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-kube-api-access-sdx2f\") pod \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") "
Oct 30 01:56:20.359588 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.359573    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-envoy\") pod \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") "
Oct 30 01:56:20.359817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.359612    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-podinfo\") pod \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") "
Oct 30 01:56:20.359817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.359635    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-workload-certs" pod "fd21ddd7-1c9b-4543-9c58-b1656798e9ef" (UID: "fd21ddd7-1c9b-4543-9c58-b1656798e9ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:56:20.359817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.359641    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-data" pod "fd21ddd7-1c9b-4543-9c58-b1656798e9ef" (UID: "fd21ddd7-1c9b-4543-9c58-b1656798e9ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:56:20.359817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.359675    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-credential-socket\") pod \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") "
Oct 30 01:56:20.359817 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.359709    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istiod-ca-cert\") pod \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\" (UID: \"fd21ddd7-1c9b-4543-9c58-b1656798e9ef\") "
Oct 30 01:56:20.360095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.359826    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-workload-socket" pod "fd21ddd7-1c9b-4543-9c58-b1656798e9ef" (UID: "fd21ddd7-1c9b-4543-9c58-b1656798e9ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:56:20.360193 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.360145    4425 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-workload-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:20.360277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.360194    4425 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-data\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:20.360277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.360208    4425 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-workload-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:20.360477 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.360447    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-credential-socket" pod "fd21ddd7-1c9b-4543-9c58-b1656798e9ef" (UID: "fd21ddd7-1c9b-4543-9c58-b1656798e9ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:56:20.360614 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.360521    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istiod-ca-cert" pod "fd21ddd7-1c9b-4543-9c58-b1656798e9ef" (UID: "fd21ddd7-1c9b-4543-9c58-b1656798e9ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:56:20.361787 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.361763    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-envoy" pod "fd21ddd7-1c9b-4543-9c58-b1656798e9ef" (UID: "fd21ddd7-1c9b-4543-9c58-b1656798e9ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:56:20.361908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.361839    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-token" pod "fd21ddd7-1c9b-4543-9c58-b1656798e9ef" (UID: "fd21ddd7-1c9b-4543-9c58-b1656798e9ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:56:20.362101 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.362084    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-podinfo" pod "fd21ddd7-1c9b-4543-9c58-b1656798e9ef" (UID: "fd21ddd7-1c9b-4543-9c58-b1656798e9ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:56:20.362178 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.362147    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-kube-api-access-sdx2f" pod "fd21ddd7-1c9b-4543-9c58-b1656798e9ef" (UID: "fd21ddd7-1c9b-4543-9c58-b1656798e9ef"). InnerVolumeSpecName "kube-api-access-sdx2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:56:20.377421 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.377397    4425 generic.go:356] "Generic (PLEG): container finished" podID="fd21ddd7-1c9b-4543-9c58-b1656798e9ef" containerID="9b23b87592b851433ec0016cbe7d9fb8e24e5f9c3a337482328ae8b30691a438" exitCode=0
Oct 30 01:56:20.377514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.377440    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg" event={"ID":"fd21ddd7-1c9b-4543-9c58-b1656798e9ef","Type":"ContainerDied","Data":"9b23b87592b851433ec0016cbe7d9fb8e24e5f9c3a337482328ae8b30691a438"}
Oct 30 01:56:20.377514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.377446    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"
Oct 30 01:56:20.377514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.377483    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg" event={"ID":"fd21ddd7-1c9b-4543-9c58-b1656798e9ef","Type":"ContainerDied","Data":"86a8b925e567e8a9cafdeb6f64f2fdec7cc204af28420dc73f75c314634800a3"}
Oct 30 01:56:20.377514 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.377505    4425 scope.go:122] "RemoveContainer" containerID="9b23b87592b851433ec0016cbe7d9fb8e24e5f9c3a337482328ae8b30691a438"
Oct 30 01:56:20.387337 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.387317    4425 scope.go:122] "RemoveContainer" containerID="9b23b87592b851433ec0016cbe7d9fb8e24e5f9c3a337482328ae8b30691a438"
Oct 30 01:56:20.387616 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:56:20.387590    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b23b87592b851433ec0016cbe7d9fb8e24e5f9c3a337482328ae8b30691a438\": container with ID starting with 9b23b87592b851433ec0016cbe7d9fb8e24e5f9c3a337482328ae8b30691a438 not found: ID does not exist" containerID="9b23b87592b851433ec0016cbe7d9fb8e24e5f9c3a337482328ae8b30691a438"
Oct 30 01:56:20.387667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.387626    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b23b87592b851433ec0016cbe7d9fb8e24e5f9c3a337482328ae8b30691a438"} err="failed to get container status \"9b23b87592b851433ec0016cbe7d9fb8e24e5f9c3a337482328ae8b30691a438\": rpc error: code = NotFound desc = could not find container \"9b23b87592b851433ec0016cbe7d9fb8e24e5f9c3a337482328ae8b30691a438\": container with ID starting with 9b23b87592b851433ec0016cbe7d9fb8e24e5f9c3a337482328ae8b30691a438 not found: ID does not exist"
Oct 30 01:56:20.423405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.423380    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"]
Oct 30 01:56:20.427307 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.427286    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9bfe36-openshift-default-7c444f5fc5-jhtwg"]
Oct 30 01:56:20.461699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.461668    4425 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-token\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:20.461699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.461689    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdx2f\" (UniqueName: \"kubernetes.io/projected/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-kube-api-access-sdx2f\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:20.461699 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.461699    4425 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-envoy\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:20.461952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.461708    4425 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istio-podinfo\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:20.461952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.461717    4425 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-credential-socket\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:20.461952 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:20.461725    4425 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd21ddd7-1c9b-4543-9c58-b1656798e9ef-istiod-ca-cert\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:21.421277 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:21.421245    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd21ddd7-1c9b-4543-9c58-b1656798e9ef" path="/var/lib/kubelet/pods/fd21ddd7-1c9b-4543-9c58-b1656798e9ef/volumes"
Oct 30 01:56:21.623443 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:21.623404    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:56:25.133419 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.133386    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"]
Oct 30 01:56:25.133891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.133777    4425 cpu_manager.go:414] "containerMap: removing container" podUID="8e763d6f-32ac-4739-9e2d-feca894a8987" containerName="payload-pre-processing"
Oct 30 01:56:25.133891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.133788    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e763d6f-32ac-4739-9e2d-feca894a8987" containerName="payload-pre-processing"
Oct 30 01:56:25.133891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.133804    4425 cpu_manager.go:414] "containerMap: removing container" podUID="be776653-8806-4154-96d5-87ce0af15d97" containerName="payload-processing"
Oct 30 01:56:25.133891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.133809    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be776653-8806-4154-96d5-87ce0af15d97" containerName="payload-processing"
Oct 30 01:56:25.133891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.133825    4425 cpu_manager.go:414] "containerMap: removing container" podUID="fd21ddd7-1c9b-4543-9c58-b1656798e9ef" containerName="istio-proxy"
Oct 30 01:56:25.133891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.133830    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd21ddd7-1c9b-4543-9c58-b1656798e9ef" containerName="istio-proxy"
Oct 30 01:56:25.133891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.133841    4425 cpu_manager.go:414] "containerMap: removing container" podUID="82735c67-cda8-490f-a0a9-42db3b56992a" containerName="revoke-keys"
Oct 30 01:56:25.133891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.133846    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82735c67-cda8-490f-a0a9-42db3b56992a" containerName="revoke-keys"
Oct 30 01:56:25.133891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.133861    4425 cpu_manager.go:414] "containerMap: removing container" podUID="998232e0-4d7b-4a4b-9fd1-5eb005157b11" containerName="istio-proxy"
Oct 30 01:56:25.133891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.133866    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998232e0-4d7b-4a4b-9fd1-5eb005157b11" containerName="istio-proxy"
Oct 30 01:56:25.134237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.133924    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e763d6f-32ac-4739-9e2d-feca894a8987" containerName="payload-pre-processing"
Oct 30 01:56:25.134237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.133934    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="be776653-8806-4154-96d5-87ce0af15d97" containerName="payload-processing"
Oct 30 01:56:25.134237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.133942    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd21ddd7-1c9b-4543-9c58-b1656798e9ef" containerName="istio-proxy"
Oct 30 01:56:25.134237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.133949    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="82735c67-cda8-490f-a0a9-42db3b56992a" containerName="revoke-keys"
Oct 30 01:56:25.134237 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.133956    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="998232e0-4d7b-4a4b-9fd1-5eb005157b11" containerName="istio-proxy"
Oct 30 01:56:25.138865 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.138845    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"
Oct 30 01:56:25.151056 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.151034    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:56:25.160770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.160747    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:56:25.162511 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.162490    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:56:25.162755 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.162734    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:56:25.163695 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.163673    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"]
Oct 30 01:56:25.308688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.308654    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zjmp\" (UniqueName: \"kubernetes.io/projected/09ba8237-56f8-42eb-bb9e-c5fb383995b3-kube-api-access-6zjmp\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv\" (UID: \"09ba8237-56f8-42eb-bb9e-c5fb383995b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"
Oct 30 01:56:25.308688 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.308692    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ba8237-56f8-42eb-bb9e-c5fb383995b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv\" (UID: \"09ba8237-56f8-42eb-bb9e-c5fb383995b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"
Oct 30 01:56:25.308951 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.308780    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ba8237-56f8-42eb-bb9e-c5fb383995b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv\" (UID: \"09ba8237-56f8-42eb-bb9e-c5fb383995b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"
Oct 30 01:56:25.409395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.409321    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zjmp\" (UniqueName: \"kubernetes.io/projected/09ba8237-56f8-42eb-bb9e-c5fb383995b3-kube-api-access-6zjmp\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv\" (UID: \"09ba8237-56f8-42eb-bb9e-c5fb383995b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"
Oct 30 01:56:25.409395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.409362    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ba8237-56f8-42eb-bb9e-c5fb383995b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv\" (UID: \"09ba8237-56f8-42eb-bb9e-c5fb383995b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"
Oct 30 01:56:25.409395 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.409395    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ba8237-56f8-42eb-bb9e-c5fb383995b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv\" (UID: \"09ba8237-56f8-42eb-bb9e-c5fb383995b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"
Oct 30 01:56:25.410135 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.410110    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ba8237-56f8-42eb-bb9e-c5fb383995b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv\" (UID: \"09ba8237-56f8-42eb-bb9e-c5fb383995b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"
Oct 30 01:56:25.411603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.411581    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ba8237-56f8-42eb-bb9e-c5fb383995b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv\" (UID: \"09ba8237-56f8-42eb-bb9e-c5fb383995b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"
Oct 30 01:56:25.430299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.430266    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zjmp\" (UniqueName: \"kubernetes.io/projected/09ba8237-56f8-42eb-bb9e-c5fb383995b3-kube-api-access-6zjmp\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv\" (UID: \"09ba8237-56f8-42eb-bb9e-c5fb383995b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"
Oct 30 01:56:25.452050 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.452022    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"
Oct 30 01:56:25.751207 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:25.751178    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"]
Oct 30 01:56:25.753125 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:56:25.753097    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ba8237_56f8_42eb_bb9e_c5fb383995b3.slice/crio-7401496b7e617dace56a1de6d6154ea8ffa2c72f211974ea594586c0601a598f WatchSource:0}: Error finding container 7401496b7e617dace56a1de6d6154ea8ffa2c72f211974ea594586c0601a598f: Status 404 returned error can't find the container with id 7401496b7e617dace56a1de6d6154ea8ffa2c72f211974ea594586c0601a598f
Oct 30 01:56:26.419090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:26.419007    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv" event={"ID":"09ba8237-56f8-42eb-bb9e-c5fb383995b3","Type":"ContainerStarted","Data":"d8802ad077a26375a5fc66afc33b3257add6ca271a7f8e4c525edd648953e237"}
Oct 30 01:56:26.419090 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:26.419074    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv" event={"ID":"09ba8237-56f8-42eb-bb9e-c5fb383995b3","Type":"ContainerStarted","Data":"7401496b7e617dace56a1de6d6154ea8ffa2c72f211974ea594586c0601a598f"}
Oct 30 01:56:26.466177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:26.466115    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv" podStartSLOduration=1.466099656 podStartE2EDuration="1.466099656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:26.46350629 +0000 UTC m=+4495.213161600" watchObservedRunningTime="2026-10-30 01:56:26.466099656 +0000 UTC m=+4495.215754964"
Oct 30 01:56:27.428470 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:27.428434    4425 generic.go:356] "Generic (PLEG): container finished" podID="09ba8237-56f8-42eb-bb9e-c5fb383995b3" containerID="d8802ad077a26375a5fc66afc33b3257add6ca271a7f8e4c525edd648953e237" exitCode=0
Oct 30 01:56:27.428882 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:27.428518    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv" event={"ID":"09ba8237-56f8-42eb-bb9e-c5fb383995b3","Type":"ContainerDied","Data":"d8802ad077a26375a5fc66afc33b3257add6ca271a7f8e4c525edd648953e237"}
Oct 30 01:56:28.030201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:28.030148    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"]
Oct 30 01:56:28.039647 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:28.039619    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb235c4-gdbdz"]
Oct 30 01:56:28.558021 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:28.557998    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"
Oct 30 01:56:28.633742 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:28.633707    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ba8237-56f8-42eb-bb9e-c5fb383995b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ba8237-56f8-42eb-bb9e-c5fb383995b3-maas-api-xks-ca\") pod \"09ba8237-56f8-42eb-bb9e-c5fb383995b3\" (UID: \"09ba8237-56f8-42eb-bb9e-c5fb383995b3\") "
Oct 30 01:56:28.633891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:28.633763    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ba8237-56f8-42eb-bb9e-c5fb383995b3-kube-api-access-6zjmp\" (UniqueName: \"kubernetes.io/projected/09ba8237-56f8-42eb-bb9e-c5fb383995b3-kube-api-access-6zjmp\") pod \"09ba8237-56f8-42eb-bb9e-c5fb383995b3\" (UID: \"09ba8237-56f8-42eb-bb9e-c5fb383995b3\") "
Oct 30 01:56:28.633891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:28.633799    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ba8237-56f8-42eb-bb9e-c5fb383995b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ba8237-56f8-42eb-bb9e-c5fb383995b3-maas-api-openshift-ca\") pod \"09ba8237-56f8-42eb-bb9e-c5fb383995b3\" (UID: \"09ba8237-56f8-42eb-bb9e-c5fb383995b3\") "
Oct 30 01:56:28.634222 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:28.634200    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ba8237-56f8-42eb-bb9e-c5fb383995b3-maas-api-openshift-ca" pod "09ba8237-56f8-42eb-bb9e-c5fb383995b3" (UID: "09ba8237-56f8-42eb-bb9e-c5fb383995b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:56:28.635758 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:28.635736    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ba8237-56f8-42eb-bb9e-c5fb383995b3-kube-api-access-6zjmp" pod "09ba8237-56f8-42eb-bb9e-c5fb383995b3" (UID: "09ba8237-56f8-42eb-bb9e-c5fb383995b3"). InnerVolumeSpecName "kube-api-access-6zjmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:56:28.635847 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:28.635821    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ba8237-56f8-42eb-bb9e-c5fb383995b3-maas-api-xks-ca" pod "09ba8237-56f8-42eb-bb9e-c5fb383995b3" (UID: "09ba8237-56f8-42eb-bb9e-c5fb383995b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:56:28.735806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:28.735743    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09ba8237-56f8-42eb-bb9e-c5fb383995b3-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:28.735806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:28.735786    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zjmp\" (UniqueName: \"kubernetes.io/projected/09ba8237-56f8-42eb-bb9e-c5fb383995b3-kube-api-access-6zjmp\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:28.735806 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:28.735802    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09ba8237-56f8-42eb-bb9e-c5fb383995b3-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:29.420572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:29.420540    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="077fb11a-0be0-4b97-ad7b-c630ba538274" path="/var/lib/kubelet/pods/077fb11a-0be0-4b97-ad7b-c630ba538274/volumes"
Oct 30 01:56:29.442260 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:29.442239    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"
Oct 30 01:56:29.442381 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:29.442278    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv" event={"ID":"09ba8237-56f8-42eb-bb9e-c5fb383995b3","Type":"ContainerDied","Data":"7401496b7e617dace56a1de6d6154ea8ffa2c72f211974ea594586c0601a598f"}
Oct 30 01:56:29.442381 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:29.442319    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7401496b7e617dace56a1de6d6154ea8ffa2c72f211974ea594586c0601a598f"
Oct 30 01:56:30.179553 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:30.179515    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv" podUID="a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:56:30.179970 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:30.179762    4425 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr" podUID="534f39ab-7cfa-43b1-bd72-ddfa1e8f702d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:56:33.732725 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:33.732668    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"
Oct 30 01:56:33.755382 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:33.755337    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"
Oct 30 01:56:33.782956 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:33.782933    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1-plugins-config-volume\") pod \"a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1\" (UID: \"a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1\") "
Oct 30 01:56:33.783095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:33.782971    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/534f39ab-7cfa-43b1-bd72-ddfa1e8f702d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/534f39ab-7cfa-43b1-bd72-ddfa1e8f702d-plugins-config-volume\") pod \"534f39ab-7cfa-43b1-bd72-ddfa1e8f702d\" (UID: \"534f39ab-7cfa-43b1-bd72-ddfa1e8f702d\") "
Oct 30 01:56:33.783095 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:33.783076    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1-kube-api-access-kswdr\" (UniqueName: \"kubernetes.io/projected/a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1-kube-api-access-kswdr\") pod \"a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1\" (UID: \"a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1\") "
Oct 30 01:56:33.783264 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:33.783096    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/534f39ab-7cfa-43b1-bd72-ddfa1e8f702d-kube-api-access-s67jh\" (UniqueName: \"kubernetes.io/projected/534f39ab-7cfa-43b1-bd72-ddfa1e8f702d-kube-api-access-s67jh\") pod \"534f39ab-7cfa-43b1-bd72-ddfa1e8f702d\" (UID: \"534f39ab-7cfa-43b1-bd72-ddfa1e8f702d\") "
Oct 30 01:56:33.783423 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:33.783400    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1-plugins-config-volume" pod "a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1" (UID: "a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:56:33.783515 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:33.783419    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/534f39ab-7cfa-43b1-bd72-ddfa1e8f702d-plugins-config-volume" pod "534f39ab-7cfa-43b1-bd72-ddfa1e8f702d" (UID: "534f39ab-7cfa-43b1-bd72-ddfa1e8f702d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:56:33.785107 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:33.785082    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534f39ab-7cfa-43b1-bd72-ddfa1e8f702d-kube-api-access-s67jh" pod "534f39ab-7cfa-43b1-bd72-ddfa1e8f702d" (UID: "534f39ab-7cfa-43b1-bd72-ddfa1e8f702d"). InnerVolumeSpecName "kube-api-access-s67jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:56:33.785231 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:33.785181    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1-kube-api-access-kswdr" pod "a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1" (UID: "a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1"). InnerVolumeSpecName "kube-api-access-kswdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:56:33.884535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:33.884489    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kswdr\" (UniqueName: \"kubernetes.io/projected/a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1-kube-api-access-kswdr\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:33.884535 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:33.884534    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s67jh\" (UniqueName: \"kubernetes.io/projected/534f39ab-7cfa-43b1-bd72-ddfa1e8f702d-kube-api-access-s67jh\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:33.884794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:33.884548    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:33.884794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:33.884562    4425 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/534f39ab-7cfa-43b1-bd72-ddfa1e8f702d-plugins-config-volume\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:34.477334 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.477295    4425 generic.go:356] "Generic (PLEG): container finished" podID="534f39ab-7cfa-43b1-bd72-ddfa1e8f702d" containerID="46d0797f7b9535068c592eacfeae58c1f50af2ea61b90ed59205ca49965b459a" exitCode=0
Oct 30 01:56:34.477530 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.477351    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"
Oct 30 01:56:34.477530 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.477378    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr" event={"ID":"534f39ab-7cfa-43b1-bd72-ddfa1e8f702d","Type":"ContainerDied","Data":"46d0797f7b9535068c592eacfeae58c1f50af2ea61b90ed59205ca49965b459a"}
Oct 30 01:56:34.477530 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.477430    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr" event={"ID":"534f39ab-7cfa-43b1-bd72-ddfa1e8f702d","Type":"ContainerDied","Data":"0c8823fe78ba3f0f7428f836624195a3f952a516dc8c84744c07794545ef9640"}
Oct 30 01:56:34.477530 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.477451    4425 scope.go:122] "RemoveContainer" containerID="46d0797f7b9535068c592eacfeae58c1f50af2ea61b90ed59205ca49965b459a"
Oct 30 01:56:34.479044 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.479021    4425 generic.go:356] "Generic (PLEG): container finished" podID="a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1" containerID="6404b1bb1d27408c32b30d3f43a9bdfe9bb9457f2e711f1a8a4372fd18ead5af" exitCode=0
Oct 30 01:56:34.479238 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.479070    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"
Oct 30 01:56:34.479238 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.479107    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv" event={"ID":"a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1","Type":"ContainerDied","Data":"6404b1bb1d27408c32b30d3f43a9bdfe9bb9457f2e711f1a8a4372fd18ead5af"}
Oct 30 01:56:34.479238 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.479181    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv" event={"ID":"a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1","Type":"ContainerDied","Data":"e2e7cba9c3b9a4080de96a963d0a908cd31a122d16109da5c4cecd661cbda3a2"}
Oct 30 01:56:34.487781 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.487767    4425 scope.go:122] "RemoveContainer" containerID="46d0797f7b9535068c592eacfeae58c1f50af2ea61b90ed59205ca49965b459a"
Oct 30 01:56:34.488084 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:56:34.488061    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d0797f7b9535068c592eacfeae58c1f50af2ea61b90ed59205ca49965b459a\": container with ID starting with 46d0797f7b9535068c592eacfeae58c1f50af2ea61b90ed59205ca49965b459a not found: ID does not exist" containerID="46d0797f7b9535068c592eacfeae58c1f50af2ea61b90ed59205ca49965b459a"
Oct 30 01:56:34.488211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.488087    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d0797f7b9535068c592eacfeae58c1f50af2ea61b90ed59205ca49965b459a"} err="failed to get container status \"46d0797f7b9535068c592eacfeae58c1f50af2ea61b90ed59205ca49965b459a\": rpc error: code = NotFound desc = could not find container \"46d0797f7b9535068c592eacfeae58c1f50af2ea61b90ed59205ca49965b459a\": container with ID starting with 46d0797f7b9535068c592eacfeae58c1f50af2ea61b90ed59205ca49965b459a not found: ID does not exist"
Oct 30 01:56:34.488211 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.488104    4425 scope.go:122] "RemoveContainer" containerID="6404b1bb1d27408c32b30d3f43a9bdfe9bb9457f2e711f1a8a4372fd18ead5af"
Oct 30 01:56:34.498618 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.498598    4425 scope.go:122] "RemoveContainer" containerID="6404b1bb1d27408c32b30d3f43a9bdfe9bb9457f2e711f1a8a4372fd18ead5af"
Oct 30 01:56:34.498879 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:56:34.498858    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6404b1bb1d27408c32b30d3f43a9bdfe9bb9457f2e711f1a8a4372fd18ead5af\": container with ID starting with 6404b1bb1d27408c32b30d3f43a9bdfe9bb9457f2e711f1a8a4372fd18ead5af not found: ID does not exist" containerID="6404b1bb1d27408c32b30d3f43a9bdfe9bb9457f2e711f1a8a4372fd18ead5af"
Oct 30 01:56:34.498937 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.498890    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6404b1bb1d27408c32b30d3f43a9bdfe9bb9457f2e711f1a8a4372fd18ead5af"} err="failed to get container status \"6404b1bb1d27408c32b30d3f43a9bdfe9bb9457f2e711f1a8a4372fd18ead5af\": rpc error: code = NotFound desc = could not find container \"6404b1bb1d27408c32b30d3f43a9bdfe9bb9457f2e711f1a8a4372fd18ead5af\": container with ID starting with 6404b1bb1d27408c32b30d3f43a9bdfe9bb9457f2e711f1a8a4372fd18ead5af not found: ID does not exist"
Oct 30 01:56:34.536541 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.536516    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"]
Oct 30 01:56:34.545780 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.545761    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1afb3e-75fcc5f87f-wltfv"]
Oct 30 01:56:34.570961 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.570939    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"]
Oct 30 01:56:34.574799 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:34.574779    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1afb3e-f98f576db-wsvhr"]
Oct 30 01:56:35.420449 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:35.420418    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="534f39ab-7cfa-43b1-bd72-ddfa1e8f702d" path="/var/lib/kubelet/pods/534f39ab-7cfa-43b1-bd72-ddfa1e8f702d/volumes"
Oct 30 01:56:35.420837 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:35.420784    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1" path="/var/lib/kubelet/pods/a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1/volumes"
Oct 30 01:56:36.961872 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:36.961842    4425 scope.go:122] "RemoveContainer" containerID="6a9ed38eda6ca3973f68f258a54d749ea1fa49ef400a3bf0d18b4cc3fdb7152d"
Oct 30 01:56:36.971245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:36.971225    4425 scope.go:122] "RemoveContainer" containerID="6942267e0e6779ee6fc33163673cae9d2d08648aa01d6133ef90b4aaa9b9cecd"
Oct 30 01:56:37.556358 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:56:37.556321    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:56:39.028012 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:39.027981    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"]
Oct 30 01:56:39.037640 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:39.037615    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84f1f1ae-ltgd5"]
Oct 30 01:56:39.420651 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:39.420580    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed14e2e8-91df-43b2-8437-10fb194339ff" path="/var/lib/kubelet/pods/ed14e2e8-91df-43b2-8437-10fb194339ff/volumes"
Oct 30 01:56:40.917874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:40.917835    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"]
Oct 30 01:56:40.927175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:40.927137    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-4107fc-8qwpv"]
Oct 30 01:56:41.422412 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:41.422379    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ba8237-56f8-42eb-bb9e-c5fb383995b3" path="/var/lib/kubelet/pods/09ba8237-56f8-42eb-bb9e-c5fb383995b3/volumes"
Oct 30 01:56:43.804710 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:43.804660    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:56:49.270279 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.270246    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"]
Oct 30 01:56:49.270747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.270714    4425 cpu_manager.go:414] "containerMap: removing container" podUID="09ba8237-56f8-42eb-bb9e-c5fb383995b3" containerName="revoke-keys"
Oct 30 01:56:49.270747 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.270730    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ba8237-56f8-42eb-bb9e-c5fb383995b3" containerName="revoke-keys"
Oct 30 01:56:49.270900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.270752    4425 cpu_manager.go:414] "containerMap: removing container" podUID="534f39ab-7cfa-43b1-bd72-ddfa1e8f702d" containerName="payload-processing"
Oct 30 01:56:49.270900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.270758    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534f39ab-7cfa-43b1-bd72-ddfa1e8f702d" containerName="payload-processing"
Oct 30 01:56:49.270900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.270773    4425 cpu_manager.go:414] "containerMap: removing container" podUID="a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1" containerName="payload-pre-processing"
Oct 30 01:56:49.270900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.270778    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1" containerName="payload-pre-processing"
Oct 30 01:56:49.270900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.270849    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ba8237-56f8-42eb-bb9e-c5fb383995b3" containerName="revoke-keys"
Oct 30 01:56:49.270900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.270857    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="534f39ab-7cfa-43b1-bd72-ddfa1e8f702d" containerName="payload-processing"
Oct 30 01:56:49.270900 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.270865    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3800dfa-f3bb-4c6d-ab7f-24cef32d43d1" containerName="payload-pre-processing"
Oct 30 01:56:49.273954 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.273938    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"
Oct 30 01:56:49.286661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.286641    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:56:49.286797 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.286777    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:56:49.297387 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.297362    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:56:49.297485 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.297458    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:56:49.299601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.299579    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"]
Oct 30 01:56:49.417680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.417655    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c10aa92-0b57-4631-8586-c3d34bc9899e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm\" (UID: \"7c10aa92-0b57-4631-8586-c3d34bc9899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"
Oct 30 01:56:49.417826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.417696    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c10aa92-0b57-4631-8586-c3d34bc9899e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm\" (UID: \"7c10aa92-0b57-4631-8586-c3d34bc9899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"
Oct 30 01:56:49.417826 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.417796    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8khb\" (UniqueName: \"kubernetes.io/projected/7c10aa92-0b57-4631-8586-c3d34bc9899e-kube-api-access-t8khb\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm\" (UID: \"7c10aa92-0b57-4631-8586-c3d34bc9899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"
Oct 30 01:56:49.518730 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.518695    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c10aa92-0b57-4631-8586-c3d34bc9899e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm\" (UID: \"7c10aa92-0b57-4631-8586-c3d34bc9899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"
Oct 30 01:56:49.518910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.518745    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c10aa92-0b57-4631-8586-c3d34bc9899e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm\" (UID: \"7c10aa92-0b57-4631-8586-c3d34bc9899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"
Oct 30 01:56:49.518910 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.518799    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8khb\" (UniqueName: \"kubernetes.io/projected/7c10aa92-0b57-4631-8586-c3d34bc9899e-kube-api-access-t8khb\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm\" (UID: \"7c10aa92-0b57-4631-8586-c3d34bc9899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"
Oct 30 01:56:49.519450 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.519425    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c10aa92-0b57-4631-8586-c3d34bc9899e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm\" (UID: \"7c10aa92-0b57-4631-8586-c3d34bc9899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"
Oct 30 01:56:49.521115 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.521061    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c10aa92-0b57-4631-8586-c3d34bc9899e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm\" (UID: \"7c10aa92-0b57-4631-8586-c3d34bc9899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"
Oct 30 01:56:49.539420 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.539397    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8khb\" (UniqueName: \"kubernetes.io/projected/7c10aa92-0b57-4631-8586-c3d34bc9899e-kube-api-access-t8khb\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm\" (UID: \"7c10aa92-0b57-4631-8586-c3d34bc9899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"
Oct 30 01:56:49.586807 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.586785    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"
Oct 30 01:56:49.883567 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:49.883540    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"]
Oct 30 01:56:49.885535 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:56:49.885506    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c10aa92_0b57_4631_8586_c3d34bc9899e.slice/crio-06e1cf95099653450a7d9a6183c81c38c8c1e29cf37dd7bc1d0f621771c70d1d WatchSource:0}: Error finding container 06e1cf95099653450a7d9a6183c81c38c8c1e29cf37dd7bc1d0f621771c70d1d: Status 404 returned error can't find the container with id 06e1cf95099653450a7d9a6183c81c38c8c1e29cf37dd7bc1d0f621771c70d1d
Oct 30 01:56:50.578978 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:50.578940    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm" event={"ID":"7c10aa92-0b57-4631-8586-c3d34bc9899e","Type":"ContainerStarted","Data":"af7decd6ae231645b14ba434b943205525dd21e57ecb6f069b7aa6fca7dd1225"}
Oct 30 01:56:50.579451 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:50.578996    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm" event={"ID":"7c10aa92-0b57-4631-8586-c3d34bc9899e","Type":"ContainerStarted","Data":"06e1cf95099653450a7d9a6183c81c38c8c1e29cf37dd7bc1d0f621771c70d1d"}
Oct 30 01:56:50.617218 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:50.617144    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm" podStartSLOduration=1.617131028 podStartE2EDuration="1.617131028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:50.614294441 +0000 UTC m=+4519.363949748" watchObservedRunningTime="2026-10-30 01:56:50.617131028 +0000 UTC m=+4519.366786336"
Oct 30 01:56:51.586374 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:51.586343    4425 generic.go:356] "Generic (PLEG): container finished" podID="7c10aa92-0b57-4631-8586-c3d34bc9899e" containerID="af7decd6ae231645b14ba434b943205525dd21e57ecb6f069b7aa6fca7dd1225" exitCode=0
Oct 30 01:56:51.586760 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:51.586432    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm" event={"ID":"7c10aa92-0b57-4631-8586-c3d34bc9899e","Type":"ContainerDied","Data":"af7decd6ae231645b14ba434b943205525dd21e57ecb6f069b7aa6fca7dd1225"}
Oct 30 01:56:52.711254 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:52.711230    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"
Oct 30 01:56:52.849014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:52.848953    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c10aa92-0b57-4631-8586-c3d34bc9899e-kube-api-access-t8khb\" (UniqueName: \"kubernetes.io/projected/7c10aa92-0b57-4631-8586-c3d34bc9899e-kube-api-access-t8khb\") pod \"7c10aa92-0b57-4631-8586-c3d34bc9899e\" (UID: \"7c10aa92-0b57-4631-8586-c3d34bc9899e\") "
Oct 30 01:56:52.849014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:52.849003    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c10aa92-0b57-4631-8586-c3d34bc9899e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c10aa92-0b57-4631-8586-c3d34bc9899e-maas-api-xks-ca\") pod \"7c10aa92-0b57-4631-8586-c3d34bc9899e\" (UID: \"7c10aa92-0b57-4631-8586-c3d34bc9899e\") "
Oct 30 01:56:52.849242 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:52.849041    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c10aa92-0b57-4631-8586-c3d34bc9899e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c10aa92-0b57-4631-8586-c3d34bc9899e-maas-api-openshift-ca\") pod \"7c10aa92-0b57-4631-8586-c3d34bc9899e\" (UID: \"7c10aa92-0b57-4631-8586-c3d34bc9899e\") "
Oct 30 01:56:52.849474 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:52.849448    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c10aa92-0b57-4631-8586-c3d34bc9899e-maas-api-openshift-ca" pod "7c10aa92-0b57-4631-8586-c3d34bc9899e" (UID: "7c10aa92-0b57-4631-8586-c3d34bc9899e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:56:52.851052 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:52.851021    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c10aa92-0b57-4631-8586-c3d34bc9899e-maas-api-xks-ca" pod "7c10aa92-0b57-4631-8586-c3d34bc9899e" (UID: "7c10aa92-0b57-4631-8586-c3d34bc9899e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:56:52.851148 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:52.851058    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c10aa92-0b57-4631-8586-c3d34bc9899e-kube-api-access-t8khb" pod "7c10aa92-0b57-4631-8586-c3d34bc9899e" (UID: "7c10aa92-0b57-4631-8586-c3d34bc9899e"). InnerVolumeSpecName "kube-api-access-t8khb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:56:52.949979 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:52.949955    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c10aa92-0b57-4631-8586-c3d34bc9899e-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:52.949979 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:52.949978    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8khb\" (UniqueName: \"kubernetes.io/projected/7c10aa92-0b57-4631-8586-c3d34bc9899e-kube-api-access-t8khb\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:52.950121 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:52.949988    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c10aa92-0b57-4631-8586-c3d34bc9899e-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:56:53.601673 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:53.601644    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"
Oct 30 01:56:53.601673 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:53.601666    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm" event={"ID":"7c10aa92-0b57-4631-8586-c3d34bc9899e","Type":"ContainerDied","Data":"06e1cf95099653450a7d9a6183c81c38c8c1e29cf37dd7bc1d0f621771c70d1d"}
Oct 30 01:56:53.601950 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:56:53.601708    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06e1cf95099653450a7d9a6183c81c38c8c1e29cf37dd7bc1d0f621771c70d1d"
Oct 30 01:57:02.834816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:02.834781    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"]
Oct 30 01:57:02.843956 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:02.843930    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-bc127f-94vwm"]
Oct 30 01:57:03.420909 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:03.420877    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c10aa92-0b57-4631-8586-c3d34bc9899e" path="/var/lib/kubelet/pods/7c10aa92-0b57-4631-8586-c3d34bc9899e/volumes"
Oct 30 01:57:05.487350 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:05.487317    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:57:07.553081 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:57:07.553048    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:57:15.027384 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:15.027294    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"]
Oct 30 01:57:15.036922 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:15.036882    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56c5bfd2-j7xsz"]
Oct 30 01:57:15.331475 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:15.331386    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"]
Oct 30 01:57:15.421454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:15.421426    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6019518-5851-47d1-8c48-c4c697264d32" path="/var/lib/kubelet/pods/d6019518-5851-47d1-8c48-c4c697264d32/volumes"
Oct 30 01:57:18.026981 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:18.026944    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"]
Oct 30 01:57:18.036440 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:18.036414    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b05436d4-gmdz5"]
Oct 30 01:57:19.421174 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:19.421128    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba" path="/var/lib/kubelet/pods/eefe3b1e-3c62-47c6-a2e5-a6ca8ec197ba/volumes"
Oct 30 01:57:24.423630 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.423587    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr"]
Oct 30 01:57:24.424143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.424021    4425 cpu_manager.go:414] "containerMap: removing container" podUID="7c10aa92-0b57-4631-8586-c3d34bc9899e" containerName="revoke-keys"
Oct 30 01:57:24.424143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.424034    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c10aa92-0b57-4631-8586-c3d34bc9899e" containerName="revoke-keys"
Oct 30 01:57:24.424143 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.424115    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c10aa92-0b57-4631-8586-c3d34bc9899e" containerName="revoke-keys"
Oct 30 01:57:24.428405 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.428383    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr"
Oct 30 01:57:24.443028 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.443005    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 01:57:24.452489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.452461    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 01:57:24.455916 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.455891    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr"]
Oct 30 01:57:24.516367 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.516338    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 01:57:24.516602 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.516433    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cmpw6\""
Oct 30 01:57:24.529876 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.529842    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhh9s\" (UniqueName: \"kubernetes.io/projected/11e155ec-1801-47f5-869f-a9fb5a1c467d-kube-api-access-qhh9s\") pod \"maas-api-revoke-sub-fe7fdd7e-k8ctr\" (UID: \"11e155ec-1801-47f5-869f-a9fb5a1c467d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr"
Oct 30 01:57:24.529997 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.529885    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e155ec-1801-47f5-869f-a9fb5a1c467d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe7fdd7e-k8ctr\" (UID: \"11e155ec-1801-47f5-869f-a9fb5a1c467d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr"
Oct 30 01:57:24.530042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.530008    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e155ec-1801-47f5-869f-a9fb5a1c467d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe7fdd7e-k8ctr\" (UID: \"11e155ec-1801-47f5-869f-a9fb5a1c467d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr"
Oct 30 01:57:24.631291 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.631243    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e155ec-1801-47f5-869f-a9fb5a1c467d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe7fdd7e-k8ctr\" (UID: \"11e155ec-1801-47f5-869f-a9fb5a1c467d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr"
Oct 30 01:57:24.631489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.631329    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhh9s\" (UniqueName: \"kubernetes.io/projected/11e155ec-1801-47f5-869f-a9fb5a1c467d-kube-api-access-qhh9s\") pod \"maas-api-revoke-sub-fe7fdd7e-k8ctr\" (UID: \"11e155ec-1801-47f5-869f-a9fb5a1c467d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr"
Oct 30 01:57:24.631489 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.631353    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e155ec-1801-47f5-869f-a9fb5a1c467d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe7fdd7e-k8ctr\" (UID: \"11e155ec-1801-47f5-869f-a9fb5a1c467d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr"
Oct 30 01:57:24.631939 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.631916    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e155ec-1801-47f5-869f-a9fb5a1c467d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe7fdd7e-k8ctr\" (UID: \"11e155ec-1801-47f5-869f-a9fb5a1c467d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr"
Oct 30 01:57:24.633945 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.633922    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e155ec-1801-47f5-869f-a9fb5a1c467d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe7fdd7e-k8ctr\" (UID: \"11e155ec-1801-47f5-869f-a9fb5a1c467d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr"
Oct 30 01:57:24.657572 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.657537    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhh9s\" (UniqueName: \"kubernetes.io/projected/11e155ec-1801-47f5-869f-a9fb5a1c467d-kube-api-access-qhh9s\") pod \"maas-api-revoke-sub-fe7fdd7e-k8ctr\" (UID: \"11e155ec-1801-47f5-869f-a9fb5a1c467d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr"
Oct 30 01:57:24.742066 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:24.741968    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr"
Oct 30 01:57:25.035213 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:25.035147    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"]
Oct 30 01:57:25.049586 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:25.049506    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a6d91ec-kmfrp"]
Oct 30 01:57:25.052201 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:25.052175    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr"]
Oct 30 01:57:25.052350 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:57:25.052327    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e155ec_1801_47f5_869f_a9fb5a1c467d.slice/crio-48f3929fc48f325476a60a477a6b8c8bd173b638806a159b8a50521fa7407530 WatchSource:0}: Error finding container 48f3929fc48f325476a60a477a6b8c8bd173b638806a159b8a50521fa7407530: Status 404 returned error can't find the container with id 48f3929fc48f325476a60a477a6b8c8bd173b638806a159b8a50521fa7407530
Oct 30 01:57:25.420645 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:25.420616    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d693b69-f319-4f95-9844-1ce2e5df425d" path="/var/lib/kubelet/pods/5d693b69-f319-4f95-9844-1ce2e5df425d/volumes"
Oct 30 01:57:25.803821 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:25.803781    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr" event={"ID":"11e155ec-1801-47f5-869f-a9fb5a1c467d","Type":"ContainerStarted","Data":"2d1c7ba7c9251f6ee7b184bb9dba59ad398cfd595ebd1c2df6f5334087603832"}
Oct 30 01:57:25.804269 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:25.803842    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr" event={"ID":"11e155ec-1801-47f5-869f-a9fb5a1c467d","Type":"ContainerStarted","Data":"48f3929fc48f325476a60a477a6b8c8bd173b638806a159b8a50521fa7407530"}
Oct 30 01:57:25.842372 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:25.842318    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr" podStartSLOduration=1.842302137 podStartE2EDuration="1.842302137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:25.83993485 +0000 UTC m=+4554.589590159" watchObservedRunningTime="2026-10-30 01:57:25.842302137 +0000 UTC m=+4554.591957447"
Oct 30 01:57:26.811804 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:26.811769    4425 generic.go:356] "Generic (PLEG): container finished" podID="11e155ec-1801-47f5-869f-a9fb5a1c467d" containerID="2d1c7ba7c9251f6ee7b184bb9dba59ad398cfd595ebd1c2df6f5334087603832" exitCode=0
Oct 30 01:57:26.812252 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:26.811816    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr" event={"ID":"11e155ec-1801-47f5-869f-a9fb5a1c467d","Type":"ContainerDied","Data":"2d1c7ba7c9251f6ee7b184bb9dba59ad398cfd595ebd1c2df6f5334087603832"}
Oct 30 01:57:27.963963 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:27.963937    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr"
Oct 30 01:57:28.028850 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:28.028815    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"]
Oct 30 01:57:28.031675 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:28.031648    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb8a211-m57ld"]
Oct 30 01:57:28.060742 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:28.060711    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e155ec-1801-47f5-869f-a9fb5a1c467d-kube-api-access-qhh9s\" (UniqueName: \"kubernetes.io/projected/11e155ec-1801-47f5-869f-a9fb5a1c467d-kube-api-access-qhh9s\") pod \"11e155ec-1801-47f5-869f-a9fb5a1c467d\" (UID: \"11e155ec-1801-47f5-869f-a9fb5a1c467d\") "
Oct 30 01:57:28.060913 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:28.060755    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e155ec-1801-47f5-869f-a9fb5a1c467d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e155ec-1801-47f5-869f-a9fb5a1c467d-maas-api-openshift-ca\") pod \"11e155ec-1801-47f5-869f-a9fb5a1c467d\" (UID: \"11e155ec-1801-47f5-869f-a9fb5a1c467d\") "
Oct 30 01:57:28.060913 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:28.060847    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11e155ec-1801-47f5-869f-a9fb5a1c467d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e155ec-1801-47f5-869f-a9fb5a1c467d-maas-api-xks-ca\") pod \"11e155ec-1801-47f5-869f-a9fb5a1c467d\" (UID: \"11e155ec-1801-47f5-869f-a9fb5a1c467d\") "
Oct 30 01:57:28.061209 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:28.061171    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e155ec-1801-47f5-869f-a9fb5a1c467d-maas-api-openshift-ca" pod "11e155ec-1801-47f5-869f-a9fb5a1c467d" (UID: "11e155ec-1801-47f5-869f-a9fb5a1c467d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:57:28.062922 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:28.062897    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11e155ec-1801-47f5-869f-a9fb5a1c467d-maas-api-xks-ca" pod "11e155ec-1801-47f5-869f-a9fb5a1c467d" (UID: "11e155ec-1801-47f5-869f-a9fb5a1c467d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:57:28.063006 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:28.062991    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e155ec-1801-47f5-869f-a9fb5a1c467d-kube-api-access-qhh9s" pod "11e155ec-1801-47f5-869f-a9fb5a1c467d" (UID: "11e155ec-1801-47f5-869f-a9fb5a1c467d"). InnerVolumeSpecName "kube-api-access-qhh9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:57:28.161947 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:28.161912    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhh9s\" (UniqueName: \"kubernetes.io/projected/11e155ec-1801-47f5-869f-a9fb5a1c467d-kube-api-access-qhh9s\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:57:28.161947 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:28.161941    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11e155ec-1801-47f5-869f-a9fb5a1c467d-maas-api-openshift-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:57:28.161947 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:28.161951    4425 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11e155ec-1801-47f5-869f-a9fb5a1c467d-maas-api-xks-ca\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:57:28.824857 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:28.824822    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr"
Oct 30 01:57:28.825034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:28.824824    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe7fdd7e-k8ctr" event={"ID":"11e155ec-1801-47f5-869f-a9fb5a1c467d","Type":"ContainerDied","Data":"48f3929fc48f325476a60a477a6b8c8bd173b638806a159b8a50521fa7407530"}
Oct 30 01:57:28.825034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:28.824949    4425 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48f3929fc48f325476a60a477a6b8c8bd173b638806a159b8a50521fa7407530"
Oct 30 01:57:29.421428 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:29.421391    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa" path="/var/lib/kubelet/pods/6f2da52c-5c3c-40c0-a3a1-b2a68c1501fa/volumes"
Oct 30 01:57:30.351180 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.351116    4425 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w" podUID="66a19982-1f16-4a03-a629-bf9b7d6b50d3" containerName="main" containerID="cri-o://45b78d12e45113ece8bb9b46515e5c2aee4dca04a368c534d048dd9b81219e45" gracePeriod=45
Oct 30 01:57:30.600100 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.600075    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 01:57:30.685177 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.685120    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a19982-1f16-4a03-a629-bf9b7d6b50d3-kube-api-access-npcqd\" (UniqueName: \"kubernetes.io/projected/66a19982-1f16-4a03-a629-bf9b7d6b50d3-kube-api-access-npcqd\") pod \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") "
Oct 30 01:57:30.685375 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.685201    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-dshm\") pod \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") "
Oct 30 01:57:30.685375 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.685281    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66a19982-1f16-4a03-a629-bf9b7d6b50d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/66a19982-1f16-4a03-a629-bf9b7d6b50d3-tls-certs\") pod \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") "
Oct 30 01:57:30.685375 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.685308    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-model-cache\") pod \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") "
Oct 30 01:57:30.685375 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.685351    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-tmp-dir\") pod \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") "
Oct 30 01:57:30.685603 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.685375    4425 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-home\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-home\") pod \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\" (UID: \"66a19982-1f16-4a03-a629-bf9b7d6b50d3\") "
Oct 30 01:57:30.685725 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.685666    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-model-cache" pod "66a19982-1f16-4a03-a629-bf9b7d6b50d3" (UID: "66a19982-1f16-4a03-a629-bf9b7d6b50d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:30.685833 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.685743    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-tmp-dir" pod "66a19982-1f16-4a03-a629-bf9b7d6b50d3" (UID: "66a19982-1f16-4a03-a629-bf9b7d6b50d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:30.685929 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.685907    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-home" pod "66a19982-1f16-4a03-a629-bf9b7d6b50d3" (UID: "66a19982-1f16-4a03-a629-bf9b7d6b50d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:30.687594 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.687572    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a19982-1f16-4a03-a629-bf9b7d6b50d3-kube-api-access-npcqd" pod "66a19982-1f16-4a03-a629-bf9b7d6b50d3" (UID: "66a19982-1f16-4a03-a629-bf9b7d6b50d3"). InnerVolumeSpecName "kube-api-access-npcqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:57:30.687818 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.687793    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-dshm" pod "66a19982-1f16-4a03-a629-bf9b7d6b50d3" (UID: "66a19982-1f16-4a03-a629-bf9b7d6b50d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:57:30.687953 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.687866    4425 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66a19982-1f16-4a03-a629-bf9b7d6b50d3-tls-certs" pod "66a19982-1f16-4a03-a629-bf9b7d6b50d3" (UID: "66a19982-1f16-4a03-a629-bf9b7d6b50d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:57:30.787034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.787002    4425 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npcqd\" (UniqueName: \"kubernetes.io/projected/66a19982-1f16-4a03-a629-bf9b7d6b50d3-kube-api-access-npcqd\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:57:30.787034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.787028    4425 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-dshm\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:57:30.787034 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.787038    4425 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66a19982-1f16-4a03-a629-bf9b7d6b50d3-tls-certs\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:57:30.787289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.787046    4425 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-model-cache\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:57:30.787289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.787055    4425 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-tmp-dir\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:57:30.787289 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.787064    4425 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66a19982-1f16-4a03-a629-bf9b7d6b50d3-home\") on node \"ip-10-0-133-32.ec2.internal\" DevicePath \"\""
Oct 30 01:57:30.838856 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.838822    4425 generic.go:356] "Generic (PLEG): container finished" podID="66a19982-1f16-4a03-a629-bf9b7d6b50d3" containerID="45b78d12e45113ece8bb9b46515e5c2aee4dca04a368c534d048dd9b81219e45" exitCode=0
Oct 30 01:57:30.839014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.838873    4425 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"
Oct 30 01:57:30.839014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.838895    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w" event={"ID":"66a19982-1f16-4a03-a629-bf9b7d6b50d3","Type":"ContainerDied","Data":"45b78d12e45113ece8bb9b46515e5c2aee4dca04a368c534d048dd9b81219e45"}
Oct 30 01:57:30.839014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.838925    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w" event={"ID":"66a19982-1f16-4a03-a629-bf9b7d6b50d3","Type":"ContainerDied","Data":"ac7be6d725b3206e2181ee61ac791ac1ce867954295b4ba5fbc0ab833adb2869"}
Oct 30 01:57:30.839014 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.838943    4425 scope.go:122] "RemoveContainer" containerID="45b78d12e45113ece8bb9b46515e5c2aee4dca04a368c534d048dd9b81219e45"
Oct 30 01:57:30.855183 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.855139    4425 scope.go:122] "RemoveContainer" containerID="45b78d12e45113ece8bb9b46515e5c2aee4dca04a368c534d048dd9b81219e45"
Oct 30 01:57:30.855478 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:57:30.855456    4425 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b78d12e45113ece8bb9b46515e5c2aee4dca04a368c534d048dd9b81219e45\": container with ID starting with 45b78d12e45113ece8bb9b46515e5c2aee4dca04a368c534d048dd9b81219e45 not found: ID does not exist" containerID="45b78d12e45113ece8bb9b46515e5c2aee4dca04a368c534d048dd9b81219e45"
Oct 30 01:57:30.855536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.855486    4425 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b78d12e45113ece8bb9b46515e5c2aee4dca04a368c534d048dd9b81219e45"} err="failed to get container status \"45b78d12e45113ece8bb9b46515e5c2aee4dca04a368c534d048dd9b81219e45\": rpc error: code = NotFound desc = could not find container \"45b78d12e45113ece8bb9b46515e5c2aee4dca04a368c534d048dd9b81219e45\": container with ID starting with 45b78d12e45113ece8bb9b46515e5c2aee4dca04a368c534d048dd9b81219e45 not found: ID does not exist"
Oct 30 01:57:30.884175 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.884121    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"]
Oct 30 01:57:30.887454 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:30.887428    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d5b2b8/premium-w4-d5b2b8-kserve-66f8d79d5c-zp89w"]
Oct 30 01:57:31.421000 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:31.420964    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66a19982-1f16-4a03-a629-bf9b7d6b50d3" path="/var/lib/kubelet/pods/66a19982-1f16-4a03-a629-bf9b7d6b50d3/volumes"
Oct 30 01:57:37.051583 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:37.051548    4425 scope.go:122] "RemoveContainer" containerID="ce8ca8abcf5212a3b61466152ea83c6a887e25f9dc203121064a1ed8f18522e4"
Oct 30 01:57:37.061145 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:37.061122    4425 scope.go:122] "RemoveContainer" containerID="728a8ef813fce21f24670bb98806507e45ab6561619d42de81d9a83442d005d7"
Oct 30 01:57:37.070042 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:37.070024    4425 scope.go:122] "RemoveContainer" containerID="a3d91893cc0ae57e8722d9c8e086719ff1a3f7bcba361270e50e227818b1fb06"
Oct 30 01:57:37.078733 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:37.078712    4425 scope.go:122] "RemoveContainer" containerID="5df0f7d6c16bc8e5d5fe550bf099e83535d885028a545ffbad2b95a507311ca4"
Oct 30 01:57:37.516596 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:37.516559    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:57:37.553778 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:57:37.553742    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:57:39.068198 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:39.068142    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"]
Oct 30 01:57:39.077804 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:39.077777    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9db93ee8-88lmk"]
Oct 30 01:57:39.421407 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:39.421330    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e622d47c-b9a0-4307-863f-84241f4bc9c8" path="/var/lib/kubelet/pods/e622d47c-b9a0-4307-863f-84241f4bc9c8/volumes"
Oct 30 01:57:48.790874 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:48.790795    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gkbtv"]
Oct 30 01:57:49.026279 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:49.026244    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-dbkj2_95a0d310-2023-40af-bf5e-d3e30e739590/kuadrant-console-plugin/0.log"
Oct 30 01:57:49.028075 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:49.028047    4425 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"]
Oct 30 01:57:49.037536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:49.037512    4425 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19f51ebe-fp2vg"]
Oct 30 01:57:49.420749 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:49.420721    4425 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6164abfa-2fd7-4e60-946d-19efeb886751" path="/var/lib/kubelet/pods/6164abfa-2fd7-4e60-946d-19efeb886751/volumes"
Oct 30 01:57:49.565935 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:49.565909    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gkbtv_7377e3da-0408-4a67-bfe1-f4e481cb09a8/limitador/0.log"
Oct 30 01:57:50.785891 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:50.785857    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9zbsm_3009c800-6df6-4c1e-935f-3a7e5be62572/manager/0.log"
Oct 30 01:57:51.192655 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:51.192577    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-n47qc_80392122-eba8-4040-a019-e44488201532/manager/0.log"
Oct 30 01:57:51.662680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:51.662646    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6496db66cc-nzwpd_858deb1c-4c93-42c2-9fc9-8a53b27fa19b/manager/0.log"
Oct 30 01:57:51.852071 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:51.852045    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5mcgv_b42c9fca-e232-4900-adca-ddd04928e24c/server/0.log"
Oct 30 01:57:52.033544 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:57:52.033519    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9xl4p_33acc2a2-ca7f-4702-be45-c4f12d99d4b7/manager/0.log"
Oct 30 01:58:03.297151 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:03.297118    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9zbsm_3009c800-6df6-4c1e-935f-3a7e5be62572/manager/0.log"
Oct 30 01:58:03.689641 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:03.689562    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-n47qc_80392122-eba8-4040-a019-e44488201532/manager/0.log"
Oct 30 01:58:04.199664 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:04.199631    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6496db66cc-nzwpd_858deb1c-4c93-42c2-9fc9-8a53b27fa19b/manager/0.log"
Oct 30 01:58:04.385794 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:04.385762    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5mcgv_b42c9fca-e232-4900-adca-ddd04928e24c/server/0.log"
Oct 30 01:58:04.576441 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:04.576402    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9xl4p_33acc2a2-ca7f-4702-be45-c4f12d99d4b7/manager/0.log"
Oct 30 01:58:05.725789 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:05.725754    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-03fd1035-9wzjk_d672322e-d956-4654-9e06-490aa9fb1518/revoke-keys/0.log"
Oct 30 01:58:05.906376 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:05.906347    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0684d5d8-hfw4s_3f615383-3fb8-4e8e-8681-465e88538678/revoke-keys/0.log"
Oct 30 01:58:06.091810 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:06.091782    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-116ece75-d5h49_c5c9ddbd-6216-401d-ad22-dc4821de1bc2/revoke-keys/0.log"
Oct 30 01:58:06.270581 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:06.270552    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1279943c-c7glm_efab5e9d-fc6d-45b2-bdee-4d1c73ffa7e7/revoke-keys/0.log"
Oct 30 01:58:06.457471 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:06.457387    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1f6db33c-lqvw5_c609f8ad-3635-43a6-966d-27a69e356480/revoke-keys/0.log"
Oct 30 01:58:06.649138 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:06.649107    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8f7125d8-2v28d_9411c92b-9579-4916-b415-95c0ceb4bb3b/revoke-keys/0.log"
Oct 30 01:58:06.838605 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:06.838569    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9f6d1463-v6wwq_019931b1-92c8-48a2-88be-2a0fb9b18450/revoke-keys/0.log"
Oct 30 01:58:07.214114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:07.214032    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b6502d6e-tgwhf_c65fec65-f9ed-410d-abd0-f9ac048defe6/revoke-keys/0.log"
Oct 30 01:58:07.553794 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:58:07.553752    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:58:07.579429 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:07.579408    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f7181d33-777cj_7d7f6bf7-22f5-4a95-adc7-9683a8c4c8c1/revoke-keys/0.log"
Oct 30 01:58:07.763057 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:07.763024    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fe7fdd7e-k8ctr_11e155ec-1801-47f5-869f-a9fb5a1c467d/revoke-keys/0.log"
Oct 30 01:58:09.951585 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:09.951554    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-dbkj2_95a0d310-2023-40af-bf5e-d3e30e739590/kuadrant-console-plugin/0.log"
Oct 30 01:58:10.454744 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:10.454716    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gkbtv_7377e3da-0408-4a67-bfe1-f4e481cb09a8/limitador/0.log"
Oct 30 01:58:11.893862 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:11.893833    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57cd69bbcd-k5fn7_ae116534-9fdc-4ce1-8385-a2c7b0738d88/kube-auth-proxy/0.log"
Oct 30 01:58:12.430224 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:12.430190    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-cxntb_50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3/istio-proxy/0.log"
Oct 30 01:58:12.619255 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:12.619225    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-m5c9r_d084a21c-a3d0-4f2a-99ac-412c560f0e03/istio-proxy/0.log"
Oct 30 01:58:31.014503 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:31.014475    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fw99d_372da31c-aac8-485a-bcc4-31a830f176a9/global-pull-secret-syncer/0.log"
Oct 30 01:58:31.924266 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:31.924224    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5lmgn_f65424f1-dbb4-4e42-ab0a-a3334cef4dba/konnectivity-agent/0.log"
Oct 30 01:58:32.214913 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:32.214843    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-32.ec2.internal_f92445811ce6c3905e7d9666e6d2a0cd/haproxy/0.log"
Oct 30 01:58:37.123686 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:37.123653    4425 scope.go:122] "RemoveContainer" containerID="3080327d50bb7796cf0e5c90aa80f7014f9224d31e77df86b1afd1924514478e"
Oct 30 01:58:37.133073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:37.133053    4425 scope.go:122] "RemoveContainer" containerID="09390c33e53705893963c0fefd4bb0a17091d12d05d5074254042efbf8f1a093"
Oct 30 01:58:37.142299 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:37.142271    4425 scope.go:122] "RemoveContainer" containerID="d9270c4fb38fc8d9f0fe3397707dbc7664578c6ef3db6585d08cf26406e56f04"
Oct 30 01:58:37.150444 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:37.150419    4425 scope.go:122] "RemoveContainer" containerID="28bc8cdd986cdfe4ac61a2e911f949c77e289bd58a23487c3bdde0fbcfd2db06"
Oct 30 01:58:37.158534 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:37.158517    4425 scope.go:122] "RemoveContainer" containerID="506b3f4f4cc565683d558a81d718d65cdb7f3f447c63513ece5a91d3be9e8b88"
Oct 30 01:58:37.166869 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:37.166846    4425 scope.go:122] "RemoveContainer" containerID="26dece2db76b053e0dd6bf7c346095ecbe703c222157987e5321241268f01bb8"
Oct 30 01:58:37.174841 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:37.174822    4425 scope.go:122] "RemoveContainer" containerID="f982d0cda0ad4faf112b1441ba0401b679cab86fe9eb515c1c4ad9e86a33a214"
Oct 30 01:58:37.185786 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:37.185761    4425 scope.go:122] "RemoveContainer" containerID="e97588cd3bd5fb75b3e5cb30149ee59b2c1b38a6a266c0a43489f720233a8076"
Oct 30 01:58:37.193703 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:37.193675    4425 scope.go:122] "RemoveContainer" containerID="1736a92513b42c80bd7d5c427e144d5eae9cf516939dde412c1e05593df37a2c"
Oct 30 01:58:37.553102 ip-10-0-133-32 kubenswrapper[4425]: E1030 01:58:37.553063    4425 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz4ff" containerName="extractor"
Oct 30 01:58:40.601080 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:40.601052    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-dbkj2_95a0d310-2023-40af-bf5e-d3e30e739590/kuadrant-console-plugin/0.log"
Oct 30 01:58:41.330849 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:41.330816    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gkbtv_7377e3da-0408-4a67-bfe1-f4e481cb09a8/limitador/0.log"
Oct 30 01:58:43.275134 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.275099    4425 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"]
Oct 30 01:58:43.276202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.276172    4425 cpu_manager.go:414] "containerMap: removing container" podUID="11e155ec-1801-47f5-869f-a9fb5a1c467d" containerName="revoke-keys"
Oct 30 01:58:43.276202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.276193    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e155ec-1801-47f5-869f-a9fb5a1c467d" containerName="revoke-keys"
Oct 30 01:58:43.276410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.276218    4425 cpu_manager.go:414] "containerMap: removing container" podUID="66a19982-1f16-4a03-a629-bf9b7d6b50d3" containerName="main"
Oct 30 01:58:43.276410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.276223    4425 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a19982-1f16-4a03-a629-bf9b7d6b50d3" containerName="main"
Oct 30 01:58:43.276410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.276281    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e155ec-1801-47f5-869f-a9fb5a1c467d" containerName="revoke-keys"
Oct 30 01:58:43.276410 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.276293    4425 memory_manager.go:365] "RemoveStaleState removing state" podUID="66a19982-1f16-4a03-a629-bf9b7d6b50d3" containerName="main"
Oct 30 01:58:43.281364 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.281347    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:43.294624 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.294599    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qqxng\"/\"openshift-service-ca.crt\""
Oct 30 01:58:43.294754 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.294647    4425 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qqxng\"/\"kube-root-ca.crt\""
Oct 30 01:58:43.306609 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.306578    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"]
Oct 30 01:58:43.331712 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.331680    4425 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qqxng\"/\"default-dockercfg-jkqcz\""
Oct 30 01:58:43.444468 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.444427    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nldxq\" (UniqueName: \"kubernetes.io/projected/0efedd19-a2a9-4225-b82d-091dac0853cb-kube-api-access-nldxq\") pod \"perf-node-gather-daemonset-vmb5r\" (UID: \"0efedd19-a2a9-4225-b82d-091dac0853cb\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:43.444644 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.444476    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0efedd19-a2a9-4225-b82d-091dac0853cb-podres\") pod \"perf-node-gather-daemonset-vmb5r\" (UID: \"0efedd19-a2a9-4225-b82d-091dac0853cb\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:43.444644 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.444583    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0efedd19-a2a9-4225-b82d-091dac0853cb-proc\") pod \"perf-node-gather-daemonset-vmb5r\" (UID: \"0efedd19-a2a9-4225-b82d-091dac0853cb\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:43.444737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.444647    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0efedd19-a2a9-4225-b82d-091dac0853cb-sys\") pod \"perf-node-gather-daemonset-vmb5r\" (UID: \"0efedd19-a2a9-4225-b82d-091dac0853cb\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:43.444737 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.444722    4425 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0efedd19-a2a9-4225-b82d-091dac0853cb-lib-modules\") pod \"perf-node-gather-daemonset-vmb5r\" (UID: \"0efedd19-a2a9-4225-b82d-091dac0853cb\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:43.545560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.545489    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0efedd19-a2a9-4225-b82d-091dac0853cb-proc\") pod \"perf-node-gather-daemonset-vmb5r\" (UID: \"0efedd19-a2a9-4225-b82d-091dac0853cb\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:43.545560 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.545542    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0efedd19-a2a9-4225-b82d-091dac0853cb-sys\") pod \"perf-node-gather-daemonset-vmb5r\" (UID: \"0efedd19-a2a9-4225-b82d-091dac0853cb\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:43.545770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.545568    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0efedd19-a2a9-4225-b82d-091dac0853cb-lib-modules\") pod \"perf-node-gather-daemonset-vmb5r\" (UID: \"0efedd19-a2a9-4225-b82d-091dac0853cb\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:43.545770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.545587    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nldxq\" (UniqueName: \"kubernetes.io/projected/0efedd19-a2a9-4225-b82d-091dac0853cb-kube-api-access-nldxq\") pod \"perf-node-gather-daemonset-vmb5r\" (UID: \"0efedd19-a2a9-4225-b82d-091dac0853cb\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:43.545770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.545603    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0efedd19-a2a9-4225-b82d-091dac0853cb-proc\") pod \"perf-node-gather-daemonset-vmb5r\" (UID: \"0efedd19-a2a9-4225-b82d-091dac0853cb\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:43.545770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.545609    4425 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0efedd19-a2a9-4225-b82d-091dac0853cb-podres\") pod \"perf-node-gather-daemonset-vmb5r\" (UID: \"0efedd19-a2a9-4225-b82d-091dac0853cb\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:43.545770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.545656    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0efedd19-a2a9-4225-b82d-091dac0853cb-podres\") pod \"perf-node-gather-daemonset-vmb5r\" (UID: \"0efedd19-a2a9-4225-b82d-091dac0853cb\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:43.545770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.545691    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0efedd19-a2a9-4225-b82d-091dac0853cb-sys\") pod \"perf-node-gather-daemonset-vmb5r\" (UID: \"0efedd19-a2a9-4225-b82d-091dac0853cb\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:43.545770 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.545696    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0efedd19-a2a9-4225-b82d-091dac0853cb-lib-modules\") pod \"perf-node-gather-daemonset-vmb5r\" (UID: \"0efedd19-a2a9-4225-b82d-091dac0853cb\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:43.566680 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.566660    4425 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nldxq\" (UniqueName: \"kubernetes.io/projected/0efedd19-a2a9-4225-b82d-091dac0853cb-kube-api-access-nldxq\") pod \"perf-node-gather-daemonset-vmb5r\" (UID: \"0efedd19-a2a9-4225-b82d-091dac0853cb\") " pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:43.594577 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.594549    4425 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:43.896864 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:43.896833    4425 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"]
Oct 30 01:58:43.898413 ip-10-0-133-32 kubenswrapper[4425]: W1030 01:58:43.898385    4425 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0efedd19_a2a9_4225_b82d_091dac0853cb.slice/crio-d896efb8099367cf79575db92d79c20e8ced1ff35d2aa758b8a428c4590845d7 WatchSource:0}: Error finding container d896efb8099367cf79575db92d79c20e8ced1ff35d2aa758b8a428c4590845d7: Status 404 returned error can't find the container with id d896efb8099367cf79575db92d79c20e8ced1ff35d2aa758b8a428c4590845d7
Oct 30 01:58:44.285139 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:44.285103    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r" event={"ID":"0efedd19-a2a9-4225-b82d-091dac0853cb","Type":"ContainerStarted","Data":"7997e4faa97af87956b5ddad1c6b09b56f281973c045586f495184600cac2c8d"}
Oct 30 01:58:44.285568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:44.285178    4425 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r" event={"ID":"0efedd19-a2a9-4225-b82d-091dac0853cb","Type":"ContainerStarted","Data":"d896efb8099367cf79575db92d79c20e8ced1ff35d2aa758b8a428c4590845d7"}
Oct 30 01:58:44.285568 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:44.285217    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:44.527931 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:44.527889    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6fbd47b8fd-xdfgg_2602823e-b519-4ecd-80f5-c8c60472145b/metrics-server/0.log"
Oct 30 01:58:44.660049 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:44.659952    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-r4jl7_b3d1f8e1-ca3d-4276-bf20-e499ab93eeee/monitoring-plugin/0.log"
Oct 30 01:58:44.806072 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:44.806043    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8g6mn_95f31ac0-37f2-455c-a543-c7feba94a43e/node-exporter/0.log"
Oct 30 01:58:44.926791 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:44.926709    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8g6mn_95f31ac0-37f2-455c-a543-c7feba94a43e/kube-rbac-proxy/0.log"
Oct 30 01:58:44.954204 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:44.954151    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9zbsm_3009c800-6df6-4c1e-935f-3a7e5be62572/manager/0.log"
Oct 30 01:58:45.042908 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:45.042882    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8g6mn_95f31ac0-37f2-455c-a543-c7feba94a43e/init-textfile/0.log"
Oct 30 01:58:45.070060 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:45.070039    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9zbsm_3009c800-6df6-4c1e-935f-3a7e5be62572/manager/0.log"
Oct 30 01:58:45.943972 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:45.943939    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-gdsjr_be4dc5c1-48ab-4001-a35e-725eb618b395/kube-rbac-proxy-main/0.log"
Oct 30 01:58:45.988020 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:45.987986    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-n47qc_80392122-eba8-4040-a019-e44488201532/manager/0.log"
Oct 30 01:58:46.071755 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:46.071710    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-gdsjr_be4dc5c1-48ab-4001-a35e-725eb618b395/kube-rbac-proxy-self/0.log"
Oct 30 01:58:46.112196 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:46.112124    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-n47qc_80392122-eba8-4040-a019-e44488201532/manager/0.log"
Oct 30 01:58:46.385711 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:46.385673    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-gdsjr_be4dc5c1-48ab-4001-a35e-725eb618b395/openshift-state-metrics/0.log"
Oct 30 01:58:46.453435 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:46.453396    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6496db66cc-nzwpd_858deb1c-4c93-42c2-9fc9-8a53b27fa19b/manager/0.log"
Oct 30 01:58:46.492463 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:46.492411    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5mcgv_b42c9fca-e232-4900-adca-ddd04928e24c/server/0.log"
Oct 30 01:58:46.563508 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:46.563469    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6496db66cc-nzwpd_858deb1c-4c93-42c2-9fc9-8a53b27fa19b/manager/0.log"
Oct 30 01:58:46.566523 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:46.566497    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1a742db-26fb-4195-9676-364b7d9487c4/prometheus/0.log"
Oct 30 01:58:46.600001 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:46.599972    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5mcgv_b42c9fca-e232-4900-adca-ddd04928e24c/server/0.log"
Oct 30 01:58:46.687426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:46.687331    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1a742db-26fb-4195-9676-364b7d9487c4/config-reloader/0.log"
Oct 30 01:58:46.807342 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:46.807294    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1a742db-26fb-4195-9676-364b7d9487c4/thanos-sidecar/0.log"
Oct 30 01:58:46.931809 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:46.931775    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1a742db-26fb-4195-9676-364b7d9487c4/kube-rbac-proxy-web/0.log"
Oct 30 01:58:47.074593 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:47.074559    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1a742db-26fb-4195-9676-364b7d9487c4/kube-rbac-proxy/0.log"
Oct 30 01:58:47.203083 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:47.203048    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1a742db-26fb-4195-9676-364b7d9487c4/kube-rbac-proxy-thanos/0.log"
Oct 30 01:58:47.329695 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:47.329617    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a1a742db-26fb-4195-9676-364b7d9487c4/init-config-reloader/0.log"
Oct 30 01:58:47.469531 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:47.469502    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-j5kh4_4a2f55a8-9612-47a7-97cd-aeeb955b6659/prometheus-operator/0.log"
Oct 30 01:58:47.582667 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:47.582557    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-j5kh4_4a2f55a8-9612-47a7-97cd-aeeb955b6659/kube-rbac-proxy/0.log"
Oct 30 01:58:47.711875 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:47.711839    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-t5vgx_f64d1922-82b0-4b57-8ec5-032d8ec8da4f/prometheus-operator-admission-webhook/0.log"
Oct 30 01:58:48.227762 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:48.227728    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bf64bb4d4-jhfpw_ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0/thanos-query/0.log"
Oct 30 01:58:48.349281 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:48.349246    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bf64bb4d4-jhfpw_ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0/kube-rbac-proxy-web/0.log"
Oct 30 01:58:48.389114 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:48.389076    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9xl4p_33acc2a2-ca7f-4702-be45-c4f12d99d4b7/manager/0.log"
Oct 30 01:58:48.471664 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:48.471622    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bf64bb4d4-jhfpw_ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0/kube-rbac-proxy/0.log"
Oct 30 01:58:48.487727 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:48.487648    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9xl4p_33acc2a2-ca7f-4702-be45-c4f12d99d4b7/manager/0.log"
Oct 30 01:58:48.590536 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:48.590509    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bf64bb4d4-jhfpw_ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0/prom-label-proxy/0.log"
Oct 30 01:58:48.711388 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:48.711355    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bf64bb4d4-jhfpw_ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0/kube-rbac-proxy-rules/0.log"
Oct 30 01:58:48.829072 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:48.829044    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bf64bb4d4-jhfpw_ee7d3dc4-01b6-4d9d-a66d-cf31bdeab3e0/kube-rbac-proxy-metrics/0.log"
Oct 30 01:58:50.121217 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:50.121176    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9zbsm_3009c800-6df6-4c1e-935f-3a7e5be62572/manager/0.log"
Oct 30 01:58:50.229426 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:50.229389    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9zbsm_3009c800-6df6-4c1e-935f-3a7e5be62572/manager/0.log"
Oct 30 01:58:50.304804 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:50.304777    4425 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r"
Oct 30 01:58:50.344795 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:50.344708    4425 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qqxng/perf-node-gather-daemonset-vmb5r" podStartSLOduration=7.34468925 podStartE2EDuration="7.34468925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:44.324240189 +0000 UTC m=+4633.073895501" watchObservedRunningTime="2026-10-30 01:58:50.34468925 +0000 UTC m=+4639.094344561"
Oct 30 01:58:51.169319 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:51.169233    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-n47qc_80392122-eba8-4040-a019-e44488201532/manager/0.log"
Oct 30 01:58:51.274893 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:51.274852    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-n47qc_80392122-eba8-4040-a019-e44488201532/manager/0.log"
Oct 30 01:58:51.647563 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:51.647518    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6496db66cc-nzwpd_858deb1c-4c93-42c2-9fc9-8a53b27fa19b/manager/0.log"
Oct 30 01:58:51.699750 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:51.699713    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5mcgv_b42c9fca-e232-4900-adca-ddd04928e24c/server/0.log"
Oct 30 01:58:51.729661 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:51.729626    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6496db66cc-nzwpd_858deb1c-4c93-42c2-9fc9-8a53b27fa19b/manager/0.log"
Oct 30 01:58:51.806933 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:51.806898    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5mcgv_b42c9fca-e232-4900-adca-ddd04928e24c/server/0.log"
Oct 30 01:58:53.307385 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:53.307337    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-fptfc_75d61c2d-5a46-45a7-b9d7-fa072c685bde/download-server/0.log"
Oct 30 01:58:53.577245 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:53.577126    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9xl4p_33acc2a2-ca7f-4702-be45-c4f12d99d4b7/manager/0.log"
Oct 30 01:58:53.692073 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:53.692034    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9xl4p_33acc2a2-ca7f-4702-be45-c4f12d99d4b7/manager/0.log"
Oct 30 01:58:55.658746 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:55.658714    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9zbsm_3009c800-6df6-4c1e-935f-3a7e5be62572/manager/0.log"
Oct 30 01:58:55.750748 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:55.750711    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9zbsm_3009c800-6df6-4c1e-935f-3a7e5be62572/manager/0.log"
Oct 30 01:58:56.737938 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:56.737900    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-n47qc_80392122-eba8-4040-a019-e44488201532/manager/0.log"
Oct 30 01:58:57.037295 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:57.037254    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-n47qc_80392122-eba8-4040-a019-e44488201532/manager/0.log"
Oct 30 01:58:57.139202 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:57.139139    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6496db66cc-nzwpd_858deb1c-4c93-42c2-9fc9-8a53b27fa19b/manager/0.log"
Oct 30 01:58:57.195601 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:57.195570    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5mcgv_b42c9fca-e232-4900-adca-ddd04928e24c/server/0.log"
Oct 30 01:58:57.831571 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:57.831531    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6496db66cc-nzwpd_858deb1c-4c93-42c2-9fc9-8a53b27fa19b/manager/0.log"
Oct 30 01:58:57.845172 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:57.845124    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-66wbz_e0ff8a2c-7788-4791-8212-84a9a683320c/dns/0.log"
Oct 30 01:58:57.979816 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:57.979766    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-66wbz_e0ff8a2c-7788-4791-8212-84a9a683320c/kube-rbac-proxy/0.log"
Oct 30 01:58:58.095121 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:58.095039    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-5mcgv_b42c9fca-e232-4900-adca-ddd04928e24c/server/0.log"
Oct 30 01:58:59.000934 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:59.000882    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9xl4p_33acc2a2-ca7f-4702-be45-c4f12d99d4b7/manager/0.log"
Oct 30 01:58:59.047738 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:59.047709    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8k6rj_f8b80e7e-75db-4e4e-9c1b-0d2f4b754414/dns-node-resolver/0.log"
Oct 30 01:58:59.969121 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:58:59.969085    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9xl4p_33acc2a2-ca7f-4702-be45-c4f12d99d4b7/manager/0.log"
Oct 30 01:59:00.944658 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:59:00.944621    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7zgwc_f6870c47-e336-47b5-adfa-7143eed6d6ab/node-ca/0.log"
Oct 30 01:59:02.190962 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:59:02.190934    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57cd69bbcd-k5fn7_ae116534-9fdc-4ce1-8385-a2c7b0738d88/kube-auth-proxy/0.log"
Oct 30 01:59:02.334665 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:59:02.334637    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-cxntb_50e0627e-b7b6-4b7e-9adb-46e0d1e0b6b3/istio-proxy/0.log"
Oct 30 01:59:02.409520 ip-10-0-133-32 kubenswrapper[4425]: I1030 01:59:02.409465    4425 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-m5c9r_d084a21c-a3d0-4f2a-99ac-412c560f0e03/istio-proxy/0.log"